Query Frequency Indexing Error

Jemetzner commented 5 years ago

We have encounterd a TypeError when executing the code coming from this line on the gui.py code. It seems the databases is not being indexed properly without this additional call. freq = self._gui.freq_db[self.laser]["freq"] -> freq = self._gui.freq_db[self.laser][0]["freq"]

hartytp commented 5 years ago

Thanks for the bug report. Can you link to the original line in the source code via git hub to make it easier to locate please?

hartytp commented 5 years ago

Also please can you post all the info we’d need to reproduce this issue like

output of ‘conda list’ and more generally how you installed /set this up
how you launched the code
logs (in verbose mode) from server and master
hardware set up
config files your using etc

As we don’t hit this bug my guess is it’s something to do with simulation mode being broken, a problem with some of the Oxford specific code like config file loading, or the config files being out of date. I’ll look at it next week, but it would help a lot to have enough info to be able to reproduce the issue rather

aquinn2 commented 5 years ago

Thank you for the feedback! The problematic line is 375 of gui.py, though every line where the code tries to access elements of freq_db runs into the same issues.

https://github.com/OxfordIonTrapGroup/wand/blob/8757c942e74de272ff4b9ea7bf9a10194b543871/wand/gui.py#L373-L376

We ran wand_server through the Anaconda prompt in an environment containing WAnD and get the following output:

$ wand_server -n example -v

INFO:wand.drivers.high_finesse:Connected to WLM server
WARNING:wand.drivers.high_finesse:No Switch Detected
INFO:wand.drivers.high_finesse:Connected to WLM 8 rev 4086, firmware 6491.3
WARNING:wand.tools:Unable to backup configuration file
INFO:wand.frontend.wand_server:server started
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_854
INFO:wand.frontend.wand_server:task 0 complete
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_850
INFO:wand.frontend.wand_server:task 1 complete
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_866
INFO:wand.frontend.wand_server:example_866 fast mode timeout
INFO:wand.frontend.wand_server:task 2 complete

We then ran wand_gui through another Anaconda prompt and get this output:

$ wand_gui -n example -v

INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
ERROR:wand.gui:Error connecting to server 'example' list indices must be integers or slices, not str here
<class 'TypeError'> gui.py 232
. . . .
WARNING:quamash.QEventLoop:Event callback failed: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
Traceback (most recent call last):
  File "c:\users\laserstation\anaconda3\envs\wand\lib\site-packages\Quamash-0.5.5-py3.6.egg\quamash\_windows.py", line 41, in _process_events
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\windows_events.py", line 437, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\laserstation\anaconda3\envs\wand\lib\site-packages\Quamash-0.5.5-py3.6.egg\quamash\_windows.py", line 44, in _process_events
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\windows_events.py", line 79, in set_exception
    super().set_exception(exception)
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\futures.py", line 361, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
asyncio.futures.InvalidStateError: CANCELLED: <_OverlappedFuture cancelled>

where . . . . indicates that the TypeError was repeatedly printed until the program terminated. While the master was running, it indicated that there was no connection between the master and the server:

We were able to correct this issue by changing every instance of data being called from freq_db (in both gui.py and ControlInterface.get_freq in server.py) from, e.g.,

freq = self._gui.freq_db[self.laser]["freq"]

to

freq = self._gui.freq_db[self.laser][0]["freq"]

When we make these changes, the logging for the server and master respectively are as follows:

$ wand_server -n example -v

INFO:wand.drivers.high_finesse:Connected to WLM server
WARNING:wand.drivers.high_finesse:No Switch Detected
INFO:wand.drivers.high_finesse:Connected to WLM 8 rev 4086, firmware 6491.3
WARNING:wand.tools:Unable to backup configuration file
INFO:wand.frontend.wand_server:server started
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_850
INFO:wand.frontend.wand_server:task 0 complete
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_854
INFO:wand.frontend.wand_server:task 1 complete
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_866
INFO:wand.frontend.wand_server:task 2 complete

$ wand_gui -n example

INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'

and the GUI indicates that it is able to communicate with the server.

Lastly, if we add a line to the very start of update_freq in gui.py:

print(self._gui.freq_db[self.laser][0])

we get the following output from the master:

$ wand_gui -n example

INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
{'status': 1, 'timestamp': 1566603453.642318, 'freq': 0}
ERROR:wand.gui:Error connecting to server 'example' list indices must be integers or slices, not str here
<class 'TypeError'> gui.py 232
. . . .
ERROR:wand.gui:Error connecting to server 'example' list indices must be integers or slices, not str here
<class 'TypeError'> gui.py 232
WARNING:quamash.QEventLoop:Event callback failed: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
Traceback (most recent call last):
  File "c:\users\laserstation\anaconda3\envs\wand\lib\site-packages\Quamash-0.5.5-py3.6.egg\quamash\_windows.py", line 41, in _process_events
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\windows_events.py", line 437, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\laserstation\anaconda3\envs\wand\lib\site-packages\Quamash-0.5.5-py3.6.egg\quamash\_windows.py", line 44, in _process_events
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\windows_events.py", line 79, in set_exception
    super().set_exception(exception)
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\futures.py", line 361, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
asyncio.futures.InvalidStateError: CANCELLED: <_OverlappedFuture cancelled>

(edit: didn't point clearly to the output of the print statement) with the significant output being

{'status': 1, 'timestamp': 1566603453.642318, 'freq': 0}

From this output, it looks like freq_db is being initialized as a dictionary of lists of dictionaries, with each list being of length 1, instead of just a dictionary of dictionaries.

It's very much worth noting that our version of the code is heavily modified. We removed all material related solely to OSA, and it's entirely possible we changed something important in the process. However, freq_db appears to be initialized in more or less the same way as laser_db is (in the below code snippet?), so we don't see why they would be initializing with different structures.

https://github.com/OxfordIonTrapGroup/wand/blob/8757c942e74de272ff4b9ea7bf9a10194b543871/wand/frontend/wand_gui.py#L219-L260

On the software environment side, we're running this code on a Windows 10 machine in an environment with the following packages:

$ conda list

# Name                    Version                   Build  Channel
aiohttp                   3.4.2            py35hfa6e2cd_0
alabaster                 0.7.11                   py35_0
artiq                     5.6911.e9b78b62.beta               0    https://conda.m-labs.hk/artiq-beta
artiq-board-kasli-mitll   5.6911.e9b78b62.beta               0    https://conda.m-labs.hk/artiq-beta
artiq-board-kc705-nist_clock 5.6911.e9b78b62.beta               0    https://conda.m-labs.hk/artiq-beta
astroid                   1.6.5                    py35_0
async-timeout             3.0.0                    py35_0
attrs                     19.1.0                     py_0
babel                     2.7.0                      py_0
backcall                  0.1.0                    py35_0
binutils-or1k-linux       2.27                 h93a10e1_6    m-labs
blas                      1.0                         mkl
bleach                    2.1.4                    py35_0
bscan-spi-bitstreams      0.10.0                        2    m-labs
ca-certificates           2019.5.15                     1
certifi                   2018.8.24                py35_1
cffi                      1.11.5           py35h74b6da3_1
chardet                   3.0.4                    py35_1
cloudpickle               1.2.1                      py_0
colorama                  0.3.9            py35h32a752f_0
curl                      7.65.2               h2a8f88b_0
decorator                 4.4.0                      py_0
defusedxml                0.6.0                      py_0
docutils                  0.14             py35h8ccb97f_0
entrypoints               0.2.3                    py35_2
git                       2.22.0                        0    conda-forge
h5py                      2.8.0            py35h3bdd7fb_2
hdf5                      1.10.2               hac2f561_1
html5lib                  1.0.1                    py35_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha66f8fd_1
idna                      2.7                      py35_0
idna_ssl                  1.1.0                    py35_0
imagesize                 1.1.0                    py35_0
intel-openmp              2019.4                      245
ipykernel                 4.10.0                   py35_0
ipython                   6.5.0                    py35_0
ipython_genutils          0.2.0            py35ha709e79_0
isort                     4.3.4                    py35_0
jedi                      0.12.1                   py35_0
jinja2                    2.10                     py35_0
jpeg                      9b                   hb83a4c4_2
jsonschema                2.6.0            py35h27d56d3_0
jupyter_client            5.3.1                      py_0
jupyter_core              4.5.0                      py_0
keyring                   13.2.1                   py35_0
krb5                      1.16.1               hc04afaa_7
lazy-object-proxy         1.3.1            py35hfa6e2cd_2
libcurl                   7.65.2               h2a8f88b_0
libgit2                   0.27.8               hfac1375_0    conda-forge
libiconv                  1.15                 h1df5818_7
libpng                    1.6.37               h2a8f88b_0
libsodium                 1.0.16               h9d3ae62_0
libssh2                   1.8.2                h7a1dbc1_0
libusb                    1.0.20                        0    m-labs
llvmlite-artiq            0.23.0.dev               py35_5    m-labs
markupsafe                1.0              py35hfa6e2cd_1
mccabe                    0.6.1                    py35_1
mistune                   0.8.3            py35hfa6e2cd_1
mkl                       2018.0.3                      1
mkl_fft                   1.0.6            py35hdbbee80_0
mkl_random                1.0.1            py35h77b88f5_1
multidict                 4.4.0            py35hfa6e2cd_0
nbconvert                 5.5.0                      py_0
nbformat                  4.4.0            py35h908c9d9_0
nidaqmx                   0.5.7                    pypi_0    pypi
numpy                     1.15.2           py35ha559c80_0
numpy-base                1.15.2           py35h8128ebf_0
numpydoc                  0.9.1                      py_0
openocd                   0.10.0                        6    m-labs
openssl                   1.1.1c               he774522_1
packaging                 17.1                     py35_0
pandoc                    2.2.3.2                       0
pandocfilters             1.4.2                    py35_1
parso                     0.5.1                      py_0
pickleshare               0.7.4            py35h2f9f535_0
pip                       18.0                  py35_1001    conda-forge
prettytable               0.7.2                      py_3    conda-forge
prompt_toolkit            1.0.15           py35h89c7cb4_0
psutil                    5.4.7            py35hfa6e2cd_0
pycodestyle               2.4.0                    py35_0
pycparser                 2.19                     py35_0
pydaqmx                   1.4.2                    pypi_0    pypi
pyflakes                  2.0.0                    py35_0
pygit2                    0.27.1           py35hfa6e2cd_0    conda-forge
pygments                  2.4.2                      py_0
pylint                    1.9.2                    py35_0
pyparsing                 2.4.2                      py_0
pyqt                      5.9.2            py35h6538335_2
pyqt5                     5.13.0                   pypi_0    pypi
pyqt5-sip                 4.19.18                  pypi_0    pypi
pyqtgraph                 0.10.0           py35h28b3542_3
python                    3.5.6                he025d50_0
python-dateutil           2.7.3                    py35_0
python-levenshtein        0.12.0                   py35_0    conda-forge
pythonparser              1.1                        py_8    m-labs
pytz                      2019.2                     py_0
pywin32                   223              py35hfa6e2cd_1
pyzmq                     17.1.2           py35hfa6e2cd_0
qt                        5.9.7            vc14h73c81de_0
qtawesome                 0.5.7                      py_0
qtconsole                 4.5.3                      py_0
qtpy                      1.9.0                      py_0
quamash                   0.5.5                      py_4    m-labs
regex                     2018.08.29       py35hfa6e2cd_0
requests                  2.13.0                   py35_0    conda-forge
rope                      0.14.0                     py_0
scipy                     1.1.0            py35h4f6bf74_1
setuptools                40.2.0                   py35_0
simplegeneric             0.8.1                    py35_2
sip                       4.19.8           py35h6538335_0
six                       1.11.0                   py35_1
snowballstemmer           1.9.0                      py_0
sphinx                    2.1.2                      py_0
sphinxcontrib-applehelp   1.0.1                      py_0
sphinxcontrib-devhelp     1.0.1                      py_0
sphinxcontrib-htmlhelp    1.0.2                      py_0
sphinxcontrib-jsmath      1.0.1                      py_0
sphinxcontrib-qthelp      1.0.2                      py_0
sphinxcontrib-serializinghtml 1.1.3                      py_0
spyder                    3.3.1                    py35_1
spyder-kernels            0.2.6                    py35_0
sqlite                    3.29.0               he774522_0
testpath                  0.3.1            py35h06cf69e_0
tk                        8.6.8                hfa6e2cd_0
tornado                   5.1.1            py35hfa6e2cd_0
traitlets                 4.3.2            py35h09b975b_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_4
wand                      0.1                      pypi_0    pypi
wcwidth                   0.1.7            py35h6e80d8a_0
webencodings              0.5.1                    py35_1
wheel                     0.31.1                   py35_0
win_unicode_console       0.5              py35h56988b5_0
wincertstore              0.2              py35hfebbdb8_0
wrapt                     1.10.11          py35hfa6e2cd_2
yarl                      1.2.6            py35hfa6e2cd_0
zeromq                    4.2.5                he025d50_1
zlib                      1.2.11               h62dcd97_3

and our configuration files (example_server_config.pyon and example_gui_config.pyon respectively) are shown below:

{"lasers": {
    "example_850": {
        "auto_exposure": true,
        "channel": 1,
        "exposure": [12, 2],
        "f_ref": 352682391500000.0,
        "fast_enabled_at": 0,
        "fast_mode": false,
        "fast_mode_set_at": 1556712340.9685378,
        "host": "192.168.1.204",
        "lock_capture_range": 2000000000.0,
        "lock_gain": 1e-10,
        "lock_owner": "",
        "lock_poll_time": 0.1,
        "lock_ready": true,
        "locked": false,
        "locked_at": 0
    },
    "example_854": {
        "auto_exposure": true,
        "channel": 2,
        "exposure": [8, 4],
        "f_ref": 350862971100000.0,
        "fast_enabled_at": 0,
        "fast_mode": false,
        "fast_mode_set_at": 1556712341.0285378,
        "host": "192.168.1.202",
        "lock_capture_range": 2000000000.0,
        "lock_gain": 2e-09,
        "lock_owner": "me",
        "lock_poll_time": 0.1,
        "lock_ready": true,
        "locked": false,
        "locked_at": 1556642908.006538
    },
    "example_866": {
        "auto_exposure": false,
        "channel": 3,
        "exposure": [5, 3],
        "f_ref": 346050071100000.0, 
        "fast_enabled_at": 0,
        "fast_mode": false,
        "fast_mode_set_at": 1566510801.3268256,
        "host": "192.168.1.202",
        "lock_capture_range": 2000000000.0,
        "lock_gain": 1e-10,
        "lock_owner": "",
        "lock_poll_time": 0.1,
        "lock_ready": true,
        "locked": false,
        "locked_at": 1556636337.6695378
    }
}}

{
    "servers": {
        "example": {
                "host": "127.0.0.1",
                "notify": 3250,
                "control": 3251
        }
    },
    "display_names": {

        "866":"example_866",
        "850":"example_850",
        "854":"example_854",

    },
    "layout": [
        ["850", "854", "866", ]

    ]
}

Thank you again for your time and energy. I don't know how reproducible this error will be with the code in this repository, since again, our code is modified. We can send the code we are using if it would help.

hartytp commented 5 years ago

Hmmm...the change you’ve made to the code doesn’t look right to me so I think something is very broken here.

Please try running an unmodified version of the code in simulation mode (no hardware) and see what happens.

hartytp commented 5 years ago

to elaborate: if you look at where the freq dB is defined it’s just a dictionary keyed by the laser names https://github.com/OxfordIonTrapGroup/wand/blob/8757c942e74de272ff4b9ea7bf9a10194b543871/wand/frontend/wand_server.py#L123 so I don’t see why the extra index would be needed

aquinn2 commented 5 years ago

We were able to make the original code work for with our setup by adding simulation modes specific to the OSA and switch. That is, we can now take readings from our wavemeter without having to make further changes to the code. We're still very unclear on why freq_db was taking on an extra index in our modified code, but that's probably a moot point now!

hartytp commented 5 years ago

Glad you were able to sort it out. Do you have everything up and running now?

BTW I've pushed a few commits recently that may make things easier for you.

OxfordIonTrapGroup / wand

Query Frequency Indexing Error #25