Closed juliomateoslangerak closed 4 years ago
Server side this is the traceback:
2020-01-07 15:31:10,740:XimeaCamera (microscope.devices):ERROR:PID 6452: in _fetch_loop:
Traceback (most recent call last):
File "C:\Users\omxt\PycharmProjects\microscope_oxford\microscope\microscope\devices.py", line 535, in _fetch_loop
data = self._fetch_data()
File "C:\Users\omxt\PycharmProjects\microscope_oxford\microscope\microscope\cameras\ximea.py", line 50, in _fetch_data
raise Exception('unhandled trigger type %s' % trigger_type)
Exception: unhandled trigger type XI_TRG_SOFTWARE
There's two different issues. The first issue is that the default is not software trigger. That's easy to solve. The second issue comes from 9fe103058e54afdee28f9808655657dff495e916 but @NickHallONBI should be able to comment (the original code only checked two conditions).
For starters, can you try my issue-131-ximea
branch?
Oh, when you call set_trigger_source
, it expects a value of 1 (edge rising) or 0 (software). Can you try:
camera.set_trigger_source(0)
edit: ignore. This is wrong. Ignore me. We have both set_trigger_source
and set_trigger_type
and got confused.
Right, so after a bit of digging, it seems that we never actually call the set_trigger_source/type in any of our other softwares (microAO, cockpit, BeamDelta, etc). It's quite possible that these methods never worked and because we were always happy with the defaults (software or hardware triggering) it just never came up.
Do not know if this has something to do with it but camera.get_is_enabled() returns nothing after changing trigger mode unless it is enabled again.
I think this is a separate issue and general to all cameras. Only CameraDevice.enable
sets the attribute of CameraDevice.enabled
to True
and it's set to False
in __init__
so until you call enable, CameraDevice.enabled
is None
.
The error seems to be caused by an uninitialized camera. Is actually 'initialize' ever called? I'm not sure cause I cannot see it in the logs. At least not at the right moment.
Is actually 'initialize' ever called?
Yes. It is called by the deviceserver. A device is only served after initialize
succeeds.
Ignoring the set/get trigger source/type, does my issue-131-ximea
branch lets you enable and then trigger and wait for an image?
Me and Nick explored this and found that when trigger source is set to edge rising, the camera still handles software triggers. This explains why the default trigger mode was edge rising, and trigger from cockpit still worked even though we never actually called any of the functions that changed the trigger source/type.
Anyway, we believe the actual issue here should have been fixed with https://github.com/MicronOxford/microscope/commit/973863886793e58314cb0e6e55ca41bc3f9109b0 we just checked on the actual camera. So closing this issue (please reopen if you still have issues).
I installed from scratch your commit and I still getting the errors. I paste output bellow.
I still think it is that the camera does not get initialized. A call to initialize should print 'Initializing.' in the log and I cannot find it.
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
>from microscope.clients import DataClient
>camera = DataClient('PYRO:XimeaCamera@127.0.0.1:8006')
>camera.enable()
>camera.get_is_enabled()
True
>data = camera.trigger_and_wait()
>data[0]
Exception(b'ERROR 45: Acquisition is stopped. It needs to be started to perform operation.')
>camera.get_trigger_type()
1
>camera.set_trigger_type(0)
>camera.get_trigger_type()
3
>data = camera.trigger_and_wait()
>data[0]
Exception(b'ERROR 45: Acquisition is stopped. It needs to be started to perform operation.')
>camera.disable()
>camera.shutdown()
2020-01-10 10:56:22,923:XimeaCamera (__mp_main__):INFO:PID 5048: Device initialized; starting daemon.
2020-01-10 10:56:22,923:XimeaCamera (__mp_main__):INFO:PID 5048: Serving PYRO:XimeaCamera@127.0.0.1:8006
2020-01-10 10:58:24,887:XimeaCamera (microscope.devices):INFO:PID 5048: Current client is <Pyro4.core.Proxy at 0x2bf5988; not connected; for PYRO:obj_8579e1e3ab0147aa895e2d6f3d47cee2@127.0.0.1:50487>.
2020-01-10 10:58:24,888:XimeaCamera (microscope.devices):DEBUG:PID 5048: Enabling ...
2020-01-10 10:58:24,888:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Preparing for acquisition.
2020-01-10 10:58:24,889:XimeaCamera (microscope.devices):ERROR:PID 5048: in _fetch_loop:
Traceback (most recent call last):
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\devices.py", line 535, in _fetch_loop
data = self._fetch_data()
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 68, in _fetch_data
raise err
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 55, in _fetch_data
self._handle.get_image(self.img)
File "C:\Users\omxt\PycharmProjects\microscope_oxford\venv\lib\site-packages\ximea\xiapi.py", line 322, in get_image
raise Xi_error(stat)
ximea.xiapi.Xi_error: ERROR 45: Acquisition is stopped. It needs to be started to perform operation.
2020-01-10 10:58:24,893:XimeaCamera (microscope.devices):ERROR:PID 5048: in _fetch_loop:
Traceback (most recent call last):
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\devices.py", line 535, in _fetch_loop
data = self._fetch_data()
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 68, in _fetch_data
raise err
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 55, in _fetch_data
self._handle.get_image(self.img)
File "C:\Users\omxt\PycharmProjects\microscope_oxford\venv\lib\site-packages\ximea\xiapi.py", line 322, in get_image
raise Xi_error(stat)
2020-01-10 10:58:24,943:XimeaCamera (microscope.devices):ERROR:PID 5048: in _fetch_loop:
Traceback (most recent call last):
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\devices.py", line 535, in _fetch_loop
data = self._fetch_data()
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 68, in _fetch_data
raise err
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 55, in _fetch_data
self._handle.get_image(self.img)
File "C:\Users\omxt\PycharmProjects\microscope_oxford\venv\lib\site-packages\ximea\xiapi.py", line 322, in get_image
raise Xi_error(stat)
--- this goes forever ---
ximea.xiapi.Xi_error: ERROR 45: Acquisition is stopped. It needs to be started to perform operation.
2020-01-10 10:58:24,949:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Acquisition enabled.
2020-01-10 10:58:24,950:XimeaCamera (microscope.devices):DEBUG:PID 5048: ... enabled.
2020-01-10 11:01:56,889:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Soft trigger received; self._acquiring is True.
2020-01-10 11:01:57,025:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Fetched imaged with dims (2048, 2048) and size 4194304.
2020-01-10 11:01:57,025:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Sending image
2020-01-10 11:05:22,961:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: called get trigger type XI_TRG_EDGE_RISING
2020-01-10 11:05:33,082:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Set trigger type 0
2020-01-10 11:05:33,082:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Disabling acquisition.
2020-01-10 11:05:33,337:XimeaCamera (microscope.devices):ERROR:PID 5048: in _fetch_loop:
Traceback (most recent call last):
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\devices.py", line 535, in _fetch_loop
data = self._fetch_data()
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 68, in _fetch_data
raise err
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 55, in _fetch_data
self._handle.get_image(self.img)
File "C:\Users\omxt\PycharmProjects\microscope_oxford\venv\lib\site-packages\ximea\xiapi.py", line 322, in get_image
raise Xi_error(stat)
ximea.xiapi.Xi_error: ERROR 45: Acquisition is stopped. It needs to be started to perform operation.
2020-01-10 11:05:33,379:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Preparing for acquisition.
2020-01-10 11:05:33,380:XimeaCamera (microscope.devices):ERROR:PID 5048: in _fetch_loop:
Traceback (most recent call last):
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\devices.py", line 535, in _fetch_loop
data = self._fetch_data()
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 68, in _fetch_data
raise err
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 55, in _fetch_data
self._handle.get_image(self.img)
File "C:\Users\omxt\PycharmProjects\microscope_oxford\venv\lib\site-packages\ximea\xiapi.py", line 322, in get_image
raise Xi_error(stat)
--- this goes forever ---
ximea.xiapi.Xi_error: ERROR 45: Acquisition is stopped. It needs to be started to perform operation.
2020-01-10 11:05:33,439:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Acquisition enabled.
2020-01-10 11:05:33,440:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Trigger type XI_TRG_SOFTWARE
2020-01-10 11:05:33,440:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: GPI Selector XI_GPI_PORT1
2020-01-10 11:05:33,440:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: GPI Mode XI_GPI_OFF
2020-01-10 11:05:36,290:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: called get trigger type XI_TRG_SOFTWARE
2020-01-10 11:05:45,115:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Soft trigger received; self._acquiring is True.
2020-01-10 11:05:45,249:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Fetched imaged with dims (2048, 2048) and size 4194304.
2020-01-10 11:05:45,249:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Sending image
2020-01-10 11:07:03,523:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Disabling acquisition.
2020-01-10 11:07:10,466:XimeaCamera (microscope.cameras.ximea):INFO:PID 5048: Disabling acquisition.
2020-01-10 11:07:10,467:XimeaCamera (microscope.devices):INFO:PID 5048: Shutting down ... ... ...
2020-01-10 11:07:10,471:XimeaCamera (microscope.devices):INFO:PID 5048: ... ... ... ... shut down completed.
funny. If I call initialize at the end of the init method, Initializing is printed twice. And things do not get better.
Maybe camera firmware of api differences? mine is:
Name: MQ042MG-CM
Sensor: Mono
Type: USB3.0
Serial number: 38983550
User ID:
I cannot reopen the issue BTW
I managed to reproduce your new issue. There's a few things going on:
When you call XimeaCamera.enable
, it first starts the fetch thread and then starts the acquisition which means that the fetch thread must handle not having the acquisition started yet. That problem should go away if on the ximea module line 117, you do self._acquiring = True
after self._handle.start_acquisition()
If start_acquisition()
fails, the camera is left as disabled but it's also set to acquiring and the fetch thread is not stopped. The end result is that each millisecond the fetch thread fails with error 45 ("Acquisition is stopped. It needs to be started to perform operation") when it tries to fetch an image. And indeed acquisition was actually never started successfully. There's probably some other error in the logs, from when start_acquisition()
failed, but it will be under thousands of tracebacks for error 45. If youu fix the point 1 above as I suggest above, it should be clear what error comes out of start_acquisition()
(on my case is error 13).
I would also recommend you try to test this without deviceserver, so you also get the xiAPI log messages. Like so:
>>> import microscope.cameras.ximea
>>> import logging
>>> import sys
>>> logging.getLogger().addHandler(logging.StreamHandler(sys.stderr))
>>> logging.getLogger().setLevel(logging.DEBUG)
>>> c = microscope.cameras.ximea.XimeaCamera()
>>> c.initialize()
xiAPI: ---- xiOpenDevice API:V4.19.11.00 started ----
[... Loads of xiAPI log messages very useful ...]
Initializing.
xiAPI: Sensor SetExposure freq=48MHz exp=28us regexp=x2
>>> c.enable()
Enabling ...
Preparing for acquisition.
[... Loads of xiAPI log messages very useful ...]
Error in _on_enable:
[... traceback information ...]
ximea.xiapi.Xi_error: ERROR 13: Attach buffers error
I still think it is that the camera does not get initialized. A call to initialize should print 'Initializing.' in the log and I cannot find it.
Are you looking at the stderr or the at log files? The individual device log files are only created after initialization so it won't appear there. They should appear on stderr though.
So I applied your change and that seems to fix the problem. As you say the camera seems to respond to software triggers even when set in HW trigger mode. However if I change to software trigger mode, something must go wrong during the 'Disabling_acquisition' and I get the error 45 again. I must disable and enable my self to get things to work again after setting the software trigger. Here are the logs
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
>from microscope.clients import DataClient
>camera = DataClient('PYRO:XimeaCamera@127.0.0.1:8006')
>camera.enable()
>data = camera.trigger_and_wait()
>data[0]
array([[3, 2, 2, ..., 3, 2, 3],
[5, 6, 6, ..., 6, 6, 5],
[5, 6, 6, ..., 5, 6, 5],
...,
[5, 5, 6, ..., 4, 5, 5],
[5, 5, 5, ..., 5, 5, 6],
[4, 4, 4, ..., 4, 5, 4]], dtype=uint8)
>camera.get_trigger_type()
1
>camera.get_trigger_source()
'XI_TRG_EDGE_RISING'
>camera.set_trigger_type(0)
>camera.get_trigger_source()
'XI_TRG_SOFTWARE'
>camera.get_trigger_type()
3
>data = camera.trigger_and_wait()
>data[0]
Exception(b'ERROR 45: Acquisition is stopped. It needs to be started to perform operation.')
>camera.get_is_enabled()
True
>camera.disable()
>camera.get_is_enabled()
False
>camera.enable()
>camera.get_is_enabled()
True
>data = camera.trigger_and_wait()
>data[0]
array([[3, 3, 3, ..., 2, 2, 2],
[6, 6, 6, ..., 6, 6, 6],
[5, 5, 6, ..., 6, 6, 6],
...,
[6, 5, 6, ..., 5, 6, 5],
[5, 5, 5, ..., 5, 6, 7],
[4, 4, 5, ..., 4, 4, 4]], dtype=uint8)
>camera.shutdown()
(venv) C:\Users\omxt\PycharmProjects\microscope_ximea>python -m microscope.deviceserver config_file.py
2020-01-13 10:14:01,210:XimeaCamera (root):DEBUG:PID 3944: Debugging messages on.
2020-01-13 10:14:03,748:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Initializing.
2020-01-13 10:14:03,795:XimeaCamera (__mp_main__):INFO:PID 3944: Device initialized; starting daemon.
2020-01-13 10:14:03,796:XimeaCamera (__mp_main__):INFO:PID 3944: Serving PYRO:XimeaCamera@127.0.0.1:8006
2020-01-13 10:14:43,793:XimeaCamera (microscope.devices):INFO:PID 3944: Current client is <Pyro4.core.Proxy at 0x41b82c8; not connected; for PYRO:obj_b62d5343f38541e69d3cb5edd67368f9@127.0.0.1:53361>.
2020-01-13 10:14:43,794:XimeaCamera (microscope.devices):DEBUG:PID 3944: Enabling ...
2020-01-13 10:14:43,794:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Preparing for acquisition.
2020-01-13 10:14:43,854:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Acquisition enabled.
2020-01-13 10:14:43,854:XimeaCamera (microscope.devices):DEBUG:PID 3944: ... enabled.
2020-01-13 10:15:07,377:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Soft trigger received; self._acquiring is True.
2020-01-13 10:15:07,491:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Fetched imaged with dims (2048, 2048) and size 4194304.
2020-01-13 10:15:07,491:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Sending image
2020-01-13 10:15:29,378:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: called get trigger type XI_TRG_EDGE_RISING
2020-01-13 10:16:27,345:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Set trigger type 0
2020-01-13 10:16:27,346:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Disabling acquisition.
2020-01-13 10:16:27,515:XimeaCamera (microscope.devices):ERROR:PID 3944: in _fetch_loop:
Traceback (most recent call last):
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\devices.py", line 535, in _fetch_loop
data = self._fetch_data()
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 68, in _fetch_data
raise err
File "C:\Users\omxt\PycharmProjects\microscope_ximea\microscope\cameras\ximea.py", line 55, in _fetch_data
self._handle.get_image(self.img)
File "C:\Users\omxt\PycharmProjects\microscope_oxford\venv\lib\site-packages\ximea\xiapi.py", line 322, in get_image
raise Xi_error(stat)
ximea.xiapi.Xi_error: ERROR 45: Acquisition is stopped. It needs to be started to perform operation.
2020-01-13 10:16:27,638:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Preparing for acquisition.
2020-01-13 10:16:27,698:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Acquisition enabled.
2020-01-13 10:16:27,698:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Trigger type XI_TRG_SOFTWARE
2020-01-13 10:16:27,698:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: GPI Selector XI_GPI_PORT1
2020-01-13 10:16:27,699:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: GPI Mode XI_GPI_OFF
2020-01-13 10:18:31,042:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: called get trigger type XI_TRG_SOFTWARE
2020-01-13 10:18:49,842:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Soft trigger received; self._acquiring is True.
2020-01-13 10:18:49,955:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Fetched imaged with dims (2048, 2048) and size 4194304.
2020-01-13 10:18:49,955:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Sending image
2020-01-13 10:19:17,187:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Disabling acquisition.
2020-01-13 10:19:34,851:XimeaCamera (microscope.devices):INFO:PID 3944: Current client is <Pyro4.core.Proxy at 0x41b8708; not connected; for PYRO:obj_b62d5343f38541e69d3cb5edd67368f9@127.0.0.1:53361>.
2020-01-13 10:19:34,852:XimeaCamera (microscope.devices):DEBUG:PID 3944: Enabling ...
2020-01-13 10:19:34,852:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Preparing for acquisition.
2020-01-13 10:19:34,912:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Acquisition enabled.
2020-01-13 10:19:34,912:XimeaCamera (microscope.devices):DEBUG:PID 3944: ... enabled.
2020-01-13 10:19:55,059:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Soft trigger received; self._acquiring is True.
2020-01-13 10:19:55,172:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Fetched imaged with dims (2048, 2048) and size 4194304.
2020-01-13 10:19:55,172:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Sending image
2020-01-13 10:21:40,501:XimeaCamera (microscope.cameras.ximea):INFO:PID 3944: Disabling acquisition.
2020-01-13 10:21:40,680:XimeaCamera (microscope.devices):INFO:PID 3944: Shutting down ... ... ...
2020-01-13 10:21:40,684:XimeaCamera (microscope.devices):INFO:PID 3944: ... ... ... ... shut down completed.
I tried as you mention without deviceserver but I don't get the ximeaAPI messages...
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
>import microscope.cameras.ximea
>import logging
>import sys
>logging.getLogger().addHandler(logging.StreamHandler(sys.stderr))
>logging.getLogger().setLevel(logging.DEBUG)
>c = microscope.cameras.ximea.XimeaCamera()
>c.initialize()
Initializing.
>c.enable()
Enabling ...
Preparing for acquisition.
Acquisition enabled.
True
enabled.
>c.shutdown()
Disabling acquisition.
Shutting down ... ... ...
... ... ... shut down completed.
So for the initialization, indeed I must have been looking at the log file. Though normally I follow the trace in the terminal... 'Initializing.' is there
Can you pop by #python-microscope on irc.freenode for a debug session? I think we would benefit for a bit less async debug session.
How do I do that? I went to the webchat but could not find #python-microscope
How do I do that? I went to the webchat but could not find #python-microscope
You can use yuour own IRC client if you have one. Otherwise, https://webchat.freenode.net/ should work. I just tried on another computer and worked fine (wait a few seconds for the connection). Make sure, there's no typo on #python-microscope
(the #
is part of channel name).
IRC client is not working. Probably blocked by the IT the webchat I'm connected as Julio55 but I cannot find the channel
I spoke with @juliomateoslangerak over IRC. The original problem on this issue was fixed by changing the order of calling start_acquisition()
and setting _acquiring
but in the mean time 657948abd8eb0c0834efb1a1f8 also fixed the issue and applies to all cameras.
The other general issue of set/get trigger type/source not working properly is now also fixed by making use of the TriggerTargetMixIn
with 6c1974ab57f5b608614d.
Closing as fixed.
I'm having issues getting data from the ximea camera on the new merged version.
Do not know if this has something to do with it but camera.get_is_enabled() returns nothing after changing trigger mode unless it is enabled again.