search

esrf-bliss / Lima

Library for Image Acquisition
https://lima1.rtfd.io
GNU General Public License v3.0
39 stars 71 forks source link

Speed (update) issues #122

Open hayg25 opened 3 years ago

hayg25 commented 3 years ago

Dear @laurent-claustre , I am using an Ace1920-40 gm in Tango. We are actually facing speed issues. In tango we use the LimaCCDs LiveViewer to check our images. But the image are updating each 10s and image counter increases by a factor close to 10 which is in agreement with the displayed Frame Rate (9.96 Hz). How could we update the image more frequently. 10s is much to long for the video in order to see any changes on parameters (for example laser mirrors).

Many thanks,

laurent-claustre commented 3 years ago

Hi, your basler camera image size is about 1920x1200 pixels 16bit, that means a image about 4.39MB. Are you running the LimaViewer on the same computer where you run LimaCCDs server? Transfering 4.39MB at 10Hz means 44MB/s (352Mbit/s) if you don't have a 1Gbit ethernet connection between the 2 computers you cannot follow the speed.

hayg25 commented 3 years ago

Dear @laurent-claustre, as I know (@PhilLAL could confirm) the camera is on Gbit network (the cameras are connected on the same network). From what I see, I could save images using LimaCCDs much faster (more than 10 Hz for sure). The problem comes from the LiveViewer which seems very slow (0.1 Hz for updating the images ...) I am wondering if there is some problems in the Ds configuration ? Or may be the method is not the good one : do people use the LiveViewer or are they using LimaCCd, getting images, and showing them directly ?

PhilLAL commented 3 years ago

Hi, I confirm we use Gbit ethernet.

laurent-claustre commented 3 years ago

Hi, do you read the image via JpegImage or Image attribute. If you use Jpeg it can explain the slow reading.

hayg25 commented 3 years ago

Hi, How do you switch from JpegImage to Image Attribute ? I do basically nothing special in Jive ...

laurent-claustre commented 3 years ago

huum, you have 2 differents attributes called "JPegImage and Image, I do not know how you are using Jive?? Are you displaying the image only using Jive? and if yes are you running Jive on a different computer where you are running the device server?

hayg25 commented 3 years ago

Hi @laurent-claustre, sorry I answered too fast. Indeed I am using the Image attribute. And the slow update is on that attribute. I display them through Jive interface and also using Taurus Image. Not only Image attribute that updates each 10s. Image counter attribute also updates every 10 s (for example at 10 Hz that number increases by 100 each 10s ). So I suspect there is an other Attribute that manages the update frequency ...

laurent-claustre commented 3 years ago

Hi, I did some test with LiveViewer and a camera (maxipix) 516x516 pixels running at 300Hz and using jive and monitor tool(atkmoni) to display attributes and images. I get a "normal" refresh time with a image every 1sec. So nothing seems wrong on the LiveViewer side. Of course for you transferring a image 2000x2000 can delay a lot with jive running on an other computer. Did you try to run jive on the computer where you have the camera connected, @PhilLAL ? In the meantime, I will test an application we called LimaGUI which can help you for live display, But I've to figure out some issues with conda before making a package available on anaconda.org.

hayg25 commented 3 years ago

Hi, don't have yet news from @PhilLAL concerning the tests on the server directly, but I have made some tests with smaller camera (Scout, about 500x500). And for that camera, I see normal refresh time with image every 1s as well. So we need to investigate more on the ACE camera ... or stick to another method to display images ... May be by using LimaCCDs (limaccd.last_image_ready). Is there a way to play with your LimaGUI package ?

hayg25 commented 3 years ago

Hi @laurent-claustre , so @PhilLAL has also made some tests directly on camera server and he see the same image rate (0.1 Hz) on video. Image saving rate is ok for me. Is there a way to use your tool LimaGUI to display our images ? Or may be you have another strategy to display images ? Thanks again

hayg25 commented 3 years ago

Dear @laurent-claustre, how could we improve our data taking, I mean the live video display ? Since the Liveviewer is too slow for us, what do you think about a solution based on LimaCCds which is fast enough for us. Could we for example combine your Lima-GUI tool with LimaCCds ?

laurent-claustre commented 3 years ago

we are fixing the some last issues with limagui to be used outside ESRF. limagui is about to connect to any LimaCCDs server running on your network and it will make you able to display images as fast as possible. But keep in mind that limagui is a test tool we will provide to test and demonstrate how lima works, it is written in python and the code can give you some clues on how to develop your own live display tool.

laurent-claustre commented 3 years ago

We still have some issues wit limagui I propose you use instead the web application LimaBpmWebServer, available from anaconda.org esrf-bcu channel (like lima), but you should have a recent enough lima tango server 1.9.x is required.

Go to this page for the installation instruction for a conda installation: https://gitlab.esrf.fr/limagroup/Lima-bpm-web-server You must first create a new conda environment and just install the package:

conda create -n lima-bpm-web-server conda activate lima-bpm-web-server conda install lima-bpm-web-server -c conda-forge -c esrf-bcu -c tango-controls

The web server, base on your TANGO_HOST env. variable, can discover all your lima camera, to run the server just type: LimaBpmWebServer

Then, using your favorite web browser, use the URL :8066

The web application is very simple to use, and powerful enough to display fast the images. Next week we will make limagui available on anaconda esrf-bcu channel, and you will have choice . Cheers

hayg25 commented 3 years ago

Dear @laurent-claustre , many thanks for your reply and indeed I will wait to also test your limagui code when available. I have tested your LimaBpmWebServer and it seems very good and functional. I have some issues I cannot solve : 

BPM Viewer running on http://client3:8066
No whitelist loaded, allowing all cameras
Subscribing to bvdata push event...
Traceback (most recent call last):
  File "/tmp_mnt/data/hayg.guler/miniconda3/envs/lima-bpm-web-server/bin/bottle.py", line 868, in _handle
    return route.call(**args)
  File "/tmp_mnt/data/hayg.guler/miniconda3/envs/lima-bpm-web-server/bin/bottle.py", line 1748, in wrapper
    rv = callback(*a, **ka)
  File "/data/hayg.guler/miniconda3/envs/lima-bpm-web-server/lib/python3.9/site-packages/LimaBpmWebServer/webserver.py", line 229, in getstatus
    "min_max": self.cameras_running[camera][1].min_max.tolist(),
  File "/data/hayg.guler/miniconda3/envs/lima-bpm-web-server/lib/python3.9/site-packages/tango/device_proxy.py", line 355, in __DeviceProxy__getattr
    six.raise_from(AttributeError(name), cause)
  File "<string>", line 3, in raise_from
AttributeError: min_max
No bvdata value returning
No bvdata value.value returning

and on the web browser I have the following message : 

Cant get camera status, check server console for more details

Thanks again, cheers

laurent-claustre commented 3 years ago

H

Dear @laurent-claustre , many thanks for your reply and indeed I will wait to also test your limagui code when available. I have tested your LimaBpmWebServer and it seems very good and functional. I have some issues I cannot solve :

BPM Viewer running on http://client3:8066
No whitelist loaded, allowing all cameras
Subscribing to bvdata push event...
Traceback (most recent call last):
  File "/tmp_mnt/data/hayg.guler/miniconda3/envs/lima-bpm-web-server/bin/bottle.py", line 868, in _handle
    return route.call(**args)
  File "/tmp_mnt/data/hayg.guler/miniconda3/envs/lima-bpm-web-server/bin/bottle.py", line 1748, in wrapper
    rv = callback(*a, **ka)
  File "/data/hayg.guler/miniconda3/envs/lima-bpm-web-server/lib/python3.9/site-packages/LimaBpmWebServer/webserver.py", line 229, in getstatus
    "min_max": self.cameras_running[camera][1].min_max.tolist(),
  File "/data/hayg.guler/miniconda3/envs/lima-bpm-web-server/lib/python3.9/site-packages/tango/device_proxy.py", line 355, in __DeviceProxy__getattr
    six.raise_from(AttributeError(name), cause)
  File "<string>", line 3, in raise_from
AttributeError: min_max
No bvdata value returning
No bvdata value.value returning

and on the web browser I have the following message :

Cant get camera status, check server console for more details

Thanks again, cheers

Hi, which version of the LimaCCDs server are you using? You should have >= 1.9.4, the latest is 1.9.12

PhilLAL commented 3 years ago

Hi @laurent-claustre thank you for the help, here are the installed versions:

lima-camera-basler        1.9.2            py37ha8d69ae_0    esrf-bcu
lima-camera-basler-tango  1.9.2                         0    esrf-bcu
lima-camera-simulator     1.9.0            py37ha8d69ae_0    esrf-bcu
lima-camera-simulator-tango 1.9.0                         0    esrf-bcu
lima-core                 1.9.1            py37ha8d69ae_0    esrf-bcu
lima-tango-server         1.9.2                         0    esrf-bcu
laurent-claustre commented 3 years ago

Hi, keep you current environment how it is, and create a new one with new list of channel order, since lima-core 1.9.8 conda-forge must be the main one. conda create -n conda activate conda config --env --add channels conda-forge conda config --env --append channels default conda config --env --append channels esrf-bcu conda config --env --append channels tango-controls conda install lima-camera-basler-tango

Et voila !!! you will get a new env with all the latest conda packages. Then bpm-web-server will be happy and same thing for limagui (soon)

laurent-claustre commented 3 years ago

Hi guys,

you can try the limagui PyQt application. For installation (under conda) please have a look here: http://gitlab.esrf.fr/limagroup/limagui

To start limagui on your camera just type: limagui tango

Cheers, Laurent

PhilLAL commented 3 years ago

Hi @laurent-claustre I notice that installation of lima with the new miniconda version 3.8 and 3.9 failed, lima seems to need miniconda 3.7 Do you think some note should be added on the doc?

Cheers

hayg25 commented 3 years ago

Hi @laurent-claustre and @PhilLAL, limagui installation needed few more packages than written in the doc : 

pip  install python-dateutil --user
pip install silx --user

and of course pyqt (4 or 5) And during execution I had some issued : 

~/.local/bin/limagui tango LimaCCDs/ccdsst1-li
#### Adding camera type=LimaCCDs, args=odict_values(['LimaCCDs/ccdsst1-li']).
Lima plugin: Trying to import ..lima_io.camera.limaccds camera specific module.
Trying to import ..lima_io.camera.limaccds module.
Failed to import camera module: No module named 'limagui.applications.limagui.lima.lima_io.camera.limaccds').
Looking for LimaCCDs (srv_name=LimaCCDs/ccdsst1-li)
LimaCCDs device found.
Camera type: Basler
Looking for BpmDeviceServer device (srv_name=LimaCCDs/ccdsst1-li)
Succes. Found a BpmDeviceServer!
Looking for Basler device (srv_name=LimaCCDs/ccdsst1-li)
Ok. Found the camera device!
Loading BpmDeviceServer io classes.
Lima plugin: Trying to import ..tango_io.camera.basler camera TANGO specific module.
Trying to import ..tango_io.camera.basler module.
Failed to import camera TANGO module: No module named 'limagui.applications.limagui.limatango.tango_io.camera.basler').
Traceback (most recent call last):
  File "/data/hayg.guler/.local/bin/limagui", line 8, in <module>
    sys.exit(limagui_main())
  File "/data/hayg.guler/.local/lib/python3.7/site-packages/limagui/applications/limagui/app.py", line 219, in limagui_main
    model = win.addCamera(name, camera_type, *camera_args)
  File "/data/hayg.guler/.local/lib/python3.7/site-packages/limagui/applications/limagui/limagui/widgets/mainwindow.py", line 249, in addCamera
    subWindow = self._createSubWindow(model)
  File "/data/hayg.guler/.local/lib/python3.7/site-packages/limagui/applications/limagui/limagui/widgets/mainwindow.py", line 277, in _createSubWindow
    return self.SubWindowClass(model, *args, **kwargs)
  File "/data/hayg.guler/.local/lib/python3.7/site-packages/limagui/applications/limagui/limagui/widgets/mainwindow.py", line 172, in __init__
    self._plot = plot = LimaImageView(model)
  File "/data/hayg.guler/.local/lib/python3.7/site-packages/limagui/applications/limagui/limagui/widgets/limaimageview.py", line 60, in __init__
    pw = self._pw = Plot2D()
  File "/data/hayg.guler/miniconda3/envs/lima_tango/lib/python3.7/site-packages/silx/gui/plot/PlotWindow.py", line 885, in __init__
    roi=False, mask=True)
  File "/data/hayg.guler/miniconda3/envs/lima_tango/lib/python3.7/site-packages/silx/gui/plot/PlotWindow.py", line 114, in __init__
    super(PlotWindow, self).__init__(parent=parent, backend=backend)
  File "/data/hayg.guler/miniconda3/envs/lima_tango/lib/python3.7/site-packages/silx/gui/plot/PlotWidget.py", line 392, in __init__
    self._backend = self.__getBackendClass(backend)(self, self)
  File "/data/hayg.guler/miniconda3/envs/lima_tango/lib/python3.7/site-packages/silx/gui/plot/PlotWidget.py", line 537, in __getBackendClass
    raise RuntimeError("None of the request backends are available")
RuntimeError: None of the request backends are available

thanks

laurent-claustre commented 3 years ago

Hi @laurent-claustre I notice that installation of lima with the new miniconda version 3.8 and 3.9 failed, lima seems to need miniconda 3.7 Do you think some note should be added on the doc?

Cheers

Salut Philippe, normally not. any miniconda version 3 can be used to install lima. lima by dependency will request python 3.7, so if you just install for instance lima-camera-baser-tango, you will get python3.7 installed and all the other necessary packages.

What's your error messages?

laurent-claustre commented 3 years ago

~/.local/bin/limagui tango LimaCCDs/ccdsst1-li

Hi Hayg,

Thanks for the feedback and sorry for the mistake in the documentation I will fix it. For silx it exist a conda package as well. When you are under conda it is recommended to install conda packages when it is possible.

For your error, I will investigate with my colleague Damien who wrote the application.

dnaudet commented 3 years ago

Hi, this looks like a silx error. I think the default backend is matlplotlib. Do you have matplotlib installed? It should have been pulled by silx, but who knows...

Can you give this a try in a python console please:

from silx import sx
p = sx.Plot2D()
p.show()
laurent-claustre commented 3 years ago

Hi, this looks like a silx error. I think the default backend is matlplotlib. Do you have matplotlib installed? It should have been pulled by silx, but who knows...

Can you give this a try in a python console please:

from silx import sx
p = sx.Plot2D()
p.show()

humm, @hayg25 did install the pip version of silx. I suppose he should move to the conda version?

dnaudet commented 3 years ago

I suppose he should move to the conda version? It's worth a try I guess!

hayg25 commented 3 years ago

Works :) thanks, so indeed conda package was needed.

hayg25 commented 3 years ago

And indeed it improved the acquisition speed. From jive, my images were updating each 10s and using lima Gui it goes much faster (each second or less). I don't understand how lima Gui makes it better :1st_place_medal:

thanks again I will investigate more with different cameras we have.

PhilLAL commented 2 years ago

Hi, by any chance, is there a way to define the update frequency of the image seen live as it can be done for acquisition? Regards, and happy new year