search

leginon-org / leginon-redmine-archive

1 stars 0 forks source link

Helios integration with Appion pipeline #5464

Open leginonbot opened 5 months ago

leginonbot commented 5 months ago

Author Name: Carl Negro (@carl9384) Original Redmine Issue: 5464, https://emg.nysbc.org/redmine/issues/5464 Original Date: 2018-01-24

Using the DSynchronize program installed on the Helios Microscope PC, we are able to synchronize the "TiffImages" folder with the mounted data drive on the support PC. I will install Python2.7 on the Microsope and Support PC's in the near future so we can write a more sophisticated transfer script.

NOTE: Using the auto-synchronize utility in DSynchronize, if you delete a file from the monitored Microscope directory, it will delete it from the monitored Support PC directory. Be careful!

leginonbot commented 5 months ago

Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2018-01-25T00:30:12Z

I installed Python2.7 on both the Helios microscope and support PC's. There is a file monitoring library called watchdog (https://pypi.python.org/pypi/watchdog) that I also installed with pip, that we may be able to use to find changes in the microscope data storage directories.

https://www.michaelcho.me/article/using-pythons-watchdog-to-monitor-changes-to-a-directory

I moved the following packages over to the microscope PC with a USB drive and installed them individually:

argh==0.26.2 pathtools==0.1.2 PyYAML==3.12 watchdog==0.8.3

leginonbot commented 5 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2018-02-13T23:01:09Z

Created 2 new scripts on node01 for automatically uploading Helios images from support PC. uploadSEM.sh finds upload.cfg files in /mnt/SharedData (mounted from support PC) modified within last 30 days. It then calls SEM_upload.py that parses upload.cfg, uploads images and stores session name. I've just runs uploadSEM.sh, it has found /mnt/SharedData/vgarcia/vgarcia_support20180105T170429/TiffImages/upload.cfg that Ashleigh created and uploaded images to 18feb13r13 session.

We can discuss how often to call this script. If we run it too often (i.e. more than one time per hour), it would use hard drive on support PC more often which can lead to a premature failure of the drive on support PC. Maybe we can call it once an hour or once a day.

leginonbot commented 5 months ago

Original Redmine Comment Author Name: Bridget Carragher (@bcarr-czi) Original Date: 2018-02-14T01:30:54Z

Great to see this moving forward! If we need to limit the number of times this launches it would be good to be able to have Ash or Bill or whoever is collecting the data force a push on demand so that they can tell a remote end user to look at the data.

leginonbot commented 5 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2018-02-26T22:16:46Z

Added cron job on node01 to run uploadSEM.sh every hour: @0 flock -n /root/.upload /root/bin/uploadSEM.sh@

leginonbot commented 5 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2018-03-19T22:47:19Z

Added a new entry InstrumentData table called SEM. Also, created corresponding entry in ScopeEMData table. Committed commit:d39fa796. Starting tomorrow all images uploaded for SEM will have this ScopeEMData and can be easily found in image viewer.