APS-2BM-MIC / ipython-user2bmb

ipython configurations for the tomography instrument
2 stars 5 forks source link

getting HDF5 files from grayhound #14

Closed prjemian closed 6 years ago

prjemian commented 6 years ago

The images are saved in HDF5 files on grayhound. The data processing steps run on linux. At present, linux cannot mount the file system with the HDF5 files .

There are problems when mounting CIFS drives hosted on Win 10 and mounted on RHEL7.4 workstations (something to do with the Windows registry). Several possibilities exist to get the image data:

  1. grab each image frame as it is acquired from EPICS by a linux-hosted python process
  2. mount grayhound file system on RHEL7 workstation
  3. write HDF5 files to DSERV filesystem
  4. copy HDF5 files from grayhound to linux-hosted filesystem
prjemian commented 6 years ago

Here are my comments:

  1. This process is completely transparent to current data acquisition. It also has a high risk of missing frames if it is slow. The acquisition process may need to be made slower so this technique does not drop or miss frames.

  2. IT tells us that they have not been successful with this scenario to date

  3. Need to mount DSERV on grayhound. The DSERV is likely much slower than the current local storage. The acquisition process may need to be made slower so this technique does not drop or miss frames.

  4. Requires software installation on Windows. Might be possible using rsync or scp but requires Windows-based tools. Python could do this (with pyepics and paramiko) by watching the area detector HDF5 plugin PVs for file write status and file name. When a new file is detected, python could initiate a file transfer to the linux system of choice. Here is an example of the file transfer from python, emulating scp. This must run on grayhound.

prjemian commented 6 years ago
  1. won't work as we hope -- see fd35392a59
  2. IT overcame the problem. This works: /mnt/WinS/
  3. not tested
  4. can use java fdt tool to copy from Windows to linux - this works

Summary: works good enough for now