Virtual environment for p2120

Calpas commented 5 years ago

Dear experts,

I created a virtual environment using (1) for p2120 for runs 96, 97 and 98.
when I ran (2) for runs 96 and 97, I got the same error message (3). For run 98 it ran fine.
I wonder: should I worry for these error messages? is this make_cxi.py the right version to use?

Best regards

(1) /gpfs/exfel/sw/software/hdf5-virtualise/make_cxi.py

(2) /gpfs/exfel/sw/software/hdf5-virtualise/make_cxi.py /gpfs/exfel/d/proc/SPB/201801/p002120/r0096 -o r96.cxi

(3) Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD13-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD02-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD12-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD08-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD06-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD00-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD04-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD14-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD09-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD11-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD03-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD07-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD01-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD10-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD05-S00020.h5 (error was: 'Unable to open object (component not found)') Skipping file /gpfs/exfel/d/proc/SPB/201801/p002120/r0096/CORR-R0096-AGIPD15-S00020.h5 (error was: 'Unable to open object (component not found)') INFO:karabo_data.components:Found 4998 trains with data for at least 9 modules INFO:karabo_data.write_cxi:128 frames per train, 639744 frames in total INFO:karabo_data.write_cxi:Identified calibrated data INFO:karabo_data.write_cxi:Virtual data shape: (639744, 16, 512, 128) INFO:karabo_data.write_cxi:Assembled 320 chunks for image.data, filling 100.00% of the hyperslab INFO:karabo_data.write_cxi:Assembled 320 chunks for image.gain, filling 100.00% of the hyperslab INFO:karabo_data.write_cxi:Assembled 320 chunks for image.mask, filling 100.00% of the hyperslab INFO:karabo_data.write_cxi:Assembled 320 chunks for image.cellId, filling 100.00% of the hyperslab INFO:karabo_data.write_cxi:Writing to r96.cxi INFO:karabo_data.write_cxi:Finished writing virtual CXI file

takluyver commented 5 years ago

Hello,

To clarify, what you're creating is a "virtual CXI file", or to be more precise, a "CXI file containing virtual datasets". A virtual environment is something totally different. I'm highlighting this because if you search for information about virtual environments, the results won't help you with this. :wink:

The script you're using should still work and point to the latest code.

The errors are telling you that it couldn't open some files in the run directory. Looking at the filenames, they all end with S00020, so they're the 21st sequence file (counting from 0) for each AGIPD module. This probably means that something went wrong with writing files at the end of that run. It's still found plenty of data - 4998 trains means 500 seconds recording time, which is over 8 minutes - and it's filled 100% of the frames in that, so you can probably ignore those errors and go ahead with analysing it.

Calpas commented 5 years ago

Dear Takluyver,

ok, thank you.
after I created 'virtual CXI file' for p2120, I run 'indexamajig' with crystfel. When I run 'hdfsee' on this output to see the images, it need like 1 minute to open each of them. Is it normal? nb: I did not have this delay with the publish ID_83 on CXIDB.

regards

takluyver commented 5 years ago

If you view a few images, then quit and start from the beginning again, is it faster?

We've found that the GPFS filesystem (all paths beginning with /gpfs) can be very slow to start reading a file, probably because it's transferring lots of data into the computer you're working on. If this is the problem, then trying to read the data again while it's still cached locally is much faster. I think you were working with CXIDB data in /beegfs, which may work differently.

Unfortunately, if this is the problem, we can't really do much about it. We don't control the filesystems on Maxwell, and I don't know how to tune GPFS even if we did.

Calpas commented 5 years ago

Dear Takluyver,

Ok, thank you.
For the "CXI file containing virtual datasets", after I created them from real files in a directory, if then I delete this directory, will I still be able to read the 'real data' via the 'virtual data'?
Logically the answer is not, but if so, are 'virtual data' just an interface btw e.xfel and crystfel?

Regards

takluyver commented 5 years ago

The virtual dataset contains links to the real data, so you can only read it if the original files are still present at the same path. You're right that it's an interface, presenting the EuXFEL data in a format that CrystFEL can use.

Calpas commented 5 years ago

Dear Takluyver, ok, thank you for your supports. Regards

takluyver commented 5 years ago

No problem!

European-XFEL / karabo_data

Virtual environment for p2120 #217