search

European-XFEL / karabo_data

Python tools to read and analyse data from European XFEL
https://karabo-data.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
13 stars 7 forks source link

Only include relevant files from .select() and .deselect() #202

Closed takluyver closed 5 years ago

takluyver commented 5 years ago

This partly addresses #131: it should make it possible to combine proc detector data with raw data, by deselecting the detector sources in the raw data.

takluyver commented 5 years ago

In the older runs, as described in #131, only data files from the fast detectors ended up in proc, so combining raw & proc was useful. In more recent experiments I just found, it looks like the DA files from the data aggregators are also copied to proc, so maybe this isn't so important any more. But maybe those files are being copied as a workaround to make it easier to process the data from a run together.

tmichela commented 5 years ago

Are all DA files copied to proc? Because some detector data (10Hz, like Jungfrau, Epix, ...) are stored in DA files. So running calibration on those detector would copy also the specific DA file.

takluyver commented 5 years ago

I thought about that, but it looks like files with no detectors in are also getting copied. See e.g. /gpfs/exfel/exp/SCS/201921/p002409/proc/r0102

tmichela commented 5 years ago

This seems like the right thing to do anyway, regardless of what file is copied or not... LGTM