Closed thespacedoctor closed 1 year ago
Data organiser is now 50% written. I found much of the functionality was needed sooner than later when preparing unit-test data suites.
As other recipes are written I will mature the organiser.
The main goals of the DO are:
soxspipe prep
Actions:
sof_map
table would link raw_frames (raw_frame_id) with product (parent_product_frame_id)sof_map
table would also link input product_frames (child_product_key) with output product frames (parent_product_frame_id)The test data can be found here:
~/Dropbox/server-sync/soxs-eso-data/xshooter-pipeline-data/unittest_data/xsh/data-organiser
beta version of data-organiser has been released
As shown at ESO by the pipeline team the data organisation (raw data into 'sets of files' or sof files) can be done with REFLEX. It was suggested that REFLEX could be run from the command-line in an automated fashion to make use of this functionality. While this means we would not have to write our own data-organiser recipe, we would again have reflex as a dependency of the final pipeline. I don't think we want this.
Most of the logic behind the REFLEX organiser can be found (and duplicated) from this paper:
Automated data reduction workflows for astronomy The ESO Reflex environment
This issue will be used to track the planning of the data-organiser.
TASKS