Definition of an observation, better if there is already an example of the real system that will be used, otherwise this can be a simple text file
Run the IDL simulator and produce the raw readouts, the raw science frame and the reduced science frame with all the necessary keywords
Offline pipeline
First run the offline pipeline, i.e. take the raw readouts FITS files, run stpipe with data processing steps (the iris_readout steps, also called ROP) adapted from JWST, for example https://github.com/spacetelescope/jwst/tree/master/jwst/ramp_fitting, and produce raw science frames and compare with simulator. is there any metadata/telemetry required to create the science frames which is NOT in the headers of the raw readouts? I think the answer is yes, how do we get that metadata? This part is performed by Python classes in iris_pipeline that run the C library iris_readout, we can also prototype in Python.
Then run second part of the offline pipeline, take the raw science frame, calibrate with local fits files and output the reduced science frame, compare with the simulator
Realtime pipeline
Create a FakeDetectorHCD that reads the observation defined by the user and the FITS files produced by the simulator from disk and then serves that to the pipeline in realtime at specific intervals, the interface is in-memory. This can also be a single object, not necessarily a separate process.
Create a FakeTelemetryServer that read the observation defined by the user and simulates some reasonable telemetry that should be coming from other subsystems (it can accept other parameters to configure its outputs). This can also even just be a single object, no need to be a separate server.
Once the modules based on iris_readout get those data, they also access the telemetry service to get the required metadata and they produce raw readout FITS files.
Once they have got all the readouts they get more telemetry which is needed for producing the raw science frame, produce it and write it to FITS file.
In real-time the Data Reduction system, a separate process, monitors the filesystem, gets the raw readouts (and optionally other telemetry) and produces the reduced science frame.
Offline pipeline
stpipe
with data processing steps (theiris_readout
steps, also called ROP) adapted from JWST, for example https://github.com/spacetelescope/jwst/tree/master/jwst/ramp_fitting, and produce raw science frames and compare with simulator. is there any metadata/telemetry required to create the science frames which is NOT in the headers of the raw readouts? I think the answer is yes, how do we get that metadata? This part is performed by Python classes iniris_pipeline
that run the C libraryiris_readout
, we can also prototype in Python.Realtime pipeline
FakeDetectorHCD
that reads the observation defined by the user and the FITS files produced by the simulator from disk and then serves that to the pipeline in realtime at specific intervals, the interface is in-memory. This can also be a single object, not necessarily a separate process.FakeTelemetryServer
that read the observation defined by the user and simulates some reasonable telemetry that should be coming from other subsystems (it can accept other parameters to configure its outputs). This can also even just be a single object, no need to be a separate server.iris_readout
get those data, they also access the telemetry service to get the required metadata and they produce raw readout FITS files.