perolavsvendsen
commented
2 years ago Off-line discussions summary:
Two-step generation of pre-processed data:
Step 1: In the template, script runs, exports "staged" data with partial metadata defined. Export to a suitable area, suggested: <template>/share/preprocessed. fmu-dataio typically runs from RMS in this context. Metadata shall contains a selection of blocks, specifically data but possibly also others, such as fmu.model perhaps.
Step 2: In a PRE_SIMULATION hook workflow, script runs, fetches the "staged" data and a) completes the metadata, and b) dumps to correct place on /scratch (<case>/share/XX).
In step 2a, case-specific parts of metadata must be inserted, e.g. fmu.case and others. tracklog must be appended.
Also possible to do sanity-checks against global_variables in this process, particularly against masterdata elements etc.
sago64
This has been on the block for a while, and is addressed also in other issues. Following the refactoring into 1.0.0, it is time to properly describe the requirements and start developing on this context.
Pre-processed data are data that are made up-front, prior to a case. Data are typically made directly on the template project, and stored with the template project. When a case is initialized, these data are ferried to the case structure usually by a HOOK workflow or similar.
Example Seismic observations represented by a seismic amplitude map (surface) are pre-made by a manual workflow in RMS. The results are stored with the template project (/share/observations). Responsibility fmu-dataio: Export template data with incomplete metadata (missing the
fmublock). When an FMU case is initiated, fetch these surfaces with a workflow job, complete the metadata (add thefmublock), store to correct location in the case structure.(edited post offline discussion aug 25 2022)