simspec files do not have a well-defined data model

[weaverba137]

The existing data model for simspec files does not reflect the actual simspec files produced in recent reference runs. Depending on the flavor of the exposure, simspec files

• Have sets of header keywords that differ from the data model.
• May or may not have EXTNAME set for all HDUs.
• May have different number of HDUs!

This last point makes it especially difficult to validate simspec files against the data model.

[sbailey]

Indeed, I can see how this can be a pain for datamodel validation. The current state requires different HDUs depending upon the FLAVOR in the FIBERMAP HDU:

Required by all flavors ('arc', 'flat', 'science'):

• FIBERMAP, WAVE[BRZ], PHOT[BRZ]

Additional HDUs required by FLAVOR='flat' and 'science':

• WAVE, FLUX

Additional HDUs required only by FLAVOR='science':

• SKYFLUX, SKYPHOT_[BRZ], TRUTH, OBSCONDITIONS

Additionally, as of desisim 0.26.0,

• FLAVOR='flat' incorrectly have meaningless SKYFLUX and SKYPHOT_[BRZ] HDUs
• FLAVOR='arc' has a blank primary HDU with no EXTNAME while the others do not

For the purposes of the 18.3 release, let's document this by hand in the datamodel and recognize that they will fail automated validation, and consider automation options for a later release.

[sbailey]

Thinking "aloud" about options:

1. Only include the HDUs that are required / meaningful for a given flavor, and handle differences on the data model validation side.
   • This is straighforward for humans reading a data model ("this HDU is only required for flavors 'flat' and 'science' but not 'arc'...") but would be a pain for automated tools.
   • a partially automated solution would be to support optional HDUs: if they exist they should follow this data model, but don't worry if they don't exist, and don't try to map which HDUs should exist for which FLAVORs.
2. Require all flavors to include all HDUs with at least dummy values, and then define in the docs which ones are actually used based upon FLAVOR.
   • this is most friendly to the automated tools, but least friendly to a human looking at a simspec file since it is ambiguous which headers are actually used without separately reading the datamodel docs.
3. Split them out into completely different names and formats for arc vs. flat vs. science simspec files.

[sbailey]

Punting to 18.6