Input data format and validation

[mlincett]

In skymist, I am experimenting pydantic templates for the parsing of realtime messages.

It could be nice to have some form of specification of the input data format of skymap_scanner to be be defined in skywriter.

Whether this should be based on pydantic or something else is TBD.

[mlincett]

Looking at i3_to_json.py it seems the current format is quite cluttered from realtime-related metadata.

I understand we chose to stick with the "standard" realtime JSON format to avoid reinventing the wheel, but I think the sky* projects are now mature enough to dictate their own rules.

I suggest to:

• [ ] track down which content is actually used in skymap scanner.
• [ ] better document the current i3_to_json code and the handling of simulated data
• [ ] define a subset of the current format for the purpose of skywriter / skymap scanner.

[ric-evans]

Sounds like a reasonable need to me. The way the event is compressed is my favorite part of the realtime format--not because it's better than an alternative, but because it's been standardized. I'm not really familiar with metadata

@tianluyuan may have some ideas

[mlincett]

Sounds like a reasonable need to me. The way the event is compressed is my favorite part of the realtime format--not because it's better than an alternative, but because it's been standardized. I'm not really familiar with metadata

@tianluyuan may have some ideas

Definitely the I3Frame encoding would follow the same standard and possibly use the same IceTray code (it just needs to be factored into more atomic functions).

To me it's mostly a matter of removing unnecessary data from the "envelope".

[mlincett]

Looking at the following:

• https://github.com/icecube/skymap_scanner/blob/main/skymap_scanner/utils/extract_json_message.py
• https://github.com/icecube/icetray/blob/main/full_event_followup/python/frame_packet_to_i3live_json.py

It seems that all that is used by the scanner is the "frame packet" (list[I3Frame]).

The information in the "envelope" was used to evaluate the alert passing criteria, but this is no longer the case for the Skymap Scanner.

[mlincett]

Related: https://github.com/icecube/icetray/issues/3132