Process new-style GCN and IGWN alerts

[martinjohndyer]

First, GCNs were published using the VOEvent schema. This was based on XML, and frankly it was a pain to work with (non-unique dictionary keys!!). We use the Python voevent-parse package (https://github.com/timstaley/voevent-parse), but they are still annoying and I'll be glad when they're gone.

NASA have been slowly working on a new schema for GCNs, based on JSON. This is their Unified Schema (https://gcn.nasa.gov/docs/notices/schema), and it is an improvement. They are getting rid of the IVORNs, which makes each notice a bit harder to refer to, but we can reconstruct them (we use the IVORNs as the unique key in the database for instance).

LIGO/Virgo/Kagra/IGWN already made a similar change for their Kafka alerts (https://emfollow.docs.ligo.org/userguide/content.html#kafka-notice-gcn-scimma), they have their own schema with examples here: https://emfollow.docs.ligo.org/userguide/content.html#kafka. Notably the alerts include the skymap data embedded in binary format, which will save having to download it separately.

These new JSON-based alerts shouldn't be confused for the JSON VOEvents that SCIMMA's Hop client produces (see #81). Those are just the old XML events converted into JSON. Here's an example:

XML VOEvent

<?xml version='1.0' encoding='UTF-8'?>
<voe:VOEvent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:voe="http://www.ivoa.net/xml/VOEvent/v2.0"
    xsi:schemaLocation="http://www.ivoa.net/xml/VOEvent/v2.0 http://www.ivoa.net/xml/VOEvent/VOEvent-v2.0.xsd"
    version="2.0"
    role="test"
    ivorn="ivo://gwnet/LVC#MS181101ab-3-Initial">
    <Who>
        <Date>2018-11-01T22:36:25Z</Date>
        <Author>
            <contactName>LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration</contactName>
        </Author>
    </Who>
    <What>
        <Param dataType="int" name="Packet_Type" value="151">
            <Description>The Notice Type number is assigned/used within GCN, eg type=151 is an LVC_INITIAL notice</Description>
        </Param>
        <Param dataType="int" name="internal" value="0">
            <Description>Indicates whether this event should be distributed to LSC/Virgo/KAGRA members only</Description>
        </Param>
        <Param dataType="int" name="Pkt_Ser_Num" value="3">
            <Description>A number that increments by 1 each time a new revision is issued for this event</Description>
        </Param>
...

And so on. It's old and bulky, with repeated tags and needless discriptions.

SCIMMA JSON VOEvents

{
    "ivorn": "ivo://gwnet/LVC#MS181101ab-3-Initial",
    "role": "test",
    "version": "2.0",
    "Who": {
        "Date": "2018-11-01T22:36:25Z",
        "Author": {
            "contactName": "LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration"
        }
    },
    "What": {
        "Param": [
            {
                "dataType": "int",
                "name": "Packet_Type",
                "value": "151",
                "Description": "The Notice Type number is assigned/used within GCN, eg type=151 is an LVC_INITIAL notice"
            },
            {
                "dataType": "int",
                "name": "internal",
                "value": "0",
                "Description": "Indicates whether this event should be distributed to LSC/Virgo/KAGRA members only"
            },
            {
                "dataType": "int",
                "name": "Pkt_Ser_Num",
                "value": "3",
                "Description": "A number that increments by 1 each time a new revision is issued for this event"
            },
...

Although it doesn't look like it saves much vertical space here, the move away from XML does make things a lot easier (e.g. by not needing closing tags). Sadly it still has to conform to the VOEvent schema, with the "Who", "What", "WhereWhen", "How", "Why" structure. Most annoyingly, VOEvents allow repeated non-unique parameter values. That's why here Params is a list of dictionaries, each with a name value, instead of just a dictionary.

New IGWN JSON alerts

{
    "alert_type": "INITIAL",
    "time_created": "2018-11-01T22:36:25Z",
    "superevent_id": "MS181101ab",
    "urls": {
        "gracedb": "https://example.org/superevents/MS181101ab/view/"
    },
    "event": {
        "time": "2018-11-01T22:22:46.654Z",
        "far": 9.11069936486e-14,
        "significant": true,
        "instruments": [
            "H1",
            "L1",
            "V1"
        ],
        "group": "CBC",
        "pipeline": "gstlal",
        "search": "MDC",
        "properties": {
            "HasNS": 0.95,
            "HasRemnant": 0.91,
            "HasMassGap": 0.01
        },
        "classification": {
            "BNS": 0.95,
            "NSBH": 0.01,
            "BBH": 0.03,
            "Terrestrial": 0.01
        },
        "duration": null,
        "central_frequency": null,
        "skymap": "U0lNUExFICA9ICAgICAgICAgICAgICAgICAgICBUIC8gY29uZm..."
    },
    "external_coinc": null
}

That's the entire message, minus the massively truncated skymap field. Because they have full control over the schema, IGWN can really cut out a lot of the chaff.

New GCN JSON alerts

{
  "$schema": "https://gcn.nasa.gov/schema/v4.0.0/gcn/notices/einstein_probe/wxt/alert.schema.json",
  "instrument": "WXT",
  "trigger_time": "2024-03-01T21:46:05.13Z",
  "ra": 120,
  "dec": 40,
  "ra_dec_error": 0.02,
  "image_energy_range": [
    0.5,
    4
  ],
  "net_count_rate": 1,
  "image_snr": 1,
  "additional_info": "The net count rate is derived from an accumulated image (up to 20 min) in 0.5-4 keV, assuming a constant flux. However, it can be significantly lower than the actual count rate of a burst with a duration much shorter than 20 min."
}

This is the sample EP alert from https://gcn.nasa.gov/docs/schema/v4.0.0/gcn/notices/einstein_probe/wxt/alert.schema.json. Nice and to-the-point.

---

Now because these new events are not VOEvents, there will have to be some major changes within the gtecs.alert code. Honestly I'd been putting it off, since only really the GW events had moved over and they were still publishing the old-style events in parallel. However now EP is only producing the new events #80 the changes can't really be held off any longer...

[martinjohndyer]

81 does most of the work rewriting the sentinel backend code to allow it to process these new alerts. The priority was being able to handle the EP alerts (#80), but at the same time it wasn't much extra effort to process the IGWN JSON/Avro alerts as well.

However, there are a few issues still to be solved before we could consider actually processing IGWN alerts, which I'll note here:

• IVORNs. The new notices don't include them, that's fine. However I was wrong above, we actually can't rebuild the old format IVORNs from the new alerts. That's because the classic GW alerts produce notices titled ivo://gwnet/LVC#S240723b-1-Preliminary, ivo://gwnet/LVC#S240723b-2-Preliminary, ivo://gwnet/LVC#S240723b-3-Update etc. The key is the increasing notice number, which is not included in the new alerts. In other words we can't tell the difference between the first and second preliminary alert, there's no indicator of which is 1 and which is 2. We could check our own database archive and guess, or we could query GraceDB, but it's a pain. We need an IVORN for the database (#85 ), for the time being I fell back to the timestamp just like EP alerts with no ID (https://github.com/GOTO-OBS/gtecs-alert/issues/80#issuecomment-2245164878). This could work as long as we don't want to run both classic and new alerts in parallel, because then we'd have no way to compare and see if we'd recieved an alert before. Again, the solution would probably be to remove IVORNs from the database entirely and have other values used to compare alerts.
• Saving the alert contents in the database is not good, because it saves the entire embedded skymap in the payload and then again in the skymap field. I think saving the avro blob is probably the wong way to do things as well, instead we should convert to JSON, remove/blank the skymap field and then save that as a string.
• It would also be good to update the GWNotice.from_gracedb() method to allow it to get the newer notices too, I'm not entirel sure how the get_slack_channel() function would work but I suppose it would be best to try to match to the same type as the new alert.