search

icecube / skymap_scanner

A distributed system that performs a likelihood scan of event directions for IceCube real-time alerts using CPU cluster(s) and queue-based message passing.
5 stars 2 forks source link

Cleanup of GCD handling code #125

Closed mlincett closed 1 year ago

mlincett commented 1 year ago

The GCD handling logic in utils.extract_json_message is currently a bit articulated and it can likely be simplified.

When I tested with GFU realtime events, the input packet would always hit the empty GCD case, mistakenly logged as:

old EHE packet with **empty GCD frames**. need to replace all geometry but is actually the current default (to my knowledge).

In this situation the function looks up a GCD file based on the run number and adds the GCD frames to the frame packet. I am not sure whether or not HESE and EHE realtime events have the same type of payload (but I guess I can verify it).

For the processing of offline events (generated from .i3 files), I think the input file is always expected to carry the GCD frames.

mlincett commented 1 year ago

Looking at: https://github.com/icecube/skymap_scanner/blob/581f48919e46c4659234ed0cc55e259e8f012fc2/skymap_scanner/config.py#L17-L26 for the time being I believe the Skymap Scanner needs one and one only authoritative source for GCD files, if any. I am leaning towards getting rid of all this fallback logic.

See also https://github.com/icecube/realtime/issues/8

dsschult commented 1 year ago

There is also 

/cvmfs/icecube.opensciencegrid.org/users/RealTime/GCD/PoleBaseGCDs/

which could be a source.

Note that these should already be in the container at /opt/i3-data/baseline_gcds via this icetray build step: https://github.com/WIPACrepo/docker-icecube-icetray/blob/master/install_icetray/Dockerfile_prod#L27

mlincett commented 1 year ago

Note that these should already be in the container at /opt/i3-data/baseline_gcds via this icetray build step: https://github.com/WIPACrepo/docker-icecube-icetray/blob/master/install_icetray/Dockerfile_prod#L27

Yes, I noticed, that is why a lot of legacy code is largely unnecessary now (but even if it wasn't there's room for simplification).

Also I expect that if we ensure that the input JSON is a fully-formed event (the GCD baseline/diff merge can be done upstream, since it is realtime-specific) the Skymap Scanner may not need to carry/access baseline GCDs at all.

dsschult commented 1 year ago

Is there any scanning being done on previous alerts that currently feeds in the diff GCD? Would you be willing to reformat that to only supply the full GCD? Or have a step before submitting to the scanner? That's about the only case I'm unsure of.

mlincett commented 1 year ago

Is there any scanning being done on previous alerts that currently feeds in the diff GCD? Would you be willing to reformat that to only supply the full GCD? Or have a step before submitting to the scanner? That's about the only case I'm unsure of.

That would be the plan, yes.

mlincett commented 1 year ago

With PR #147 I try to simplify the handling of GCD information.

The GCD diff/uncompress logic is still in place. It seems possible to decouple it by letting prepare_frames store the full GCD and avoid the uncompress segment in reco_icetray. This is better addressed in a separate PR.

mlincett commented 1 year ago

Tentatively closing this after merge of #149

Moving all the GCD uncompress logic to the server side will be addressed in a new issue.