icecube / skymap_scanner

A distributed system that performs a likelihood scan of event directions for IceCube real-time alerts using CPU cluster(s) and queue-based message passing.
5 stars 2 forks source link

Scan preparation: event dictionary dumped to JSON and back to dictionary again #32

Closed mlincett closed 1 year ago

mlincett commented 2 years ago

The extract_json_message() function that is run in the preparation of an event for the scan calls the following function:

frame_packet = full_event_followup.i3live_json_to_frame_packet(json.dumps(json_data), pnf_framing=True)

  1. what is the virtue of converting to JSON the event we received from ZMQ?
  2. is this operation supposed to mutate the data format / frame arrangement?
  3. where is the JSON format used otherwise?
mlincett commented 2 years ago

Giving a look at the source of i3live_json_to_frame_packet, the only non-trivial operation that is performed is the zlib decompression.

The rest of the code is fairly redundant with what the skymap scanner code already does to check the integrity of the packet.

ric-evans commented 1 year ago

realtime's json format is a standard, so let's preserve it despite its trivialness. I agree the json.dumps() + json.loads() is inefficient, but I'd argue i3live_json_to_frame_packet() should be able to take an object as input.

I'm closing until we see empirical evidence that this is something that needs to be optimized.