Sep 22 firedrill

[tsengj10]

Additional code and data files for running the firedrill of Sep 2022.

fd2209b-data.json is the adapted data packet, adding the action and burst id fields, followed by a report action. Next time, it may be better if have input modules which can just take the packet as given and add these fields.

The json file is piped into snewpdag with the configuration file fd2209.csv. This should result in a skymap showing the calculated localization.

test-fd2209.csv is meant to generate a SN, from the direction which was calculated above, to check the relative timings of the detectors. It doesn't generate a skymap. Instead, run it at least at loglevel INFO to get the neutrino burst times at each detector.

[tsengj10]

Added a JsonAlertInput plugin for reading a json file into the payload, along with the original packet, and configuration file fd2209-direct.csv.

We can use this pattern for reading from the SNEWS server as well, extracting Riccardo's code from app into a stand-alone input plugin.

[mcolomermolla]

I think this can be merged before Thursday firedrill and contribute to it! I actually prefer this alternative way where you pipe the file:

python -m snewpdag --log INFO --jsonlines snewpdag/data/fd2209.csv < snewpdag/data/fd2209b-data.json

So that the config needs no change for news firedrills, you just pipe a different input file from the snews coincidence server.

Concerning this:

Right now it seems it is not possible to pipe directly the original fd_sep_alert.json as input to the fd2209-direct.csv which converts to "the needed json format for snewpdag". And that would be ideally what we want for the firedrills, so that everything that needs to be done is:

snews_pt subscribe --no-firedrill -p "-m snewpdag --log INFO --jsonlines snewpdag/data/fdout-process-direct.csv <" which actually does: os.system(f"python -m snewpdag --log INFO --jsonlines snewpdag/data/fd2209.csv < {saved_json (firedrill output) }")

when an alert is published by snews_pt on the coincidence server

[tsengj10]

Thanks for pointing that out. I'll take a look at snews_pt. This may just be a case of making a snewpdag plugin read stdin to the end of stream.

[mcolomermolla]

Thanks for pointing that out. I'll take a look at snews_pt. This may just be a case of making a snewpdag plugin read stdin to the end of stream.

This is the error when the direct firedrill output json is used as input to snewpdag with the *direct.csv:

(snewpy_env) marta@marta:~/snewpdag-firedrill$ python -m snewpdag --log INFO --jsonlines snewpdag/data/fd2209-direct.csv < snewpdag/data/fd_october_trial.json Traceback (most recent call last): File "/home/marta/miniconda3/envs/snewpy_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/marta/miniconda3/envs/snewpy_env/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/marta/snewpdag-firedrill/snewpdag/main.py", line 5, in app.run() File "/home/marta/snewpdag-firedrill/snewpdag/dag/app.py", line 119, in run data = ast.literal_eval(jsonline) File "/home/marta/miniconda3/envs/snewpy_env/lib/python3.8/ast.py", line 59, in literal_eval node_or_string = parse(node_or_string, mode='eval') File "/home/marta/miniconda3/envs/snewpy_env/lib/python3.8/ast.py", line 47, in parse return compile(source, filename, mode, flags, File "", line 1 { ^ SyntaxError: unexpected EOF while parsing

which one does not get with the alternative command line when you pipe as input the modified json already and not the direct one

[tsengj10]

It's because --jsonlines expects the whole json packet (everything within {}) to come in on one line.

The other easy possibility to code is to read the whole available stdin stream and parse it as one json packet.

If it's not one of those, the code has to parse the input to find the end of the json packet. Which is probably not worth the effort if there are easier ways to handle the json input.

[tsengj10]

Turns out the command-line option --input is supposed to do something like this, but the packet still needs an action and a starting point. The first part is easy (just add an action field). For the second part we can use the first plugin defined, but I have to think about exactly how to do that, since a dictionary is in principle unordered.

[mcolomermolla]

Turns out the command-line option --input is supposed to do something like this, but the packet still needs an action and a starting point. The first part is easy (just add an action field). For the second part we can use the first plugin defined, but I have to think about exactly how to do that, since a dictionary is in principle unordered.

Yes, ideally:

Take original input json from firedrill with --input, and as starting point use JsonAlertInput plugin on that file. (I guess action should just be "alert" right?)

[tsengj10]

snewpdag can now take a firedrill packet and add a default action and injection point from the command line:

python -m snewpdag --log INFO snewpdag/data/fd2209.csv --input snewpdag/data/fd_sep_alert.json

Perhaps one can use "-m snewpdag snewpdag/data/fd2209.csv --input" as an argument for snews_pt subscribe?

[mcolomermolla]

snewpdag can now take a firedrill packet and add a default action and injection point from the command line:

python -m snewpdag --log INFO snewpdag/data/fd2209.csv --input snewpdag/data/fd_sep_alert.json

Perhaps one can use "-m snewpdag snewpdag/data/fd2209.csv --input" as an argument for snews_pt subscribe?

We tried that (which should have worked, as it worrks out of snews_pt, just snewpdag), but the output files (fits and png) are not created, maybe do to the warning. That's the output:

`INFO:root:Read detector database file snewpdag/data/detector_location_zerobias.csv Control: received 1 alerts /Users/amandadepoian/snews/supernova_communication/venv/lib/python3.8/site-packages/erfa/core.py:154: ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)" warnings.warn('ERFA function "{}" yielded {}'.format(func_name, wmsg), Dts-out: received 1 alerts

Dts-out >>>> (1) alert _id: 'SNEWS_Coincidence_ALERT 2022-10-19T18:18:30.118295' detector_names: ['DS-20K', 'XENONnT'] false_alarm_probability: 0.00044 neutrino_times: ['2030-01-01T12:34:47.678999', '2030-01-01T12:34:45.678999'] p_values: [0.98, 0.98] p_values average: 0.98 sent_time: '2022-10-19T18:18:30.118295' server_tag: 'PURDUE SERVER' sub list number: 0 action: 'alert' name: 'Control' history: ('Control', 'Dts') dts: ('DS-20K', 'XENONnT'): dt: 2.0 t1: 1893501287.678999 t2: 1893501285.678999 bias: 0.0 var: 1.8e-05 dsig1: 0.003 dsig2: -0.003 <<<< Dts-out <<<< INFO:root:cache (DS-20K, XENONnT): {'dt': 2000.0, 't1': 1893501287678.999, 't2': 1893501285678.999, 'bias': 0.0, 'var': 18.000000000000004, 'dsig1': 3.0, 'dsig2': -3.0}`

[tsengj10]

Are there only two detectors? The min_dts argument for DiffPointing specifies the minimum number of time differences before it will try to calculate a sky map. Usually I've set it to 3, which requires at least 3 detectors.

Not sure what's going on with the warning message.

[tsengj10]

Regarding the warning message: the burst times are set in the year 2030, which is around the time the Unix epoch overflows 32 bits. Which may be why it's being flagged as possibly a dubious year. (This is one of the things I was trying to avoid with the weird 64-bit timestamps, and later with a "local" epoch, but I guess if one is converting a text string you're limited by the conversion function.)

[mcolomermolla]

Are there only two detectors? The min_dts argument for DiffPointing specifies the minimum number of time differences before it will try to calculate a sky map. Usually I've set it to 3, which requires at least 3 detectors.

Not sure what's going on with the warning message.

Oh right I completely forgot about the minimum of three detectors, that makes sense that the skymap was not generales in that case then

[mcolomermolla]

Regarding the warning message: the burst times are set in the year 2030, which is around the time the Unix epoch overflows 32 bits. Which may be why it's being flagged as possibly a dubious year. (This is one of the things I was trying to avoid with the weird 64-bit timestamps, and later with a "local" epoch, but I guess if one is converting a text string you're limited by the conversion function.)

Okey I see, I was not expecting this to happen in the near future given the base epoch chosen. But yeah maybe we need to think more carefully about this. I was wanted to avoid the tuple [s,ns] but I am fine with 64bit if that's the most appropriate.

[tsengj10]

In this case it's a library function, so maybe we can figure out a way to give it a different epoch starting point. In any case the unix timestamp problem will need to be solved in standard libraries at some point.

[tsengj10]

Is it all right to run the next firedrill from this branch? It may be better to leave things alone until then. I can merge this PR afterwards.

[mcolomermolla]

Is it all right to run the next firedrill from this branch? It may be better to leave things alone until then. I can merge this PR afterwards.

I can run a firedrill with this branch, yes, but still the skymap is not created...

[mcolomermolla]

This is the output I get with three detectors sending alerts:

`(snewpy_env) marta@marta:~/snewpdag-firedrill$ snews_pt subscribe --firedrill -p "-m snewpdag --log INFO snewpdag/data/fd2209.csv --input" /home/marta/SNEWS_Publishing_Tools/snews_pt/init.py:23: UserWarning: You are using default detector name "TEST" Please change this by snews_pt.snews_pt_utils.set_name() warnings.warn(warning_text, UserWarning) Redirecting output to -m snewpdag --log INFO snewpdag/data/fd2209.csv --input You are subscribing to ALERT Broker:kafka://kafka.scimma.org/snews.alert-firedrill __ALERT MESSAGE__ _id :SNEWS_Coincidence_ALERT 2022-10-20T08:53:07.879195 server_tag :iap-nb-034
False Alarm Prob :2.23%
detector_names :DUNE DS-20K
sent_time :2022-10-20T08:53:07.879195
p_values :0.98 0.98
neutrino_times :2030-01-01T12:34:55.678999 2030-01-01T12:34:47.678999 p_values average :0.98
sub list number :0

---

INFO:root:Read detector database file snewpdag/data/detector_location_zerobias.csv Control: received 1 alerts /home/marta/miniconda3/envs/snewpy_env/lib/python3.8/site-packages/erfa/core.py:154: ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)" warnings.warn('ERFA function "{}" yielded {}'.format(func_name, wmsg), Dts-out: received 1 alerts

Dts-out >>>> (1) alert False Alarm Prob: '2.23%' _id: 'SNEWS_Coincidence_ALERT 2022-10-20T08:53:07.879195' detector_names: ['DUNE', 'DS-20K'] neutrino_times: ['2030-01-01T12:34:55.678999', '2030-01-01T12:34:47.678999'] p_values: [0.98, 0.98] p_values average: 0.98 sent_time: '2022-10-20T08:53:07.879195' server_tag: 'iap-nb-034' sub list number: 0 action: 'alert' name: 'Control' history: ('Control', 'Dts') dts: ('DUNE', 'DS-20K'): dt: 8.0 t1: 1893501295.678999 t2: 1893501287.678999 bias: 0.0 var: 1.8e-05 dsig1: 0.003 dsig2: -0.003 <<<< Dts-out <<<< INFO:root:cache (DUNE, DS-20K): {'dt': 8000.0, 't1': 1893501295678.999, 't2': 1893501287678.999, 'bias': 0.0, 'var': 18.000000000000004, 'dsig1': 3.0, 'dsig2': -3.0} __ALERT MESSAGE__ _id :SNEWS_Coincidence_ALERT-UPDATE 2022-10-20T08:53:12.284550 server_tag :iap-nb-034
False Alarm Prob :2.86%
detector_names :DUNE DS-20K XENONnT
sent_time :2022-10-20T08:53:12.284550
p_values :0.98 0.98 0.98
neutrino_times :2030-01-01T12:34:55.678999 2030-01-01T12:34:47.678999 2030-01-01T12:34:45.678999 p_values average :0.98
sub list number :0

---

INFO:root:Read detector database file snewpdag/data/detector_location_zerobias.csv Control: received 1 alerts /home/marta/miniconda3/envs/snewpy_env/lib/python3.8/site-packages/erfa/core.py:154: ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)" warnings.warn('ERFA function "{}" yielded {}'.format(func_name, wmsg), Dts-out: received 1 alerts Dts-out >>>> (1) alert False Alarm Prob: '2.86%' _id: 'SNEWS_Coincidence_ALERT-UPDATE 2022-10-20T08:53:12.284550' detector_names: ['DUNE', 'DS-20K', 'XENONnT'] neutrino_times: ['2030-01-01T12:34:55.678999', '2030-01-01T12:34:47.678999', '2030-01-01T12:34:45.678999'] p_values: [0.98, 0.98, 0.98] p_values average: 0.98 sent_time: '2022-10-20T08:53:12.284550' server_tag: 'iap-nb-034' sub list number: 0 action: 'alert' name: 'Control' history: ('Control', 'Dts') dts: ('DUNE', 'DS-20K'): dt: 8.0 t1: 1893501295.678999 t2: 1893501287.678999 bias: 0.0 var: 1.8e-05 dsig1: 0.003 dsig2: -0.003 ('DUNE', 'XENONnT'): dt: 10.0 t1: 1893501295.678999 t2: 1893501285.678999 bias: 0.0 var: 1.8e-05 dsig1: 0.003 dsig2: -0.003 <<<< Dts-out <<<< INFO:root:cache (DUNE, DS-20K): {'dt': 8000.0, 't1': 1893501295678.999, 't2': 1893501287678.999, 'bias': 0.0, 'var': 18.000000000000004, 'dsig1': 3.0, 'dsig2': -3.0} INFO:root:cache (DUNE, XENONnT): {'dt': 10000.0, 't1': 1893501295678.999, 't2': 1893501285678.999, 'bias': 0.0, 'var': 18.000000000000004, 'dsig1': 3.0, 'dsig2': -3.0} `

[tsengj10]

Whoops - forgot how the time differences work. We choose one detector, and then calculate the differences relative to that one, so 3 detectors give 2 time differences. The third time difference doesn't add any information. Sorry about that.

If you don't want to add another detector, you can change min_dts to 2. It should produce a skymap.

[mcolomermolla]

Whoops - forgot how the time differences work. We choose one detector, and then calculate the differences relative to that one, so 3 detectors give 2 time differences. The third time difference doesn't add any information. Sorry about that.

If you don't want to add another detector, you can change min_dts to 2. It should produce a skymap.

mmm but three detectors should give three time differences:

dt1 = tDUNE - tDS-20 dt2 = tDUNE - tXENONnT dt3 = tDS-20 - tXENONnT

(and all should give information?) right?

But okey, I will anyway try with min_dts =2

[mcolomermolla]

Whoops - forgot how the time differences work. We choose one detector, and then calculate the differences relative to that one, so 3 detectors give 2 time differences. The third time difference doesn't add any information. Sorry about that. If you don't want to add another detector, you can change min_dts to 2. It should produce a skymap.

mmm but three detectors should give three time differences:

dt1 = tDUNE - tDS-20 dt2 = tDUNE - tXENONnT dt3 = tDS-20 - tXENONnT

(and all should give information?) right?

But okey, I will anyway try with min_dts =2

I do get the skymap changing min_dts=2, but I get a very precise point with only two dts, which is not what I expect... I expected a band

[tsengj10]

I'll merge the PR tonight.

In the meantime, a couple of notes from today's firedrill:

• null in p_values was causing ast.literal_eval() to throw a ValueError. Need to see if there's a way to ignore malformed data.
• an unknown detector also shouldn't cause a problem, but should result in a non-fatal error message of some kind. DetectorDB returns None when it can't find a detector. DiffPointing (among others) need to handle that.