Closed genghisken closed 1 year ago
genghisken
commented
1 year ago First draft working (with my local ATLAS database) of my crossmatchGWEvents code with simple coordinate tagging. I presume the probabilities come back in the order of the coordinates array!
Next stage - pull out the earliest MJD so I can get time deltas going. Then write the results to the database. Then test on both ATLAS and Pan-STARRS.
thespacedoctor
commented
1 year ago I presume the probabilities come back in the order of the coordinates array!
Yes :)
genghisken
commented
1 year ago Just a comment (extremely low priority - or unnecessary), but it would be really handy to be able to send a list of coords or coords + mjd along with an object ID list and get back the object ID along with the results.
At the moment I'm creating a list of RAs, list of Decs, list of earliest MJDs, assuming all are equal in size, and creating a list of IDs at the same time. When I get the results back I just the zip the IDs with the results. Works fine, but I need to actively housekeep (which is OK). Got it working now.
Now need to decide what to write back to the database. Simplest to reuse the existing tables (since I already have web queries working on those). I can pre-populate the tcs_gravity_events when the event arrives and then populate the tcs_gravity_event_annotations table. I already have the map name. I just need the event name which will be buried in the directory path to the actual map. (This will be automounted onto the relevant servers.)
mysql> select * from tcs_gravity_event_annotations limit 10;
+-----------+---------------------+------------------+------------+-------------------+------------------+------------------+---------+-------------+
| primaryId | transient_object_id | gravity_event_id | gracedb_id | enclosing_contour | map_name | dateLastModified | updated | dateCreated |
+-----------+---------------------+------------------+------------+-------------------+------------------+------------------+---------+-------------+
| 2780 | 1141802031301348000 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2781 | 1154322861153147300 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2782 | 1055152621605243200 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2783 | 1090842931444846700 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2784 | 1095312471622257500 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2785 | 1134639391232359100 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2786 | 1153514621151832400 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2787 | 1172730111331325300 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2788 | 1184311120412441600 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
| 2789 | 1063420761402410100 | S190408an | S190408an | 100 | bayestar.fits.gz | NULL | 0 | NULL |
+-----------+---------------------+------------------+------------+-------------------+------------------+------------------+---------+-------------+mysql> select * from tcs_gravity_events;
+------------------+------------+---------------+------------------+---------+-------------+
| gravity_event_id | gracedb_id | mjd | dateLastModified | updated | dateCreated |
+------------------+------------+---------------+------------------+---------+-------------+
| GW190408 | S190408an | 58581.76281 | NULL | 0 | NULL |
| GW190412 | S190412m | 58585.2296779 | NULL | 0 | NULL |
| GW190425 | S190425z | 58598.3458914 | NULL | 0 | NULL |
| GW190426 | S190426c | 58599.6402241 | NULL | 0 | NULL |
| GW190510 | S190510g | 58613.1247603 | NULL | 0 | NULL |
| GW190512 | S190512at | 58615.755028 | NULL | 0 | NULL |
| S190408an | S190408an | 58581.76281 | NULL | 0 | NULL |
| S190521r | S190521r | 58624.322216 | NULL | 0 | NULL |
| S190727h | S190727h | 58691.2524767 | NULL | 0 | NULL |
| S190728q | S190728q | 58692.2813719 | NULL | 0 | NULL |
| S190814bv | S190814bv | 58709.882396 | NULL | 0 | NULL |
| S190828j | S190828j | 58723.2736777 | NULL | 0 | NULL |
| S190828l | S190828l | 58723.2883089 | NULL | 0 | NULL |
| S190901ap | S190901ap | 58727.9798824 | NULL | 0 | NULL |
| S190910d | S190910d | 58736.0599449 | NULL | 0 | NULL |
| S190910h | S190910h | 58736.3541498 | NULL | 0 | NULL |
| S190930s | S190930s | 58756.5664496 | NULL | 0 | NULL |
| S190930t | S190930t | 58756.6070334 | NULL | 0 | NULL |
| S191110af | S191110af | 58797.9630093 | NULL | 0 | NULL |
| S191205ah | S191205ah | 58822.9112103 | NULL | 0 | NULL |
| S191213g | S191213g | 58830.190372 | NULL | 0 | NULL |
| S191216ap | S191216ap | 58833.898362 | NULL | 0 | NULL |
| S200105ae | S200105ae | 58853.6836349 | NULL | 0 | NULL |
| S200112r | S200112r | 58860.6657187 | NULL | 0 | NULL |
| S200114f | S200114f | 58862.0890999 | NULL | 0 | NULL |
| S200115j | S200115j | 58863.1827515 | NULL | 0 | NULL |
| S200129m | S200129m | 58877.2881763 | NULL | 0 | NULL |
| S200213t | S200213t | 58892.1740767 | NULL | 0 | NULL |
+------------------+------------+---------------+------------------+---------+-------------+
28 rows in set (0.00 sec)In fact, the way the original code worked was to insert the GW event ID into the database along with its MJD, and use that to calculate the time delta by joining with the object. Hence I could get away with just grabbing the contour for each object and using a table join to calculate the time delta dynamically (assuming I've got the event data already in the database).
genghisken
commented
1 year ago Dave's code returns a float for probability. I've modified the schema to add time_delta (float) so I can also write the days since event occurred. Currently the database looks like this now... Note that enclosing_contour was originally a multiple of 10 between 0 and 100. I've now just written the probability into that column, but it might not be the best way forward (because contour is an int). Might be better to add probability to a new column and generate the contour on insert.
mysql> select * from tcs_gravity_event_annotations order by primaryId desc limit 6;
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------+---------------------+---------+---------------------+
| primaryId | transient_object_id | gravity_event_id | gracedb_id | enclosing_contour | map_name | time_delta | dateLastModified | updated | dateCreated |
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------+---------------------+---------+---------------------+
| 2311684 | 1235015071223320700 | MS230506p | MS230506p | 87 | bayestar.multiorder.fits | -1937.43 | 2023-05-11 14:39:04 | 0 | 2023-05-11 14:39:04 |
| 2311683 | 1234301251185939200 | MS230506p | MS230506p | 5 | bayestar.multiorder.fits | -1999.31 | 2023-05-11 14:39:04 | 0 | 2023-05-11 14:39:04 |
| 2311682 | 1233831971184127400 | MS230506p | MS230506p | 45 | bayestar.multiorder.fits | -2017.28 | 2023-05-11 14:39:04 | 0 | 2023-05-11 14:39:04 |
| 2311681 | 1233830771192148900 | MS230506p | MS230506p | 66 | bayestar.multiorder.fits | -2025.27 | 2023-05-11 14:39:04 | 0 | 2023-05-11 14:39:04 |
| 2311680 | 1133151361052951700 | S200213t | S200213t | NULL | NULL | NULL | NULL | 0 | NULL |
| 2311679 | 1104356441580727700 | S200213t | S200213t | NULL | NULL | NULL | NULL | 0 | NULL |
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------+---------------------+---------+---------------------+Code was executed thus:
./crossmatchGWEvents.py ../../../../../atlas/config/config4_db1.yaml \
MS230506p \
/data/psdb3data1/o4_events/mockevents/MS230506p/20230506T160233_initial/bayestar.multiorder.fits \
atlas \
--list=4 \
--timedeltas \
--update
./crossmatchGWEvents.py -h
Crossmatch GW events using Dave's skytag code.
Usage:
./crossmatchGWEvents.py <configFile> <event> <gwEventMap> <survey> [<candidate>...] [--list=<list>] [--customlist=<customlist>] [--timedeltas] [--update]
./crossmatchGWEvents.py (-h | --help)
./crossmatchGWEvents.py --version
Survey must be panstarrs | atlas
Options:
-h --help Show this screen.
--version Show version.
--list=<list> List ID [default: 4].
--customlist=<list> Custom list ID [default: 4].
--timedeltas Pull out the earliest MJD (if present) for time delta calculations.
--update Update the database.
Example:
./crossmatchGWEvents.py /tmp/config.yaml MS230506p /data/psdb3data1/o4_events/mockevents/MS230506p/20230506T160233_initial/bayestar.multiorder.fits --list=2
./crossmatchGWEvents.py /tmp/config.yaml MS230506p /data/psdb3data1/o4_events/mockevents/MS230506p/20230506T160233_initial/bayestar.multiorder.fits panstarrs --list=5 --timedeltasI decided to add the probability column after all. So enclosing_contour and probability (as returned by Dave's code) are now stored. I also renamed time_delta to days_since_event. Note that a probability of 20.0 will be allocated to the 20 enclosing_contour, not 30. But a probability of 20.01 will be allocated to the 30 contour.
mysql> select * from tcs_gravity_event_annotations order by primaryId desc limit 6;
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------------+-------------+---------------------+---------+---------------------+
| primaryId | transient_object_id | gravity_event_id | gracedb_id | enclosing_contour | map_name | days_since_event | probability | dateLastModified | updated | dateCreated |
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------------+-------------+---------------------+---------+---------------------+
| 2311688 | 1235015071223320700 | MS230506p | MS230506p | 90 | bayestar.multiorder.fits | NULL | 87.01 | 2023-05-11 17:41:40 | 0 | 2023-05-11 17:41:40 |
| 2311687 | 1234301251185939200 | MS230506p | MS230506p | 10 | bayestar.multiorder.fits | NULL | 5.01 | 2023-05-11 17:41:40 | 0 | 2023-05-11 17:41:40 |
| 2311686 | 1233831971184127400 | MS230506p | MS230506p | 50 | bayestar.multiorder.fits | NULL | 45.12 | 2023-05-11 17:41:40 | 0 | 2023-05-11 17:41:40 |
| 2311685 | 1233830771192148900 | MS230506p | MS230506p | 70 | bayestar.multiorder.fits | NULL | 65.82 | 2023-05-11 17:41:40 | 0 | 2023-05-11 17:41:40 |
| 2311680 | 1133151361052951700 | S200213t | S200213t | NULL | NULL | NULL | NULL | NULL | 0 | NULL |
| 2311679 | 1104356441580727700 | S200213t | S200213t | NULL | NULL | NULL | NULL | NULL | 0 | NULL |
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------------+-------------+---------------------+---------+---------------------+Might be best to ignore (not insert) transients outside the 90% credibility. May be you are already ignoring?
genghisken
commented
1 year ago Yep. Already ignoring anything outside 90% area.
genghisken
commented
1 year ago Also now tested with and tagged Pan-STARRS objects.
mysql> select * from tcs_gravity_event_annotations order by primaryId desc limit 4;
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------------+-------------+---------------------+---------+---------------------+
| primaryId | transient_object_id | gravity_event_id | gracedb_id | enclosing_contour | map_name | days_since_event | probability | dateLastModified | updated | dateCreated |
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------------+-------------+---------------------+---------+---------------------+
| 3273966 | 1233831861184127900 | MS230506p | MS230506p | 50 | bayestar.multiorder.fits | NULL | 45.12 | 2023-05-12 01:10:23 | 0 | 2023-05-12 01:10:23 |
| 3273965 | 1233639601165036400 | MS230506p | MS230506p | 60 | bayestar.multiorder.fits | NULL | 55.3 | 2023-05-12 01:10:23 | 0 | 2023-05-12 01:10:23 |
| 3273964 | 1015408931320435200 | S200115j | S200115j | NULL | NULL | NULL | NULL | NULL | 0 | NULL |
| 3273963 | 1015408931320435200 | S200105ae | S200105ae | NULL | NULL | NULL | NULL | NULL | 0 | NULL |
+-----------+---------------------+------------------+------------+-------------------+--------------------------+------------------+-------------+---------------------+---------+---------------------+One final thing to implement is the --flagdate option so that we don't bother looking for objects that were flagged way before the event.
thespacedoctor
commented
1 year ago I think we had agreed on a temporal window of -0.5d < discovery epoch < 7d
genghisken
commented
1 year ago I've added a new map_iteration column. The code will attempt to derive it from the filename - e.g. 20230506T160233_initial, otherwise the --mapiteration option can be used to override it. I think this is all we need. We then just need to figure out how to get the event details (e.g. event MJD, BNS, etc) into tcs_gravity_events table. I'll look at that over the weekend.
mysql> select * from tcs_gravity_event_annotations order by primaryId desc limit 6;
+-----------+---------------------+------------------+------------+-------------------+--------------------------+-------------------------+------------------+-------------+---------------------+---------+---------------------+
| primaryId | transient_object_id | gravity_event_id | gracedb_id | enclosing_contour | map_name | map_iteration | days_since_event | probability | dateLastModified | updated | dateCreated |
+-----------+---------------------+------------------+------------+-------------------+--------------------------+-------------------------+------------------+-------------+---------------------+---------+---------------------+
| 2311692 | 1235015071223320700 | MS230506p | MS230506p | 90 | bayestar.multiorder.fits | 20230506T160233_initial | NULL | 87.01 | 2023-05-12 17:26:01 | 0 | 2023-05-12 17:26:01 |
| 2311691 | 1234301251185939200 | MS230506p | MS230506p | 10 | bayestar.multiorder.fits | 20230506T160233_initial | NULL | 5.01 | 2023-05-12 17:26:01 | 0 | 2023-05-12 17:26:01 |
| 2311690 | 1233831971184127400 | MS230506p | MS230506p | 50 | bayestar.multiorder.fits | 20230506T160233_initial | NULL | 45.12 | 2023-05-12 17:26:01 | 0 | 2023-05-12 17:26:01 |
| 2311689 | 1233830771192148900 | MS230506p | MS230506p | 70 | bayestar.multiorder.fits | 20230506T160233_initial | NULL | 65.82 | 2023-05-12 17:26:01 | 0 | 2023-05-12 17:26:01 |
| 2311680 | 1133151361052951700 | S200213t | S200213t | NULL | NULL | NULL | NULL | NULL | NULL | 0 | NULL |
| 2311679 | 1104356441580727700 | S200213t | S200213t | NULL | NULL | NULL | NULL | NULL | NULL | 0 | NULL |
+-----------+---------------------+------------------+------------+-------------------+--------------------------+-------------------------+------------------+-------------+---------------------+---------+---------------------+Code is now done. It reads from a new tcs_transient_alerts table in each database (loaded automatically from db3) and triggers crossmatches based currently on the basic criteria listed below. These criteria are up for discussion but is currently a superset (I think) of our ATLAS & Pan-STARRS followup criteria.
The code can also implement a distance trigger and with a few tiny modifications can trigger on the properties (e.g. HasNS > threshold) and/or classification (e.g. BNS or NSBH > threshold).
I'm intending to run the code hourly on each database, but can run more frequently if necessary.
genghisken
commented
1 year ago FYI, here are all the options implemented so far:
./crossmatchGWEvents.py -h
Crossmatch GW events using Dave's skytag code.
Usage:
./crossmatchGWEvents.py <configFile> <survey> [<candidate>...] [--listid=<listid>] [--customlist=<customlist>] [--datethreshold=<datethreshold>] [--timedeltas] [--distances] [--mapiteration=<mapiteration>] [--event=<event>] [--gwEventMap=<gwEventMap>] [--areaThreshold=<areaThreshold>] [--areaContour=<areaContour>] [--distanceThreshold=<distanceThreshold>] [--far=<far>] [--mapRootLocation=<mapRootLocation>] [--daysBeforeEvent=<daysBeforeEvent>] [--daysAfterEvent=<daysAfterEvent>] [--update]
./crossmatchGWEvents.py (-h | --help)
./crossmatchGWEvents.py --version
Survey must be panstarrs | atlas
Options:
-h --help Show this screen.
--version Show version.
--listid=<listid> List ID [default: 4].
--customlist=<customlist> Custom list ID.
--datethreshold=<datethreshold> Only pull out objects with a flag date greater than the datethreshold in YYYMMDD format. Only applied to listid [defaut: 20230101].
--timedeltas Pull out the earliest MJD (if present) for time delta calculations.
--distances Add the individual distances to the results.
--mapiteration=<mapiteration> By default code tries to derive the map iteration from the directory name. Override this.
--event=<event> GW event. Override a database check of events.
--gwEventMap=<gwEventMap> GW event map. Override the map location derived from the database.
--areaThreshold=<areaThreshold> Area threshold before doing the crossmatch, ignored if map provided [default: 2000].
--areaContour=<areaContour> Area contour to which to apply the area check - 90, 50 or 10, ignored if map provided [default: 90].
--distanceThreshold=<distanceThreshold> Distance threshold to apply if present (Mpc).
--mapRootLocation=<mapRootLocation> Root location where the maps are stored [default: /data/psdb3data1/o4_events].
--daysBeforeEvent=<daysBeforeEvent> Days before the event to trigger search [default: 10].
--daysAfterEvent=<daysAfterEvent> Days after the event to trigger search [default: 21].
--far=<far> False alarm rate (e.g. 1 in 6 months) [default: 0.000000015844369].
--update Update the database.
Example:
./crossmatchGWEvents.py /tmp/config.yaml --event=MS230506p --gwEventMap=/data/psdb3data1/o4_events/mockevents/MS230506p/20230506T160233_initial/bayestar.multiorder.fits --list=2
./crossmatchGWEvents.py /tmp/config.yaml --event=MS230506p --gwEventMap=/data/psdb3data1/o4_events/mockevents/MS230506p/20230506T160233_initial/bayestar.multiorder.fits panstarrs --list=5 --timedeltas
./crossmatchGWEvents.py ../../../../../atlas/config/config4_db1_readonly.yaml --event=S230518h --gwEventMap=/data/psdb3data1/o4_events/superevents/S230518h/20230526T221627_update/ligo-skymap-from-samples.multiorder.fits atlas --listid=10 --update --timedeltas --datethreshold=20230517
./crossmatchGWEvents.py ../../../../../atlas/config/config4_db1_readonly.yaml atlas --listid=2 --update --timedeltas --distanceThreshold=500Code is configured initially to run every hour on atlas (db4) and ps13pi (db0). I will increase the frequency once I'm satisfied everything is running OK.
There is no data in the O4 database yet, but it will be easy to fix once we start a targeted campaign.
Now that the tagging API code is created we now want to implement some code that reads the Pan-STARRS and ATLAS databases and takes the name, ra, dec and first detection time and write the tag information into the (existing) tables. The
tcs_gravity_eventsandtcs_gravity_event_annotationstables can be easily reused. (We may want to add some new columns.) Reusing the existing tables allows us to exploit query code already written.