search

ML4GW / aframe-legacy

Detecting binary black hole mergers in LIGO with neural networks
MIT License
18 stars 17 forks source link

Online deployment #463

Closed wbenoit26 closed 9 months ago

wbenoit26 commented 1 year ago

Re-did the online deployment code to work with the local whitening, and the event detection all seems to work.

wbenoit26 commented 12 months ago

@alecgunny @EthanMarx I've had things running online for a couple of days now, and everything seems like it's working, so this is probably ready for an initial review. Still need to add some tests, and it would be good to let things go for at least a week to see if we hit any edge cases.

EthanMarx commented 12 months ago

@wbenoit26 Awesome stuff will take a look sometime this week. Keep us updated on any potential detections / non detections if the matched filtering pipelines find something

wbenoit26 commented 12 months ago

For sure! We've had two triggers at the 1/day level, the first corresponding to this superevent. The latter, at 1385025527, doesn't correspond to anything in GraceDB

wbenoit26 commented 12 months ago

Additional to-do: Ran in to RuntimeError: Missing FrEndOfFile structure, which ended the run. Need to figure out why that happens, or just add something to ignore it when it does.

alecgunny commented 12 months ago

@wbenoit26 did this happen during read_channel? If so, sounds like another edge case of weird things that can happen when you try to read a frame (hey there's another entendre) that hasn't fully written yet. Basically all the checks in there are meant to address that exact problem, so could be worth just adding another check for that error message and sleeping for a tick.

wbenoit26 commented 12 months ago

Yeah, I think that's probably the right call. I just want to check whether it's something that gets fixed after a second, or if the file will need to be skipped entirely.

EthanMarx commented 12 months ago

@wbenoit26 Another todo thats not necessary for this PR would be to query iDQ log likelihood channel for events. The iDQ channel should be distributed in the same frames as the strain. Taking a maximum +- 1s from the event time or something like that should be good.

wbenoit26 commented 9 months ago

@EthanMarx I think it would be good to get this into the repo for the review process