search

lsst-uk / lasair-project-management

Event handling site for LSST:UK
Apache License 2.0
9 stars 0 forks source link

Stream logs lag behind? #249

Closed Fingel closed 2 weeks ago

Fingel commented 2 years ago

Hi, StarDestroyers is running into an issue where alerts are appearing in the real time queries, but are absent in the JSON stream logs. Is there some cadence when the stream logs run that might introduce some kind of delay? A current example would be ZTF22aalzyzl which appears when running this query: https://lasair-iris.roe.ac.uk/query/126/ But is absent from it's stream log: https://lasair-iris.roe.ac.uk/lasair/static/streams/lasair_165NewTDEs

Thanks for any possible explanation you can give!

RoyWilliams commented 2 years ago

Hi Austin Thanks again for your invaluable help improving Lasair. I fixed it. The TNS clone table gets pushed between nodes every 4 hours, so the streaming queries can use it. But that was failing, so the main database has the right table, but the "other" database did not. Thats why your streaming query failed.

The daily log of this process is linked from the bottom of the status page (https://lasair-iris.roe.ac.uk/status/) as "services log". You can see the line from 3 hours ago where a file of zero size is made: -rw-rw-r-- 1 ubuntu ubuntu 0 Jun 10 04:11 /mnt/cephfs/lasair/crossmatch_tns/crossmatch_tns.sql

Now that I fixed the dumper code, you can see where I just remade it at 28 Mb -rw-rw-r-- 1 ubuntu ubuntu 28181410 Jun 10 06:55 /mnt/cephfs/lasair/crossmatch_tns/crossmatch_tns.sql

I will be off the grid for 3 days, would be great if you could help us by looking for this working properly .... :-) Roy

tms-epcc commented 2 weeks ago

24/OCT/24 agreed to close at SE meeting