search

LCOGT / ztf-alert-server

Alert repository for ZTF alerts
GNU General Public License v3.0
6 stars 2 forks source link

Missing recent photometry. #19

Open Fingel opened 2 years ago

Fingel commented 2 years ago

Hi, it appears that MARS is missing photometry from recent ZTF events. This seems to have been an issue since the end of the Great ZTF Outage of late 2021, early 2022.

Here are two examples:

ZTF20abbhlct Photometry is as recent as April 11, on alerce: https://alerce.online/object/ZTF20abbhlct

Nothing since June 2021 on MARS: https://mars.lco.global/?objectId=ZTF20abbhlct

Similarly with ZTF18acwyvak Alerce, photometry from today: https://alerce.online/object/ZTF18acwyvak

MARS, nothing since November 2021: https://mars.lco.global/?objectId=ZTF18acwyvak

This could be an issue with the listener not properly handling the large outage properly. I'd be happy to help, but it's a bit harder for me to poke around in production these days :)

mgdaily commented 2 years ago

Here's the traceback

mars-ingest-8647b85747-nwq7k mars
mars-ingest-8647b85747-nwq7k mars Traceback (most recent call last):
mars-ingest-8647b85747-nwq7k mars   File "ingest.py", line 245, in <module>
mars-ingest-8647b85747-nwq7k mars     start_consumer()
mars-ingest-8647b85747-nwq7k mars   File "ingest.py", line 228, in start_consumer
mars-ingest-8647b85747-nwq7k mars     success, candid = do_ingest(alert)
mars-ingest-8647b85747-nwq7k mars   File "ingest.py", line 82, in do_ingest
mars-ingest-8647b85747-nwq7k mars     ingestion_status, candid = ingest_avro(packet)
mars-ingest-8647b85747-nwq7k mars   File "ingest.py", line 119, in ingest_avro
mars-ingest-8647b85747-nwq7k mars     count = db.session.query(NonDetection).filter(NonDetection.objectId == nd.objectId).filter(NonDetection.jd==nd.jd).count()
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3145, in count
mars-ingest-8647b85747-nwq7k mars     return self._from_self(col).enable_eagerloads(False).scalar()
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2879, in scalar
mars-ingest-8647b85747-nwq7k mars     ret = self.one()
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2856, in one
mars-ingest-8647b85747-nwq7k mars     return self._iter().one()
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2897, in _iter
mars-ingest-8647b85747-nwq7k mars     execution_options={"_sa_orm_load_options": self.load_options},
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1646, in execute
mars-ingest-8647b85747-nwq7k mars     _parent_execute_state is not None,
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/context.py", line 318, in orm_pre_session_exec
mars-ingest-8647b85747-nwq7k mars     session._autoflush()
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2237, in _autoflush
mars-ingest-8647b85747-nwq7k mars     util.raise_(e, with_traceback=sys.exc_info()[2])
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
mars-ingest-8647b85747-nwq7k mars     raise exception
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2226, in _autoflush
mars-ingest-8647b85747-nwq7k mars     self.flush()
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 3363, in flush
mars-ingest-8647b85747-nwq7k mars     self._flush(objects)
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 3503, in _flush
mars-ingest-8647b85747-nwq7k mars     transaction.rollback(_capture_exception=True)
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
mars-ingest-8647b85747-nwq7k mars     with_traceback=exc_tb,
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
mars-ingest-8647b85747-nwq7k mars     raise exception
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 3463, in _flush
mars-ingest-8647b85747-nwq7k mars     flush_context.execute()
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
mars-ingest-8647b85747-nwq7k mars     rec.execute(self)
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 633, in execute
mars-ingest-8647b85747-nwq7k mars     uow,
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 249, in save_obj
mars-ingest-8647b85747-nwq7k mars     insert,
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1224, in _emit_insert_statements
mars-ingest-8647b85747-nwq7k mars     execution_options=execution_options,
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
mars-ingest-8647b85747-nwq7k mars     return meth(self, args_10style, kwargs_10style, execution_options)
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
mars-ingest-8647b85747-nwq7k mars     self, multiparams, params, execution_options
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
mars-ingest-8647b85747-nwq7k mars     cache_hit=cache_hit,
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
mars-ingest-8647b85747-nwq7k mars     e, statement, parameters, cursor, context
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2027, in _handle_dbapi_exception
mars-ingest-8647b85747-nwq7k mars     sqlalchemy_exception, with_traceback=exc_info[2], from_=e
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
mars-ingest-8647b85747-nwq7k mars     raise exception
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
mars-ingest-8647b85747-nwq7k mars     cursor, statement, parameters, context
mars-ingest-8647b85747-nwq7k mars   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
mars-ingest-8647b85747-nwq7k mars     cursor.execute(statement, parameters)
mars-ingest-8647b85747-nwq7k mars sqlalchemy.exc.DataError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
mars-ingest-8647b85747-nwq7k mars (psycopg2.errors.SequenceGeneratorLimitExceeded) nextval: reached maximum value of sequence "non_detection_id_seq" (2147483647)
mars-ingest-8647b85747-nwq7k mars
mars-ingest-8647b85747-nwq7k mars [SQL: INSERT INTO non_detection ("objectId", diffmaglim, jd, fid) VALUES (%(objectId)s, %(diffmaglim)s, %(jd)s, %(fid)s) RETURNING non_detection.id]
mars-ingest-8647b85747-nwq7k mars [parameters: {'objectId': 'ZTF18acsosen', 'diffmaglim': 19.388099670410156, 'jd': 2459685.6911111, 'fid': 2}]
mars-ingest-8647b85747-nwq7k mars (Background on this error at: https://sqlalche.me/e/14/9h9h)

Looks like this is a postgresql int overflow issue. I think the most expedient fix would be to bring the application down, do quick migration on the offending table to make the PK a bigint, then bring the updated application up with the NonDetection's id set to sqlalchemy.types.BigInteger.

https://www.postgresql.org/docs/current/datatype-numeric.html https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.BigInteger

Fingel commented 2 years ago

Sounds right to me. The index will also have to be updated, something like this: ALTER SEQUENCE non_detection_id_seq AS bigint;

This is a large database so I expect this will take a long time. But by most practical means MARS is offline now anyway, so I see no harm in some downtime to run the migration.

mgdaily commented 2 years ago

Migration is complete. New alerts now present. We will work on back-populating to fill the gap.

Here's a summary of what was required.