Development of communication infrastructure between Lasair, TiDES candidate selection function, and 4MOST transient API.

lsst-uk / Spectro-Class-Transients

0 stars 0 forks source link

Development of communication infrastructure between Lasair, TiDES candidate selection function, and 4MOST transient API. #37

Closed tms-epcc closed 1 year ago

tms-epcc commented 1 year ago

From AE FY22,

The development of the communication infrastructure between the LSST alert broker Lasair, the TiDES candidate selection function, and the 4MOST transient API;

tms-epcc commented 1 year ago

05/OCT/22

created from FY22 Annual evaluation Oct'22 to Dec'22 quarterly plan

tms-epcc commented 1 year ago

20/OCT/22 @chrisfrohmaier reported

the Lasair infrastructure is working
working on setting up the 4MOST infrastructure, once ready will set-up interface between the two

tms-epcc commented 1 year ago

09/NOV/22

using Somerville, will speak to 4MOST team about their API
WP3.3 will be the heaviest user and so it will be designed around WP3.3 needs

tms-epcc commented 1 year ago

01/DEC/22

Jake Laas (4MOST-Munich) working on @chrisfrohmaier API suggestions. Jake says none have been showstoppers thus far.
Little other progress on this task since @chrisfrohmaier been asked to redo #28 "Perform new simulations using the new LSST v2.0 simulations" by 01/DEC/22

tms-epcc commented 1 year ago

19/DEC/22

TidES -> 4 MOST done
Lasair -> TiDes to be done

tms-epcc commented 1 year ago

@chrisfrohmaier reported

have decided to use Apache Airflow to develop Lasair -> TiDES
currently getting up to speed on using Airflow
Airflow is for dealing with repetitive tasks
currently expect solution will involve
- scraping information Lasair
- process the information
- send results to 4MOST
Airflow forces you to document as you go along
Will create building blocks for different interfaces

tms-epcc commented 1 year ago

03/FEB/23

still doing Airflow familiarisation, this proving to be quite difficult to learn
already have a basic version of pipeline written in bash but want to convert this to Airflow since Airflow is more sustainable than bash

tms-epcc commented 1 year ago

24/FEB/23 @chrisfrohmaier reported:

now using Prefect instead of Airflow since easier
Prefect more sustainable than bash
now making progress beyond tutorials
at 4MOST, they may change their requirements after D3.3.4 deliverable date.
The change is likely to involve the addition of a small number of columns and so there are no concerns about this.

tms-epcc commented 1 year ago

10/MAR/23

making progress with Prefect
writing code

tms-epcc commented 1 year ago

04/MAY/23

Agreed at WP-PM meeting that this is finshed and so can be closed