The implementation is not yet finished, that’s why this is a draft, but people can start review it. I’ll explain the gist of what has been done so far, and I’ll later explain what needs to be done.
The gw/survey_queries module
The core of the implementation is the new survey_queries module. All the hard coded stuff is included in the __init__.py. These things include instrument names and specifications (like resolution and FOV). The function generate_FOV_grid is also included there. This function creates a grid of coordinates which will serve as a check to know if the entire FOV of LCO is covered by the final template.
The module defines a template_query class, inside query.py. This is a survey-agnostic class, that will collect all the necessary info and distribute them to each specific survey query. The class will then gather the list of tiles from each survey, and then reproject and merge them. The path for each final template fits file is also stored in the class.
Each survey-specific query (like PanSTARRS_search.py ) will handle the actually query of the data, as well as the necessary steps to make the images ready to be used as templates.
The gw/run_template_search.py file
This file is basically just a wrapper for the whole process. A template_query object is created, then each survey is called.
The gw/views.py and gw/hooks.py edits
Previously, the databases were updated one object at a time. Now each target is included in a list and then all targets are uploaded in one go. This is done to favor a cleaner parallelization, so that only one process managing all downloads will be created.
What needs to be done
The downloading and reprojecting takes a lot of time. The run_template_search function is called inside the ingest_gw_galaxy_into_snex1 function in gw/hooks.py and will cause the function to hang, causing also the triggering of the observations to hang. In order to avoid that, the download needs to be run asynchronously. This has not been implemented yet, but TOM base has already some built functionalities to handle it and I am trying to apply them.
New surveys will need to be added. DECam templates will be among the first to be included. The data are spread over numerous separate surveys and these are not all included in NOIRLab. A more accurate search for alternative endpoints needs to be done.
Finally, the templates need to be fully tested by the pipeline, which has not been done yet. They should be ok, but time will tell.
The implementation is not yet finished, that’s why this is a draft, but people can start review it. I’ll explain the gist of what has been done so far, and I’ll later explain what needs to be done.
The
gw/survey_queriesmoduleThe core of the implementation is the new
survey_queriesmodule. All the hard coded stuff is included in the__init__.py. These things include instrument names and specifications (like resolution and FOV). The functiongenerate_FOV_gridis also included there. This function creates a grid of coordinates which will serve as a check to know if the entire FOV of LCO is covered by the final template.The module defines a
template_queryclass, insidequery.py. This is a survey-agnostic class, that will collect all the necessary info and distribute them to each specific survey query. The class will then gather the list of tiles from each survey, and then reproject and merge them. The path for each final template fits file is also stored in the class.Each survey-specific query (like
PanSTARRS_search.py) will handle the actually query of the data, as well as the necessary steps to make the images ready to be used as templates.The
gw/run_template_search.pyfileThis file is basically just a wrapper for the whole process. A
template_queryobject is created, then each survey is called.The
gw/views.pyandgw/hooks.pyeditsPreviously, the databases were updated one object at a time. Now each target is included in a list and then all targets are uploaded in one go. This is done to favor a cleaner parallelization, so that only one process managing all downloads will be created.
What needs to be done
The downloading and reprojecting takes a lot of time. The
run_template_searchfunction is called inside theingest_gw_galaxy_into_snex1function ingw/hooks.pyand will cause the function to hang, causing also the triggering of the observations to hang. In order to avoid that, the download needs to be run asynchronously. This has not been implemented yet, but TOM base has already some built functionalities to handle it and I am trying to apply them.New surveys will need to be added. DECam templates will be among the first to be included. The data are spread over numerous separate surveys and these are not all included in NOIRLab. A more accurate search for alternative endpoints needs to be done.
Finally, the templates need to be fully tested by the pipeline, which has not been done yet. They should be ok, but time will tell.