MVM: Plan how to implement remaining development work

[stscijgbot-hstdp]

Issue HLA-496 was created on JIRA by Warren Hack:

A plan needs to be developed in order to most efficiently complete the rest of the development work for the MVM processing code. This planning effort will result in either updates to existing tickets or new tickets to cover all the identified issues remaining to be resolved. In addition, this plan will need to provide some initial indications as to who will be working on resolving the various issues, whether that means we all work on the issues one at a time or whether we can work on some in parallel.

The remaining issues which have been identified to date include:

• Creating poller files for standard test SkyCells to be used for development.  An example of the most up-to-date (March 2021) MVM poller file is in HSTSDP-1040. At least one of the test SkyCells should contain data spanning at least 10 years for the same detector/filter. A suitable dataset has been identified and documented in HLA-488; namely, the NGC45 field (32 exposures, ACS&WFC3, spanning >10 years).
• Implementing the AWG provided algorithm for aligning to multiple catalogs (see HLA-487) ** Actual implementation will take place through HLA-509, HLA-510, and HLA-511.
• Working out how to apply this new multi-catalog fitting logic to standard pipeline and SVM processing as well as MVM processing, since it would increase the number of fields which could be aligned by a significant fraction Work for HLA-509, HLA-510, and HLA-511 will actually update SVM code. Integrating this into standard pipeline will start with HLA-486 where latest hybrid catalog will be used as the primary catalog instead of GAIADR2. Use of SMV/MVM code may never get used directly given the differences in APIs between the code bases.
• Developing additional verification algorithms to insure alignment of exposures in the SkyCell mosaic is not compromised. SECONDARY PRIORITY??? REQUIRED??? Planning will be required before starting any actual coding (HLA-475) Standard pipeline verification looked for double-images of the same sources in the output drizzle product MVM products with inputs spanning 10 or more years will almost always result in the sources not aligning producing double- or smeared-images when alignment is correct. MVM products will also include fields where some of the inputs do NOT overlap (HLA-433)
• Implementing what is essentially a replacement for the tweakwcs fitting algorithm to fit for proper motions at the same time as we fit for global offsets and rotations.  (HLA-497 will serve as the 'epic' for this specific effort) HLA-515, HLA-516, HLA-517, HLA-518, HLA-519, HLA-520, and HLA-521 will track this effort. This may also lead to developing catalogs with proper motions which may be suitable as new archive products that are complimentary to or extensions of the work done by the HSC. This work has been put on hold** until a viable algorithm can be developed by SCSB as they learn how to use astropy.modeling and it's full set of capabilities. (2021-06-01)
• Investigate the feasibility of using the SVM source catalogs generated from previous fits to align new data to the SkyCell SECONDARY PRIORITY ** This could have a significant impact on how long it takes to add new data to an existing SkyCell product.  
• Sky matching algorithm is currently time and memory intensive SECONDARY PRIORITY Work may need to be done to make this more efficient to support SkyCell. AL-620 Use of HDRTAB sky values for all affected/common exposures may allow for more efficient sky matching ** Initial options for pipeline could simply include not using the 'match' options for some/all datasets
• If cosmic-ray identification is not needed for the MVM products, we could instead use the standalone drizzle package to add new data to a SkyCell while using less memory than AstroDrizzle as a whole SECONDARY PRIORITY This may also help with the population of SkyCells with vast amounts of data.   Discussions with INS would be required to determine whether or not CR rejection is required for MVM products ** Logic may be developed to identify where CR rejection is not needed in MVM data due to quality of input CR rejection, then only in remaining cases, do the CR rejection with full AstroDrizzle.
• Devise an interface to astropy.io.fits that would use less memory when processing data that only updates a small fraction (<10%) of the output SkyCell drizzled products. REQUIRED Considered a requirement due to the extreme amount of memory and disk space needed to manage mostly empty files or arrays. Such an algorithm may rely on working with subarrays using the LTM keywords. Changes to use a contiguous subarray region for the drizzling are being developed as a starting point. (HLA-569 and HLA-570) This would hopefully be less work than trying to implement section-by-section compression provided by CFITSIO into astropy.io.fits, work described in HLA-279, HLA-297, HLA-298 and HLA-299. This may also be necessary to address the issue raised in HLA-415.  Standalone compression tools (python-blosc) can be used internally on the SkyCell class for the masks (under HLA-568) to help with memory use in address HLA-415. Consider use of ftools off-line tiled compression for SkyCell products

These issues represent what have been identified as the minimum set of issues which need to be addressed in order to have what can be considered pipeline-ready code suitable for creating nearly science-capable results.  This list also includes some research items which would improve performance in one way or another (memory use, runtime speed, ...).  The performance issues should be prioritized based on how feasible it is to process the deepest SkyCells without the improvements. 

Items marked as SECONDARY PRIORITY would make processing either more memory efficient or otherwise make it easier to process cells with the largest amount of data, but does not impact whether the data can be combined in general. These issues can be implemented as improvements on an implementation that has already been delivered to the pipeline for initial generation of SkyCell products.

[stscijgbot-hstdp]

Comment by Warren Hack on JIRA:

Initial discussion of this plan on 18/Mar/21 resulted in the following estimates of how long it might take to implement major aspects of this plan:

• Updated logic for using multiple catalogs (HLA-487): 2 full Sprints
• Replacement fitting routine to fit for proper motions as well as offsets (HLA-497): 3 full Sprints

[stsci-hack]

With the release of 3.4.0rc1, all operationally important aspects of this plan have been successfully completed and delivered for use in operations.