Investigate the viability of "background matching" technique

[tms-epcc]

From Dec'22 quarterly update

• Investigate the viability of a second-order solution proposed by DM called "background matching" for possible use in full survey.
• this was a stretch goal for Phase B
• not a first order priority

[tms-epcc]

10/MAY/23

• This ticket corresponds to the following task as described in the FY23 Q2 March'23 quarterly update
  • Discuss "background-matching" sky-subtraction technique with Data Management to investigate viability.
  • this is a stretch goal and is not fundamental to the strategy
  • currently not convinced it will provide an improvement
  • met DM on 08/MAY/23 to discuss this
• This ticket also corresponds to the further task as described in the FY23 Q2 March'23 quarterly update
  • Continue ongoing work on LSB-friendly sky subtraction algorithms and work with DM on strategies for incorporation into pipeline.
  • This tasks will be ongoing for some time.
  • Discussions on incorporation will be ongoing. This will require DM approval. Expect more iteration on this with DM team.

[tms-epcc]

30/MAY/23

• working on other priorities so no progress

[tms-epcc]

15/JUN/23

• no progress due to other priorities

[tms-epcc]

04/JUL/23

• expect to start discussions with DM team on implementation
• automation a key consideration for metrics

[tms-epcc]

20/JUL/23

• continuing to investigate viability
• on hold
• this is a stretch goal

[tms-epcc]

11/APR/24

• This was originally a stretch goal but further investigations has taken place
• Aaron made synthetic images and assessed what is meant to happen and why.
• Able to get same sky across all coadds
• DM have some existing software that Aaron may finish if appropriate

[tms-epcc]

02/MAY/24

• progressing this
• there is a version in the LSST github.
• asked Robert Lupton about status of this but no response yet
• broadening strategy by considering other options
• challenge now is image quality which has a lot of factors

[tms-epcc]

23/MAY/24

• recvd confirmation that the DM Github package is doing background matching
• found that package had been tried on HSC pipeline but DM team found it did not work as they wanted so it has been set aside by DM
• One option is therefore for @aemerywatkins to get involved and take this package further
• For the moment will investigate this on HSC and will talk to Robert and Yusra on why exactly the package was shelved.

[tms-epcc]

17/JUN/24

• got technique working on own images
• indicates this is viable
• presented to Lee and others in DM
• will discuss more with Lee
• will then set up ticket to revise the code and then test on HSC images

[tms-epcc]

09/JUL/24

• See DM JIRA ticket - https://rubinobs.atlassian.net/browse/DM-44889
• existing code in LSST Github - matchbackgrounds.py written some time ago
• @aemerywatkins getting this working with Gen3
• mostly done - code executes apart from last part when writing images to disk

[tms-epcc]

23/JUL/24

• code is functioning
• run on series of images, output was sensible
• working with Lee on how to optimise and meet DM's code standards
• this approach is still the most attractive since it means people can reprocess images themselves rather than having to supercomputing time to do this.
• still to try running this at scale and obviously still to convince DM that this is the most appropriate technique
• so currently ironing out various details

[tms-epcc]

06/AUG/24

• Robert Lupton taking great interest and providing ideas
• Suggested try the technique on old M31 galaxy images since if it works on these it should work on many others
• So currently working on this