Scale method up to work efficiently on HSC images

lsst-uk / sky-estimation-WP3.7

Repository for sky-subtraction routines to preserve LSB features, including tests with mock data

Apache License 2.0

2 stars 3 forks source link

Closed tms-epcc closed 1 year ago

tms-epcc commented 2 years ago

19/MAY/22

in the absence of other appropriate data at the moment working with a student from Oulu on a large dataset to investigate the performance of the algorithm
other datasets are being considered to help with the performance investigation

tms-epcc commented 2 years ago

09/JUN/22

99% of the solution is expected to be simple to implement
Determined that there is only one circumstance where a 2nd co-add is needed. This will be explained in the paper.
no updates from the Oulu student
will continue with refinements and tests

tms-epcc commented 2 years ago

01/JUL/22

meeting with DM
DM started their own experiments with some other ideas
no longer convinced that novel method is the way to go so now considering going with what the DM are experimenting
looking at three options, one is simple and easy to implement, the others require WP3.7 to develop
method 1 solves 90% of the problems
need to discuss with others in the DM team, not just Lee,
will still complete the novel method since likely to be useful even if not included in the pipeline. This could fix some of the worse frames in the dataset.
will instigate a more regular meeting schedule with the DM team

tms-epcc commented 2 years ago

28/JUL/22

@aemerywatkins preparing for August'22 PCW where will discuss the 3 options (referred to above) with the DM team. This input from the DM team is needed before progressing this task.

tms-epcc commented 2 years ago

19/AUG/22 @aemerywatkins reported

spoke to Susrya at PCW'22 who showed Aaron the software. It appears that some of the routines required for LSB are already available in the pipeline. Expect therefore to be able to scale this up to HSC.
will first confirm the routines do as expected
will tackle this after completing the notebook in #33

tms-epcc commented 1 year ago

23/SEP/22

tms-epcc commented 1 year ago

04/NOV/22

working with DM to determine algorithm to use and implementing this will in tandem with the scaling
will start in December '22
this task will extend into Phase C since requires iteration and pipeline is still changing

tms-epcc commented 1 year ago

19/DEC/22

@aemerywatkins visited DM team in December
wanted to confirm with DM team how to converge on strategy and scaling up
agreed Aaron will access HSC tracks direct from DM server and will also look at other datasets if available eg AuxTel.
Unclear how much scaling up can be achieved before Phase C starts

tms-epcc commented 1 year ago

At WP-PM meeting

Aaron explained the method was discussed with the DM team during Princeton visit
It has been agreed that the best option is to use the existing pipeline but to change its various parameters eg bin size, spline, etc to make it better suited to LSB science
the plan for scaling up to HSC is to use existing software pipeline but to perform tests to establish the appropriate parameters for LSB science (see #43)
agreed to close this ticket