Stress Test

[guynir42]

We've been asked to stress test the pipeline. I'm not sure what that means exactly, as we are already running lots of tests. Let's discuss.

[rknop]

Here's what I envision for this. This is something we'd do ad-hoc for demonstration and experimentation purposes, and not as one of the regular regression tests we have in our tests directory.

Overview: start with a DECam exposure. Run 60 processes that each process one of the chips of the exposure. Look at what gets loaded in as detections (cutout and measurements).

To this end, we'll need to make sure we have a command line thing we can run to which we can pass the exposure that will then launch the 60 different processes, each being given a single chip to run. We'll also have to have some way of poking at the database. I'm thinking of quickly adapting the lensgrinder web server I have that lets us look at measurements and cutouts so that we'll have something to use just for this demo.

We'll want to see how long it takes to run. Ideally, the whole runtime should not be much longer than the expected LS4 exposure time. (If it's up to a factor of 2, we're still OK, as it means we won't fall behind. Of course, we could also just run on more nodes at once in that case.)

[guynir42]

• [ ] Solve memory issue
• [ ] Make sure subtractions look ok
• [ ] web interface
• [ ] Report class etc.
• [x] Run on NERSC node (including postgres server on Spin)
• [ ] Script to run multiple instances on the same exposure
• [ ] Get 1 or 2 exposures to run
• [ ] Rehearse

[rknop]

I've started work on the web interface

[rknop]

I'm going to be offline tomorrow (Fri May 3).

Here's my current status:

• I have a prelimiary version of the web interface. It can list exposures and images, and show cutouts from images.
• I've created postgres and archive servers on spin
• I've succesfully run SeeChange in podman-hpc on nersc
• I have a hack that will import DECam references from existing FITS files. (Mostly... it loads the Image, doesn't create the actual reference entry yet, but soon.)

Next week, I will finish the ref importer, and add something somewhere that processes an entire exposure by launching subprocesses.

[rknop]

My working branch is in https://github.com/rknop/SeeChange/tree/webap

(The branch name is not quite right, but whatever.)

[guynir42]

Rob, please make sure to make pull request(s) with all the useful code you've used to make this happen.

Great job on this!