Closed EiffL closed 4 years ago
@joezuntz I'm done implementing the mass-mapping pipeline with descpipe. Works very nicely, but I have one small issue at NERSC. Because this is a private project, I'm hosting the docker images on the private repository at NERSC ,which means that to load the shifter images I need to use something like:
shifter --image=registry.services.nersc.gov/flanusse/descpipe-wlmassmap-mock_observation:1.0
but descpipe automatically generates a docker based command:
shifter --image=docker:flanusse/descpipe-wlmassmap-mock_observation:1.0
Right now I'm manually replacing these lines in the generated bash script, but do you think there is a way to specify to descpipe which repository to use ?
I've added a few notes in the code. Good point about the shifter script. The whole script generation is a total hack and it would be useful to have a quick think about how it should look for local systems and NERSC.
I'm closing this one as we now interface with ceci and TXPipe
The mass-mapping pipeline will use a number of different mapping algorithms, each one with their own dependencies. The descpipe (https://github.com/joezuntz/descpipe) framework provides a flexible way to handle this problem by packaging each step of the pipeline in a docker container.
I open this issue to work on the implementation of the mass-mapping pipeline with descpipe.