Issues/315 - Githubissues

LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.

BSD 3-Clause "New" or "Revised" License

11 stars 7 forks source link

Issues/315 #321

Closed wmwv closed 5 years ago

wmwv commented 5 years ago

Run 1.2p files prepared and installed. DPDD versions prepared. README updated.

I'm particularly interested in suggestions for making the README clearer.

johannct commented 5 years ago

If the question is about the scripts README, I think it can be improved indeed :

dependencies are not spelled out to run the suite (tables of course, which should soon be part of the stack conda, but also GCR and GCRCatalogs)
output products are not explicitely stated, especially the naming of the hdf5 files
specifics of the run 1.Xy actual command lines should come after a generic exposition of the sequence of command lines to be executed.
I iterate here a remark I made before : it is not super pleasant to create products in-place, id est to be forced to do cd $WORKDIR in order to have the outputs be saved in $WORKDIR. I got tricked by that once, forgetting to move back outputs that had been produced in-place..... in a node scratch space that got deleted at the end of a batch job.

HTH

wmwv commented 5 years ago

I guess we could clean it to include only the general (and up-to-date) instructions, and create a different file to document the exact procedure used for each run/version. Is this related to what you're thinking of?

Yes, that's exactly the tension I'm concerned about. I think this is in part why @johannct isn't happy with the current README. It's useful guide to what was done, but it's not a full recreation of exactly what was done. It's also very much centered around doing things at NERSC.

This isn't going to be a complete re-working. But for this ticket I'll

[x] Add an overall introduction as a conceptual guide to what's happening.
[x] List tables as a pretty key Python dependency that can easily not be in an install.
[x] List GCRCatalogs and gcr-catalogs as a DESC-specific dependencies.

In a future ticket, @johannct can add instructions for how to run these at IN2P3.

wmwv commented 5 years ago

@johannct

I iterate here a remark I made before : it is not super pleasant to create products in-place, id est to be forced to do cd $WORKDIR in order to have the outputs be saved in $WORKDIR. I got tricked by that once, forgetting to move back outputs that had been produced in-place..... in a node scratch space that got deleted at the end of a batch job.

I think I understand the complaint, but what's your suggestion for positive action?

johannct commented 5 years ago

ah sorry, just a --outdir option maybe?

wmwv commented 5 years ago

ah sorry, just a --outdir option maybe?

Sounds reasonable to me. https://github.com/LSSTDESC/DC2-production/issues/322