search

NeoGeographyToolkit / StereoPipeline

The NASA Ames Stereo Pipeline is a suite of automated geodesy & stereogrammetry tools designed for processing planetary imagery captured from orbiting and landed robotic explorers on other planets.
Apache License 2.0
492 stars 173 forks source link

add more configuration of gnu_parallel in parallel scripts #335

Closed AndrewAnnex closed 3 years ago

AndrewAnnex commented 3 years ago

Is your feature request related to a problem? Please describe. Recently I've been experimenting with using ASP in a SLURM cluster and I have found that ASP makes a few choices on using gnu_parallel that I would like to have more control over.

Describe the solution you'd like I think that the parallel commands, mainly parallel_stereo should add a few options:

  1. a "dry-run" mode that would just print out the individual tile job commands that gnu_parallel would run for the user to allow users options to delegate in other ways like creating individual slurm jobs per tile. Depending on how different cluster layouts it may be easier/faster/more resource efficient to allocate many small/fast jobs rather than keeping larger nodes running.
  2. a way to pass in additional optional parameters like "--sshdelay" or "--controlmaster" to the call to parallel, ideally accept any keyword or flags specified in https://www.gnu.org/software/parallel/parallel_tutorial.html with no validation

I haven't looked at all of the parallel scripts so I don't know how much shared code their is between them, but parallel_sfs would likely benefit as well.

oleg-alexandrov commented 3 years ago

Andrew, since you are the amazing Python and ASP power user expert I should be learning from, it will be honor if you implement these and submit a pull request. :)

We actually have a --dry-run option. For the second item, maybe a new option called --parallel-options "--opt1 --opt2" can be added which will be passed to GNU Parallel.

oleg-alexandrov commented 3 years ago

Apparently the --dry-run option was not documented, which I just rectified. Oh, and I would not be concerned with parallel_sfs much, that one is a more specialized tool than parallel_stereo.

AndrewAnnex commented 3 years ago

@oleg-alexandrov well I guess it was worth a try ;) I'll maybe be able to find a free weekend soon but I also just wanted to ensure this idea was documented for others to see.