Updates for (unrelased) DataLad 0.13.0

[kyleam]

This series updates run-related functionality for changes in the next feature release of DataLad. That version will contain some functionality that was prompted by issues on ReproMan's end, in particular improvements to datalad update and datalad create-sibling.

The changes above will require bumping our dependency to DataLad 0.13.0, so other commits in this series prune no longer needed compatibility kludges. In addition, 0.13 removes the deprecated datalad add command, so a commit here migrates to datalad save (which was held off to retain compatibility with DataLad 0.11.x line).

I'm marking this as a draft because (1) this shouldn't be merged until DataLad 0.13.0 is released and (2) this should resolve at least a few open issues, but I still need to reference them in the commit messages and add relevant tests. The latter will probably point out the need for at least minor tweaks.

[codecov[bot]]

Codecov Report

Merging #506 into master will increase coverage by 0.05%. The diff coverage is 96.47%.

@@            Coverage Diff             @@
##           master     #506      +/-   ##
==========================================
+ Coverage   89.60%   89.66%   +0.05%     
==========================================
  Files         148      148              
  Lines       12290    12388      +98     
==========================================
+ Hits        11013    11108      +95     
- Misses       1277     1280       +3     

Impacted Files	Coverage Δ
reproman/support/jobs/orchestrators.py	93.54% <88.88%> (+1.69%)	:arrow_up:
reproman/support/jobs/tests/test_orchestrators.py	94.89% <100.00%> (+1.15%)	:arrow_up:
reproman/support/exceptions.py	75.00% <0.00%> (-3.50%)	:arrow_down:
reproman/interface/jobs.py	96.11% <0.00%> (-2.89%)	:arrow_down:
reproman/interface/tests/test_run.py	98.46% <0.00%> (-1.15%)	:arrow_down:
reproman/interface/run.py	100.00% <0.00%> (ø)
reproman/support/jobs/submitters.py	79.25% <0.00%> (+1.04%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 214dbea...a1653bd. Read the comment docs.

[kyleam]

The Travis run isn't included in the checks box (at least at the moment), but here's the run:

https://travis-ci.org/github/ReproNim/reproman/builds/673081638

The build that covers the most run functionality is still in progress, but scanning a couple of runs that already failed, it looks like the failures are due to test_conda_init_install_and_detect failing. That's unlikely to be related to this PR (*) and is probably a failure we need to address on master. (Perhaps this is related to gh-446, and the workaround from gh-443 is no longer sufficient.)

(*) edit: As expected, the test_conda_init_install_and_detect failure is showing up after the latest push to gh-505.

[kyleam]

Updated to resolve conflicts with master.

range-diff

``` 1: 32938c829 = 1: 4345efaf6 TST: travis: Install development version of DataLad 2: afecf0060 ! 2: 7192205d4 RF: run: Use 'datalad save' instead of 'datalad add' @@ reproman/support/jobs/tests/test_orchestrators.py: def dataset(tmpdir_factory): return ds @@ reproman/support/jobs/tests/test_orchestrators.py: def run_and_check(spec): - # The submitter log file is ignored (currently only relevant for - # condor; see b9277ebc0 for more details). Add the directory to get to - # a clean state. -- dataset.add(".reproman") -+ dataset.save(".reproman") - orc = run_and_check(dumped_spec) - + # The submitter log file is ignored (currently only relevant for + # condor; see b9277ebc0 for more details). Add the directory to get + # to a clean state. +- dataset.add(".reproman") ++ dataset.save(".reproman") + orc = run_and_check(dumped_spec) + return fn @@ reproman/support/jobs/tests/test_orchestrators.py: def test_orc_datalad_run_change_head(job_spec, dataset, shell): 3: b413335f7 = 3: db477e416 RF: orchestrators: Prune now unneeded SSH URL compatibility kludge 4: f3effe526 = 4: ace092835 RF: orchestrators: Adjust datalad run imports 5: 20c6b9125 = 5: fe302e8a2 MNT: orchestrators: Drop warning about an unsupported DataLad version 6: 4f3e114f1 < -: --------- RF: test_orchestrators: Use DataLad's repo.call_git() -: --------- > 6: b87cbe762 RF: test_orchestrators: Use DataLad's repo.call_git() 7: ba6ca4bb6 ! 7: 493ac8533 ENH: orchestrators: Meld handling of local and SSH sessions @@ reproman/support/jobs/orchestrators.py: def fetch(self, on_remote_finish=None): ## reproman/support/jobs/tests/test_orchestrators.py ## @@ reproman/support/jobs/tests/test_orchestrators.py: def run_and_check(spec): - # Our reproman-based copying of data doesn't isn't (yet) OK with data - # files that already exist. - dumped_spec["inputs"] = [] -- dataset.repo.call_git(["reset", "--hard", "start-pt"]) -+ dataset.repo.call_git(["checkout", "-b", "other", "start-pt"]) - if dataset.repo.dirty: - # The submitter log file is ignored (currently only relevant for - # condor; see b9277ebc0 for more details). Add the directory to get to + # Our reproman-based copying of data doesn't isn't (yet) OK with + # data files that already exist. + dumped_spec["inputs"] = [] +- dataset.repo.call_git(["reset", "--hard", "start-pt"]) ++ dataset.repo.call_git(["checkout", "-b", "other", "start-pt"]) + if dataset.repo.dirty: + # The submitter log file is ignored (currently only relevant for + # condor; see b9277ebc0 for more details). Add the directory to get 8: da5bd5f4d = 8: 7c9e6cee8 BF: orchestrators: Initialize submodules 9: a41694f1e = 9: 3ff339655 ENH: orchestrators: Fetch with 'update --follow=parentds' 10: 776ffb290 = 10: a902ac9cc BF: orchestrators: Fetch with --recurse-submodules=no 11: 190abdb1e = 11: c4347675a [todo] Add placeholder comment ```