ReproNim / reproman

ReproMan (AKA NICEMAN, AKA ReproNim TRD3)
https://reproman.readthedocs.io
Other
24 stars 14 forks source link

datalad 0.14.0rc1 breaks test_orc_datalad_pair_new_submodule #568

Closed yarikoptic closed 3 years ago

yarikoptic commented 3 years ago

discovered while testing https://github.com/ReproNim/reproman/pull/567

copy/paste of output for test_orc_datalad_pair_new_submodule from raw log ```shell INFO  datalad.gitrepo:log.py:323 Finished writing objects _____________________ test_orc_datalad_pair_new_submodule ______________________ job_spec = {'_command_array': ["sh -c 'echo a >sub/a'"], '_extra_inputs_array': [[]], '_inputs_array': [[]], '_outputs_array': [[]], ...} dataset = Dataset('/tmp/pytest-of-travis/pytest-0/dataset33') shell = Shell(name='localshell', id=None, type='shell', status='available') def test_orc_datalad_pair_new_submodule(job_spec, dataset, shell): with chpwd(dataset.path): orc = orcs.DataladPairOrchestrator( shell, submission_type="local", job_spec=job_spec) orc.prepare_remote() orc.submit() orc.follow() orc.fetch() # prepare_remote() doesn't fail when a new subdataset is added after # the first run. sub = dataset.create("sub") dataset.save() job_spec["_resolved_command_str"] = "sh -c 'echo a >sub/a'" job_spec["inputs"] = [] job_spec["outputs"] = [] orc = orcs.DataladPairOrchestrator( shell, submission_type="local", job_spec=job_spec) > orc.prepare_remote() reproman/support/jobs/tests/test_orchestrators.py:867: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ reproman/support/jobs/orchestrators.py:830: in prepare_remote self._fix_up_dataset() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def _fix_up_dataset(self): """Try to get datataset and subdatasets into the correct state.  """ self._checkout_target() # fixup 0: 'datalad create-sibling --recursive' leaves the subdataset # uninitialized (see DataLad's 78e00dcd2). self._execute_in_wdir(["git", "submodule", "update", "--init"]) # fixup 1: Check out target commit in subdatasets. This should later be # replaced by the planned Datalad functionality to get an entire # dataset hierarchy to a recorded state. # # fixup 2: Autoenable remotes: # 'datalad publish' does not autoenable remotes, and 'datalad # create-sibling' calls 'git annex init' too early to trigger # autoenabling. Temporarily work around this issue, though this # should very likely be addressed in DataLad. And if this is here # to stay, we should avoid this call for non-annex datasets. lgr.info("Adjusting state of remote dataset") self._execute_in_wdir(["git", "annex", "init"]) for res in self._execute_datalad_json_command( ["subdatasets", "--fulfilled=true", "--recursive"]): cwd = res["path"] self._assert_clean_repo(cwd=cwd) lgr.debug("Adjusting state of %s", cwd) > cmds = [["git", "checkout", res["revision"]], ["git", "annex", "init"]] E KeyError: 'revision' reproman/support/jobs/orchestrators.py:754: KeyError ```

we installed rc:

Collecting datalad>=0.13.0rc2
  Downloading datalad-0.14.0rc1.tar.gz (1.6 MB)

It also failed similarly in recent master run but was all green a few days back (whenever there were no datalad rc) https://travis-ci.org/github/ReproNim/reproman/builds/755466822 . So -- must be datalad rc related .