dandi / dandisets

760 Dandisets, 817.2 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

reports dataset being dirty whenever it is not #204

Closed yarikoptic closed 2 years ago

yarikoptic commented 2 years ago

might be a bug in datalad? @jwodder please help -- I am blind

(base) dandi@drogon:/mnt/backup/dandi/dandisets$ tools/backups2datalad-update-cron-108
> eval python -m tools.backups2datalad -l WARNING --pdb --backup-root . --config tools/backups2datalad.cfg.yaml update-from-backup 000108
>> python -m tools.backups2datalad -l WARNING --pdb --backup-root . --config tools/backups2datalad.cfg.yaml update-from-backup 000108
A newer version (0.40.1) of dandi/dandi-cli is available. You are using 0.40.0
Traceback (most recent call last):
  File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 354, in <module>
    main()
  File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 132, in update_from_backup
    datasetter.update_from_backup(dandisets, exclude=exclude)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 75, in update_from_backup
    changed = self.sync_dataset(d, ds)
  File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 129, in sync_dataset
    raise RuntimeError(f"Dirty {dandiset}; clean or save before running")
RuntimeError: Dirty DANDI:000108/draft; clean or save before running

> /mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py(129)sync_dataset()
-> raise RuntimeError(f"Dirty {dandiset}; clean or save before running")
(Pdb) c
(base) dandi@drogon:/mnt/backup/dandi/dandisets$ git -C 000108 status
On branch draft
Your branch is up to date with 'github/draft'.

nothing to commit, working tree clean
(base) dandi@drogon:/mnt/backup/dandi/dandisets$ git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 4 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 462 bytes | 462.00 KiB/s, done.
Total 5 (delta 4), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To github.com:dandi/dandisets.git
   ad539eb..e548d57  draft -> draft

NB I just added including dataset which is found dirty in the exception msg, otherwise was not sure what it finds dirty

yarikoptic commented 2 years ago

uff... because RF did screw up in paths specification -- will need to see to have it adjusted and dandiset/dandiset "prevented" ...

(base) dandi@drogon:/mnt/backup/dandi/dandisets$ git -C /mnt/backup/dandi/dandisets/dandisets/000108 status | head
On branch draft
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        new file:   dandiset.yaml
        new file:   dataset_description.json
        new file:   samples.tsv
        new file:   sub-MITU01/ses-20210521h17m17s06/micr/sub-MITU01_sample-178_photo.jpg
        new file:   sub-MITU01/ses-20210521h17m17s06/micr/sub-MITU01_ses-20210521h17m17s06_sample-178_stain-LEC_run-1_chunk-1_SPIM.json
        new file:   sub-MITU01/ses-20210521h17m17s06/micr/sub-MITU01_ses-20210521h17m17s06_sample-178_stain-LEC_run-1_chunk-2_SPIM.json
        new file:   sub-MITU01/ses-20210521h17m17s06/micr/sub-MITU01_ses-20210521h17m17s06_sample-178_stain-LEC_run-1_chunk-3_SPIM.json
...
yarikoptic commented 2 years ago

af10b63bd29c0bba59832b49625399dfabd9e981 should have fixed that... apparently dirname . is .