dandi / dandisets

735 Dandisets, 812.2 TB total. DataLad super-dataset of all Dandisets from https://github.com/dandisets
10 stars 0 forks source link

Set web URL priorities when initting a dataset #323

Closed jwodder closed 1 year ago

jwodder commented 1 year ago

Closes #320.

Blocked by the lack of git-annex 10.20230126 in conda-forge.

yarikoptic commented 1 year ago

related issue: https://github.com/conda-forge/git-annex-feedstock/issues/152 . I will touch base with @notestaff but we might need to help there. The tricky part is that "nodep" build for which I forgot the "workflow" to get it updated.

yarikoptic commented 1 year ago

tests fail since need recent git-annex release. I will build one now for neurodebian.

yarikoptic commented 1 year ago

now installs most recent one but still fails

2023-02-01T16:29:03.3153839Z Preparing to unpack .../074-git-annex-standalone_10.20230126-1~ndall+1_amd64.deb ...
2023-02-01T16:29:03.3165031Z Unpacking git-annex-standalone (10.20230126-1~ndall+1) ...

I do not see why ... is there stderr ?

jwodder commented 1 year ago

@yarikoptic The current failures appear to be due to "bogus format in GIT_CONFIG_PARAMETERS". For the record, that envvar is set as follows when running datalad create:

https://github.com/dandi/dandisets/blob/199dbbfc495c07f4f74c1f617bed6d8dce7768d6/tools/backups2datalad/adataset.py#L73

yarikoptic commented 1 year ago

Iirc, this is happening with standalone build when system git is too old, I will later push a try to fix workaround which would export assisted PATH...

codecov[bot] commented 1 year ago

Codecov Report

Base: 76.19% // Head: 75.84% // Decreases project coverage by -0.36% :warning:

Coverage data is based on head (c31e763) compared to base (50d5167). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## draft #323 +/- ## ========================================== - Coverage 76.19% 75.84% -0.36% ========================================== Files 14 14 Lines 2399 2401 +2 Branches 381 381 ========================================== - Hits 1828 1821 -7 - Misses 417 426 +9 Partials 154 154 ``` | [Impacted Files](https://codecov.io/gh/dandi/dandisets/pull/323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | Coverage Δ | | |---|---|---| | [tools/backups2datalad/adataset.py](https://codecov.io/gh/dandi/dandisets/pull/323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL2FkYXRhc2V0LnB5) | `79.31% <100.00%> (+0.07%)` | :arrow_up: | | [tools/backups2datalad/aioutil.py](https://codecov.io/gh/dandi/dandisets/pull/323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL2Fpb3V0aWwucHk=) | `72.77% <0.00%> (-3.02%)` | :arrow_down: | | [tools/backups2datalad/logging.py](https://codecov.io/gh/dandi/dandisets/pull/323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL2xvZ2dpbmcucHk=) | `85.00% <0.00%> (-2.50%)` | :arrow_down: | | [tools/backups2datalad/util.py](https://codecov.io/gh/dandi/dandisets/pull/323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-dG9vbHMvYmFja3VwczJkYXRhbGFkL3V0aWwucHk=) | `77.14% <0.00%> (-1.43%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

yarikoptic commented 1 year ago

yippee -- it is green, I thought to just uninstall git-annex in dandisets conda and proceed BUT datalad depends on git-annex so I can't uninstall it :-/ so - although ready, we should wait until we manage to build fresh git-annex in conda-forge.

yarikoptic commented 1 year ago

FWIW -- we have (standalone) build of recent git-annex installed already in new dandisets-2 environment. Let's proceed! I will manually go through existing ones with e.g.

 for d in 000*; do  git -C $d annex initremote --sameas=web dandiapi type=web urlinclude='*//api.dandiarchive.org/*' cost=300; done

and now through dandizarrs

yarikoptic commented 1 year ago

note -- I will also need to push git-annex branch to github... should have added to the loop