search

datalad / datalad-osf

DataLad extension to interface with the Open Science Framework
Other
14 stars 12 forks source link

Datalad OSF sibling not working #202

Closed stebo85 closed 2 months ago

stebo85 commented 2 months ago

What is the problem?

datalad push --to osf fails with:

osf git-annex' failed with exitcode 128                                                           | 2.00/4.00 [00:00<00:00, 4.72k Steps/s]
[DATALAD-ANNEX] fatal: couldn't find remote refs (repository deposit does not exist, or is inaccessible```

What steps will reproduce the problem?

datalad create test
cd test
touch test
datalad osf-credentials
datalad save -m "added new files"
datalad create-sibling-osf --title best-study-ever-test -s osf
datalad push --to osf

DataLad information

[WARNING] Could not determine filesystem types due to KeyError(None) 
# WTF
## configuration <SENSITIVE, report disabled by configuration>
## credentials 
  - keyring: 
    - active_backends: 
      - PlaintextKeyring with no encyption v.1.0 at /home/jovyan/.local/share/python_keyring/keyring_pass.cfg
    - config_file: /home/jovyan/.config/python_keyring/keyringrc.cfg
    - data_root: /home/jovyan/.local/share/python_keyring
## datalad 
  - version: 1.1.3
## dataset 
  - branches: 
    - git-annex@a9ea447
    - master@36baec9
  - id: 479d5b8e-0946-496a-9205-35d1b966eb32
  - path: /home/jovyan/test
  - repo: AnnexRepo
## dependencies 
  - annexremote: 1.6.5
  - boto3: 1.35.15
  - cmd:7z: 16.02
  - cmd:annex: 10.20221003
  - cmd:bundled-git: UNKNOWN
  - cmd:git: 2.34.1
  - cmd:ssh: 8.9p1
  - cmd:system-git: 2.34.1
  - cmd:system-ssh: 8.9p1
  - git: 3.1.43
  - gitdb: 4.0.11
  - humanize: 4.10.0
  - iso8601: 2.1.0
  - keyring: 25.3.0
  - keyrings.alt: 5.0.2
  - msgpack: 1.0.8
  - platformdirs: 3.11.0
  - requests: 2.32.3
## environment 
  - GIT_PYTHON_REFRESH: quiet
  - LANG: 
  - LANGUAGE: 
  - LC_ALL: 
  - LC_CTYPE: C.UTF-8
  - PATH: /opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jovyan/.local/bin:/opt/conda/bin:/opt/conda/condabin
  - PYTHONPATH: 
## extensions 
  - container: 
    - description: Containerized environments
    - entrypoints: 
      - datalad_container.containers_add.ContainersAdd: 
        - class: ContainersAdd
        - module: datalad_container.containers_add
        - names: 
          - containers-add
          - containers_add
      - datalad_container.containers_list.ContainersList: 
        - class: ContainersList
        - module: datalad_container.containers_list
        - names: 
          - containers-list
          - containers_list
      - datalad_container.containers_remove.ContainersRemove: 
        - class: ContainersRemove
        - module: datalad_container.containers_remove
        - names: 
          - containers-remove
          - containers_remove
      - datalad_container.containers_run.ContainersRun: 
        - class: ContainersRun
        - module: datalad_container.containers_run
        - names: 
          - containers-run
          - containers_run
    - module: datalad_container
    - version: 1.2.5
  - next: 
    - description: What is next in DataLad
    - entrypoints: 
      - datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV: 
        - class: CreateSiblingWebDAV
        - module: datalad_next.commands.create_sibling_webdav
        - names: 
          - create-sibling-webdav
      - datalad_next.commands.credentials.Credentials: 
        - class: Credentials
        - module: datalad_next.commands.credentials
        - names: 
      - datalad_next.commands.download.Download: 
        - class: Download
        - module: datalad_next.commands.download
        - names: 
          - download
      - datalad_next.commands.ls_file_collection.LsFileCollection: 
        - class: LsFileCollection
        - module: datalad_next.commands.ls_file_collection
        - names: 
          - ls-file-collection
      - datalad_next.commands.status.Status: 
        - class: Status
        - module: datalad_next.commands.status
        - names: 
          - next-status
          - next_status
      - datalad_next.commands.tree.TreeCommand: 
        - class: TreeCommand
        - module: datalad_next.commands.tree
        - names: 
          - tree
    - module: datalad_next
    - version: 1.5.0
  - osf: 
    - description: DataLad extension for OSF support
    - entrypoints: 
      - datalad_osf.create_sibling_osf.CreateSiblingOSF: 
        - class: CreateSiblingOSF
        - module: datalad_osf.create_sibling_osf
        - names: 
          - create-sibling-osf
          - create_sibling_osf
      - datalad_osf.credentials.OSFCredentials: 
        - class: OSFCredentials
        - module: datalad_osf.credentials
        - names: 
          - osf-credentials
          - osf_credentials
    - module: datalad_osf
    - version: 0.3.0
## git-annex 
  - build flags: 
    - Assistant
    - Webapp
    - Pairing
    - Inotify
    - DBus
    - DesktopNotify
    - TorrentParser
    - MagicMime
    - Benchmark
    - Feeds
    - Testsuite
    - S3
    - WebDAV
  - dependency versions: 
    - aws-0.22
    - bloomfilter-2.0.1.0
    - cryptonite-0.26
    - DAV-1.3.4
    - feed-1.3.0.1
    - ghc-8.8.4
    - http-client-0.6.4.1
    - persistent-sqlite-2.10.6.2
    - torrent-10000.1.1
    - uuid-1.3.13
    - yesod-1.6.1.0
  - key/value backends: 
    - SHA256E
    - SHA256
    - SHA512E
    - SHA512
    - SHA224E
    - SHA224
    - SHA384E
    - SHA384
    - SHA3_256E
    - SHA3_256
    - SHA3_512E
    - SHA3_512
    - SHA3_224E
    - SHA3_224
    - SHA3_384E
    - SHA3_384
    - SKEIN256E
    - SKEIN256
    - SKEIN512E
    - SKEIN512
    - BLAKE2B256E
    - BLAKE2B256
    - BLAKE2B512E
    - BLAKE2B512
    - BLAKE2B160E
    - BLAKE2B160
    - BLAKE2B224E
    - BLAKE2B224
    - BLAKE2B384E
    - BLAKE2B384
    - BLAKE2BP512E
    - BLAKE2BP512
    - BLAKE2S256E
    - BLAKE2S256
    - BLAKE2S160E
    - BLAKE2S160
    - BLAKE2S224E
    - BLAKE2S224
    - BLAKE2SP256E
    - BLAKE2SP256
    - BLAKE2SP224E
    - BLAKE2SP224
    - SHA1E
    - SHA1
    - MD5E
    - MD5
    - WORM
    - URL
    - X*
  - local repository version: 10
  - operating system: linux x86_64
  - remote types: 
    - git
    - gcrypt
    - p2p
    - S3
    - bup
    - directory
    - rsync
    - web
    - bittorrent
    - webdav
    - adb
    - tahoe
    - glacier
    - ddar
    - git-lfs
    - httpalso
    - borg
    - hook
    - external
  - supported repository versions: 
    - 8
    - 9
    - 10
  - upgrade supported from repository versions: 
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    - 8
    - 9
    - 10
  - version: 10.20221003
## location 
  - path: /home/jovyan/test
  - type: dataset
## metadata.extractors 
  - container_inspect: 
    - distribution: datalad_container 1.2.5
    - load_error: ModuleNotFoundError(No module named 'datalad_metalad')
    - module: datalad_container.extractors.metalad_container
## metadata.filters 
## metadata.indexers 
## python 
  - implementation: CPython
  - version: 3.11.6
## system 
  - distribution: ubuntu/22.04/jammy
  - encoding: 
    - default: utf-8
    - filesystem: utf-8
    - locale.prefered: UTF-8
  - filesystem: 
    - CWD: 
      - max_pathlength: 4096
      - mount_opts: rw,relatime
      - path: /home/jovyan/test
      - type: ext4
    - HOME: 
      - max_pathlength: 4096
      - mount_opts: rw,relatime
      - path: /home/jovyan
      - type: ext4
    - TMP: 
      - path: /tmp
  - max_path_length: 273
  - name: Linux
  - release: 5.15.0-112-generic
  - type: posix
  - version: datalad/datalad#122-Ubuntu SMP Thu May 23 07:48:21 UTC 2024

Additional context

No response

Have you had any success using DataLad before?

yes, the same workflow worked a few months ago :)

adswa commented 2 months ago

Many thanks for your detailed issue! Upon a first try in a fresh virtual environment, I wasn't able to reproduce this problem (thanks much for the reproducer!):

(osf) adina@muninn in /tmp
❱ datalad create test
create(ok): /tmp/test (dataset)
(osf) adina@muninn in /tmp
❱ cd test
(osf) adina@muninn in /tmp/test on git:master
❱ touch test
(osf) adina@muninn in /tmp/test on git:master
❱ datalad osf-credentials
osf_credentials(ok): [authenticated as Adina Svenja Wagner <adina.wagner@t-online.de>]
(osf) adina@muninn in /tmp/test on git:master
❱ datalad save -m "added new files"
add(ok): test (file)                                                            
save(ok): . (dataset)                                                           
action summary:                                                                 
  add (ok: 1)
  save (ok: 1)
(osf) adina@muninn in /tmp/test on git:master
❱ datalad create-sibling-osf --title best-study-ever-test -s osf
create-sibling-osf(ok): https://osf.io/bhkvq/
[INFO   ] Configure additional publication dependency on "osf-storage" 
configure-sibling(ok): . (sibling)
(osf) adina@muninn in /tmp/test on git:master
❱ datalad push --to osf
copy(ok): test (file) [to osf-storage...]                                       
publish(ok): . (dataset) [refs/heads/master->osf:refs/heads/master [new branch]]
publish(ok): . (dataset) [refs/heads/git-annex->osf:refs/heads/git-annex [new branch]]                                                                          
                                                                               action summary:                                                                  
  copy (ok: 1)
  publish (ok: 2)
(osf) adina@muninn in /tmp/test on git:master

I wonder if its the specific git-annex version that's the culprit - I wont get to it in the next hour, but I will later today try toying with different annex versions.

stebo85 commented 2 months ago

Yes, I think it’s related to the git annex version because it was working with the standard git annex version in Ubuntu 22.04 and datalad 0.7 from memory - but datalad 1.0 wasn’t compatible with that git annex version anymore, so after upgrading the git annex version using the new git annex version from neurodebian this problem appeared.

Thank you

Steffen

--

W: https://mri.sbollmann.nethttps://mri.sbollmann.net/ | W: https://www.neurodesk.orghttps://www.neurodesk.org/ | T: https://twitter.com/sbollmann_MRI | G: https://github.com/stebo85

Book meeting: https://calendly.com/s-bollmann/meeting

From: Adina Wagner @.> Sent: Friday, September 13, 2024 5:21:05 PM To: datalad/datalad @.> Cc: Steffen Bollmann @.>; Author @.> Subject: Re: [datalad/datalad] Datalad OSF sibling not working (Issue datalad/datalad-osf#202)

Many thanks for your detailed issue! Upon a first try in a fresh virtual environment, I wasn't able to reproduce this problem (thanks much for the reproducer!):

(osf) @. in /tmp ❱ datalad create test create(ok): /tmp/test (dataset) (osf) @. in /tmp ❱ cd test (osf) @. in /tmp/test on git:master ❱ touch test (osf) @. in /tmp/test on git:master ❱ datalad osf-credentials osf_credentials(ok): [authenticated as Adina Svenja Wagner @.>] (osf) @. in /tmp/test on git:master ❱ datalad save -m "added new files" add(ok): test (file) save(ok): . (dataset) action summary: add (ok: 1) save (ok: 1) (osf) @. in /tmp/test on git:master ❱ datalad create-sibling-osf --title best-study-ever-test -s osf create-sibling-osf(ok): https://osf.io/bhkvq/ [INFO ] Configure additional publication dependency on "osf-storage" configure-sibling(ok): . (sibling) (osf) @. in /tmp/test on git:master ❱ datalad push --to osf copy(ok): test (file) [to osf-storage...] publish(ok): . (dataset) [refs/heads/master->osf:refs/heads/master [new branch]] publish(ok): . (dataset) [refs/heads/git-annex->osf:refs/heads/git-annex [new branch]] action summary: copy (ok: 1) publish (ok: 2) (osf) @.*** in /tmp/test on git:master

I wonder if its the specific git-annex version that's the culprit - I wont get to it in the next hour, but I will later today try toying with different annex versions.

— Reply to this email directly, view it on GitHubhttps://github.com/datalad/datalad-osf/issues/202, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA6V2W35W4L5DVQRUXHRTBTZWKG6DAVCNFSM6AAAAABOEUIMHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBYGIYTIMBZGU. You are receiving this because you authored the thread.Message ID: @.***>

adswa commented 2 months ago

hey, I got it - can you check if you have the datalad-next extension configured to be "loaded"? Its a bug from our side, though, for some reason we haven't included it in the docs... and at the moment, it is still a necessary step.

I'll transfer this issue to datalad-osf, and add it to the docs.

try running:

git config --global --add datalad.extensions.load next

and then retry push

stebo85 commented 2 months ago

wonderful - that worked perfectly now :)