Closed kevinCassou closed 6 days ago
Hi, thanks for the issue! Could you let me know which version of datalad-next is installed (e.g. with pip freeze | grep next
)?
Hi thanks for quick comment !
pip freeze | grep next
datalad-next==1.5.0
Yes, in fact the datalad
extension is installed.
so I try
pip uninstall datalad-next
then
datalad create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/data_store/
[INFO ] create siblings 'silverTemp' and 'silverTemp-storage' ... [INFO ] Fetching updates for Dataset(/temp/sandbox/mydataset) update(ok): . (dataset) update(ok): . (dataset) [INFO ] Configure additional publication dependency on "silverTemp-storage" configure-sibling(ok): . (sibling) create-sibling-ria(ok): /sandbox/mydataset (dataset) action summary: configure-sibling (ok: 1) create-sibling-ria (ok: 1) update (ok: 1) 0.00 [00:04, ?/s]%
checking
datalad siblings
.: here(+) [git] .: silverTemp(-) [ssh://user@server/silver/data_store/b1d/10d7e-64b0-4d60-a4f2-9a14d366dfb0 (git)] .: silverTemp-storage(+) [ora]
👍 Thanks ! So datalad-next is not compatible with the RIA store feature ?
Thanks ! So datalad-next is not compatible with the RIA store feature ?
Actually, it should be. Quite the opposite, it should improve RIA functionality - so what you are seeing is certainly a bug. datalad-next
patches the RIA/ORA code in datalad
with the url2transport_path
method to provide support for Windows. I will transfer this issue to datalad-next for for further exploration.
@christian-monch do you maybe have an immediate intuition what is going on here (given that you wrote #669)? It looks like SSHRemoteIO
somehow has not (yet?) gotten patched when next
-patched ORA code calls url2transport_path
? I haven't found a way to reproduce this, though.
Ah ok, I was testing from a Mac arm64 with python=Python 3.10.6 Thanks for all
Hi @kevinCassou thanks for the report. I tried to reproduce the error locally but did not yet succeed. The commands you provided worked for me.
Would it be possible for you to set the environment variable DATALAD_LOG_LEVEL
to debug
, run the same command again, and post the output here?
Something like:
> export DATALAD_LOG_LEVEL=debug
> datalad create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/data_store/
Hi, So try to reproduce it:
conda create -n dltest python=3.10
conda activate dltest
conda install -c conda-forge datalad
python -m pip install datalad-next
datalad --version
datalad 1.1.3
then creating a dataset for test :
datalad create -c text2git mydataset01
.. adding some files
datalad save -m'first files add'
add(ok): README.md (file)
add(ok): data/spec1-test1.pickle (file)
add(ok): test.txt (file)
save(ok): . (dataset)
action summary:
add (ok: 3)
save (ok: 1)
then
export DATA_LOG_LEVEL=debug
datalad create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/temp/data_store/
and I got
[DEBUG ] Not retro-fitting GitRepo with deprecated symbols, datalad-deprecated package not found
[DEBUG ] Command line args 1st pass for DataLad 1.1.3. Parsed: Namespace() Unparsed: ['create-sibling-ria', '-s', 'silverTemp', '--new-store-ok', 'ria+ssh://user@server/silver/temp/data_store/']
[DEBUG ] Building doc for <class 'datalad.core.distributed.clone.Clone'>
[DEBUG ] Building doc for <class 'datalad.distributed.create_sibling_ria.CreateSiblingRia'>
[DEBUG ] Parsing known args among ['/opt/miniconda3/envs/py39/dltest/bin/datalad', 'create-sibling-ria', '-s', 'silverTemp', '--new-store-ok', 'ria+ssh://user@server/silver/temp/data_store/']
[DEBUG ] Determined class of decorated function: <class 'datalad.distributed.create_sibling_ria.CreateSiblingRia'>
[DEBUG ] Resolved dataset to create RIA sibling(s): /Users/user/temp/sandbox/mydataset01
[DEBUG ] Run ['git', 'version'] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG ] Finished ['git', 'version'] with status 0
[DEBUG ] Run ['git', 'config', '-z', '-l', '--show-origin'] (protocol_class=StdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG ] Finished ['git', 'config', '-z', '-l', '--show-origin'] with status 0
[DEBUG ] Run ['git', 'config', '-z', '-l', '--show-origin', '--file', '/Users/user/temp/sandbox/mydataset01/.datalad/config'] (protocol_class=StdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG ] Finished ['git', 'config', '-z', '-l', '--show-origin', '--file', '/Users/user/temp/sandbox/mydataset01/.datalad/config'] with status 0
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'rev-parse', '--quiet', '--verify', 'HEAD^{commit}'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:remote.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:remote.log' failed with exitcode 128 [err: 'fatal: path 'remote.log' does not exist in 'git-annex'']
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:trust.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:trust.log' failed with exitcode 128 [err: 'fatal: path 'trust.log' does not exist in 'git-annex'']
[DEBUG ] Opening MultiplexSSHConnection(ctrl_path=<<PosixPath('/Us++48 chars++4e')>>, sshri=<<SSHRI(hostname++46 chars++ou')>>) by calling ['ssh', '-fN', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=15m', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server']
[DEBUG ] Importing datalad.api to possibly discover possibly not yet bound method 'siblings'
[DEBUG ] Building doc for <class 'datalad.core.local.create.Create'>
[DEBUG ] Building doc for <class 'datalad.core.local.status.Status'>
[DEBUG ] Building doc for <class 'datalad.core.local.save.Save'>
[DEBUG ] Building doc for <class 'datalad.local.subdatasets.Subdatasets'>
[DEBUG ] Building doc for <class 'datalad.distribution.get.Get'>
[DEBUG ] Building doc for <class 'datalad.core.local.diff.Diff'>
[DEBUG ] Building doc for <class 'datalad.core.distributed.push.Push'>
[DEBUG ] Building doc for <class 'datalad.distribution.install.Install'>
[DEBUG ] Building doc for <class 'datalad.local.unlock.Unlock'>
[DEBUG ] Building doc for <class 'datalad.core.local.run.Run'>
[DEBUG ] Building doc for <class 'datalad.distributed.create_sibling_github.CreateSiblingGithub'>
[DEBUG ] Building doc for <class 'datalad.distribution.update.Update'>
[DEBUG ] Building doc for <class 'datalad.distribution.siblings.Siblings'>
[DEBUG ] Building doc for <class 'datalad.distributed.create_sibling_gitlab.CreateSiblingGitlab'>
[DEBUG ] Building doc for <class 'datalad.distributed.create_sibling_gogs.CreateSiblingGogs'>
[DEBUG ] Building doc for <class 'datalad.distributed.create_sibling_gin.CreateSiblingGin'>
[DEBUG ] Building doc for <class 'datalad.distributed.create_sibling_gitea.CreateSiblingGitea'>
[DEBUG ] Building doc for <class 'datalad.distribution.create_sibling.CreateSibling'>
[DEBUG ] Building doc for <class 'datalad.distributed.drop.Drop'>
[DEBUG ] Building doc for <class 'datalad.local.remove.Remove'>
[DEBUG ] Building doc for <class 'datalad.local.addurls.Addurls'>
[DEBUG ] Building doc for <class 'datalad.local.copy_file.CopyFile'>
[DEBUG ] Building doc for <class 'datalad.local.download_url.DownloadURL'>
[DEBUG ] Building doc for <class 'datalad.local.foreach_dataset.ForEachDataset'>
[DEBUG ] Building doc for <class 'datalad.local.rerun.Rerun'>
[DEBUG ] Building doc for <class 'datalad.local.run_procedure.RunProcedure'>
[DEBUG ] Building doc for <class 'datalad.local.configuration.Configuration'>
[DEBUG ] Building doc for <class 'datalad.local.wtf.WTF'>
[DEBUG ] Building doc for <class 'datalad.local.clean.Clean'>
[DEBUG ] Building doc for <class 'datalad.local.add_archive_content.AddArchiveContent'>
[DEBUG ] Building doc for <class 'datalad.local.add_readme.AddReadme'>
[DEBUG ] Building doc for <class 'datalad.local.export_archive.ExportArchive'>
[DEBUG ] Building doc for <class 'datalad.distributed.export_archive_ora.ExportArchiveORA'>
[DEBUG ] Building doc for <class 'datalad.distributed.export_to_figshare.ExportToFigshare'>
[DEBUG ] Building doc for <class 'datalad.local.no_annex.NoAnnex'>
[DEBUG ] Building doc for <class 'datalad.local.check_dates.CheckDates'>
[DEBUG ] Building doc for <class 'datalad.distribution.uninstall.Uninstall'>
[DEBUG ] Building doc for <class 'datalad.distribution.create_test_dataset.CreateTestDataset'>
[DEBUG ] Building doc for <class 'datalad.support.sshrun.SSHRun'>
[DEBUG ] Building doc for <class 'datalad.interface.shell_completion.ShellCompletion'>
[DEBUG ] Processing entrypoints
[DEBUG ] Loading entrypoint next from datalad.extensions
[DEBUG ] Enable posting DataLad config overrides CLI/ENV as GIT_CONFIG items in process ENV
[DEBUG ] Apply datalad-next patch to annexrepo.py:AnnexRepo.enable_remote
[DEBUG ] Building doc for <class 'datalad_next.patches.configuration.Configuration'>
[DEBUG ] Building doc for <class 'datalad.local.configuration.Configuration'>
[DEBUG ] Apply datalad-next patch to create_sibling_ghlike.py:_GitHubLike._set_request_headers
[DEBUG ] Apply datalad-next patch to interface.(utils|base).py:_execute_command_
[DEBUG ] Apply patch to datalad.core.distributed.push._transfer_data
[DEBUG ] Patching datalad.core.distributed.push.Push docstring and parameters
[DEBUG ] Building doc for <class 'datalad.core.distributed.push.Push'>
[DEBUG ] Patching datalad.support.AnnexRepo.get_export_records (new method)
[DEBUG ] Apply patch to datalad.core.distributed.push._push
[DEBUG ] Apply patch to datalad.distribution.siblings._enable_remote
[DEBUG ] Retrofit `SpecialRemote` with a `close()` handler
[DEBUG ] Replace special remote _main() with datalad-next's progress logging enabled variant
[DEBUG ] Apply patch to datalad.distributed.create_sibling_gitlab._proc_dataset
[DEBUG ] Stop advertising discontinued "hierarchy" layout for `create_siblign_gitlab()`
[DEBUG ] Building doc for <class 'datalad.distributed.create_sibling_gitlab.CreateSiblingGitlab'>
[DEBUG ] Apply patch to datalad.distributed.ora_remote.url2transport_path
[DEBUG ] Apply patch to datalad.distributed.ora_remote.url2transport_path
[DEBUG ] Apply patch to datalad.distributed.ora_remote.SSHRemoteIO
[DEBUG ] Apply patch to datalad.distributed.ora_remote.close
[DEBUG ] Apply patch to datalad.customremotes.ria_utils.create_store
[DEBUG ] Apply patch to datalad.customremotes.ria_utils.create_ds_in_store
[DEBUG ] Apply patch to datalad.customremotes.ria_utils._ensure_version
[DEBUG ] Apply patch to datalad.distributed.ora_remote.ORARemote
[DEBUG ] Building doc for <class 'datalad_next.patches.replace_create_sibling_ria.CreateSiblingRia'>
[DEBUG ] Apply patch to datalad.distributed.create_sibling_ria.CreateSiblingRia
[DEBUG ] Apply patch to datalad.distribution.dataset.create_sibling_ria
[DEBUG ] Apply patch to datalad.core.local.run.format_command
[DEBUG ] Apply patch to datalad.distribution.update._choose_update_target
[DEBUG ] Apply patch to datalad.support.sshconnector._exec_ssh
[DEBUG ] Apply patch to datalad.support.sshconnector.get
[DEBUG ] Apply patch to datalad.support.sshconnector.put
[DEBUG ] Building doc for <class 'datalad_next.commands.credentials.Credentials'>
[DEBUG ] Building doc for <class 'datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV'>
[DEBUG ] Building doc for <class 'datalad_next.commands.tree.TreeCommand'>
[DEBUG ] Building doc for <class 'datalad_next.commands.download.Download'>
[DEBUG ] Building doc for <class 'datalad_next.commands.ls_file_collection.LsFileCollection'>
[DEBUG ] Building doc for <class 'datalad_next.commands.status.Status'>
[DEBUG ] Loaded entrypoint next from datalad.extensions
[DEBUG ] Done processing entrypoints
[DEBUG ] Determined class of decorated function: <class 'datalad.distribution.siblings.Siblings'>
[DEBUG ] Command parameter validation skipped. <class 'datalad.distribution.siblings.Siblings'> declares no validator
[DEBUG ] Resolved dataset to configure sibling: /Users/user/temp/sandbox/mydataset01
[DEBUG ] Run ['git', 'annex', 'version', '--raw'] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG ] Finished ['git', 'annex', 'version', '--raw'] with status 0
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'info', '--bytes', '--fast', '--json', '--json-error-messages', '-c', 'annex.dotfiles=true'] (protocol_class=AnnexJsonProtocol) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG ] Finished ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'info', '--bytes', '--fast', '--json', '--json-error-messages', '-c', 'annex.dotfiles=true'] with status 0
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:remote.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:remote.log' failed with exitcode 128 [err: 'fatal: path 'remote.log' does not exist in 'git-annex'']
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:trust.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:trust.log' failed with exitcode 128 [err: 'fatal: path 'trust.log' does not exist in 'git-annex'']
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'wanted', '.', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'required', '.', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'group', '.', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'groupwanted', 'here', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/Users/user/temp/sandbox/mydataset01)
[DEBUG ] MultiplexSSHConnection(ctrl_path=<<PosixPath('/Us++48 chars++4e')>>, sshri=<<SSHRI(hostname++46 chars++ou')>>) is used to run ['ssh', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server', '[ -e /silver/temp/data_store/702/811de-9f93-48f3-9e59-1487a32dfbc5/config ]']
[DEBUG ] Run ['ssh', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server', '[ -e /silver/temp/data_store/702/811de-9f93-48f3-9e59-1487a32dfbc5/config ]'] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG ] Finished ['ssh', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server', '[ -e /silver/temp/data_store/702/811de-9f93-48f3-9e59-1487a32dfbc5/config ]'] with status 1
[INFO ] create siblings 'silverTemp' and 'silverTemp-storage' ...
[ERROR ] 'SSHRemoteIO' object has no attribute 'url2transport_path'
[DEBUG ] Closing 1 SSH connections...
[DEBUG ] Closing MultiplexSSHConnection(ctrl_path=<<PosixPath('/Us++48 chars++4e')>>, sshri=<<SSHRI(hostname++46 chars++ou')>>) by calling ['ssh', '-O', 'stop', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server']
[DEBUG ] Run ['ssh', '-O', 'stop', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server'] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG ] Finished ['ssh', '-O', 'stop', '-o', 'ControlPath=/Users/user/Library/Caches/datalad/sockets/64fec64e', 'user@server'] with status 0
0.00 [00:01, ?/s]%
Thanks @kevinCassou for the info. It seems like the problem is that the "Building doc"-steps are executed too early, i.e. before SSHRemoteIO
is patched. I am not sure why this is the case yet (I was unable to reproduce the error in a Conda environment on Linux-x86 or an Intel Mac). The problem might be related to an extension that is installed on your site.
I will have to go through the starting process. It would be helpful if you could provide a Traceback of the error by executing datalad
with --dbg
. I know this "remote debugging" is tedious, but it is quite helpful until I can recreate the error locally.
Another useful source of information would be the output of datalad wtf
(should tell us about installed extensions) and git config -l|grep datalad|grep -v credential|grep -v identityfile
in the dataset mydataset01
.
Hi @christian-monch
Here the result of datalad wtf
datalad wtf
# WTF
## configuration <SENSITIVE, report disabled by configuration>
## credentials
- keyring:
- active_backends:
- macOS Keyring
- PlaintextKeyring with no encyption v.1.0 at /Users/user/.local/share/python_keyring/keyring_pass.cfg
- config_file: /Users/user/.config/python_keyring/keyringrc.cfg
- data_root: /Users/user/.local/share/python_keyring
## datalad
- version: 1.1.3
## dataset
- branches:
- git-annex@0b4815d
- master@170d1e3
- id: 702811de-9f93-48f3-9e59-1487a32dfbc5
- path: /Users/user/temp/sandbox/mydataset01
- repo: AnnexRepo
## dependencies
- annexremote: 1.2.1
- boto3: 1.35.13
- cmd:7z: 16.02
- cmd:annex: 10.20240831
- cmd:bundled-git: UNKNOWN
- cmd:git: 2.46.0
- cmd:ssh: 9.0p1
- cmd:system-git: 2.46.0
- cmd:system-ssh: 9.0p1
- humanize: 4.10.0
- iso8601: 2.1.0
- keyring: 25.3.0
- keyrings.alt: 5.0.1
- msgpack: 1.0.8
- platformdirs: 4.2.2
- requests: 2.32.3
## environment
- LANG: en_US.UTF-8
- LANGUAGE: en_US
- LC_ALL: en_US.UTF-8
- LC_CTYPE: UTF-8
- PATH: /opt/homebrew/opt/imagemagick@6/bin:/opt/miniconda3/envs/py39/dltest/bin:/Users/user/mambaforge/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/opt/X11/bin:/Applications/quarto/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
- PYTHONEXE: python3
## extensions
## git-annex
- build flags:
- Assistant
- Webapp
- Pairing
- FsEvents
- TorrentParser
- MagicMime
- Servant
- Benchmark
- Feeds
- Testsuite
- S3
- WebDAV
- dependency versions:
- aws-0.24.2
- bloomfilter-2.0.1.2
- crypton-1.0.0
- DAV-1.3.4
- feed-1.3.2.1
- ghc-9.8.2
- http-client-0.7.17
- persistent-sqlite-2.13.3.0
- torrent-10000.1.3
- uuid-1.3.16
- yesod-1.6.2.1
- key/value backends:
- SHA256E
- SHA256
- SHA512E
- SHA512
- SHA224E
- SHA224
- SHA384E
- SHA384
- SHA3_256E
- SHA3_256
- SHA3_512E
- SHA3_512
- SHA3_224E
- SHA3_224
- SHA3_384E
- SHA3_384
- SKEIN256E
- SKEIN256
- SKEIN512E
- SKEIN512
- BLAKE2B256E
- BLAKE2B256
- BLAKE2B512E
- BLAKE2B512
- BLAKE2B160E
- BLAKE2B160
- BLAKE2B224E
- BLAKE2B224
- BLAKE2B384E
- BLAKE2B384
- BLAKE2BP512E
- BLAKE2BP512
- BLAKE2S256E
- BLAKE2S256
- BLAKE2S160E
- BLAKE2S160
- BLAKE2S224E
- BLAKE2S224
- BLAKE2SP256E
- BLAKE2SP256
- BLAKE2SP224E
- BLAKE2SP224
- SHA1E
- SHA1
- MD5E
- MD5
- WORM
- URL
- GITBUNDLE
- GITMANIFEST
- VURL
- X*
- local repository version: 10
- operating system: darwin aarch64
- remote types:
- git
- gcrypt
- p2p
- S3
- bup
- directory
- rsync
- web
- bittorrent
- webdav
- adb
- tahoe
- glacier
- ddar
- git-lfs
- httpalso
- borg
- rclone
- hook
- external
- supported repository versions:
- 8
- 9
- 10
- upgrade supported from repository versions:
- 0
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- version: 10.20240831
## location
- path: /Users/user/temp/sandbox/mydataset01
- type: dataset
## metadata.extractors
## metadata.filters
## metadata.indexers
## python
- implementation: CPython
- version: 3.10.14
## system
- distribution: darwin/22.6.0 13.6.9/arm64
- encoding:
- default: utf-8
- filesystem: utf-8
- locale.prefered: UTF-8
- filesystem:
- CWD:
- mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
- path: /Users/user/temp/sandbox/mydataset01
- type: apfs
- HOME:
- mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
- path: /Users/user
- type: apfs
- TMP:
- mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
- path: /var/folders/6r/wvdg2x2s6szd3tqk7kx441f80000gn/T
- type: apfs
- max_path_length: 294
- name: Darwin
- release: 22.6.0
- type: posix
- version: Darwin Kernel Version 22.6.0: Mon Jun 24 01:22:14 PDT 2024; root:xnu-8796.141.3.706.2~1/RELEASE_ARM64_T6000
and git config -l|grep datalad|grep -v credential|grep -v identityfile
is giving
remote.silverTemp.datalad-publish-depends=silverTemp-storage
remote.origin.url=git@gitlab.in2p3.fr:user/dataladtest01.git
then I did
datalad --dbg create-sibling-ria -s silverTemp --new-store-ok ria+ssh://user@server/silver/temp/data_store/
create-sibling-ria(error): /Users/user/temp/sandbox/mydataset01 (sibling) [a sibling 'silverTemp' is already configured in dataset '/Users/user/temp/sandbox/mydataset01']
create-sibling-ria(error): /Users/user/temp/sandbox/mydataset01 (sibling) [a sibling 'silverTemp-storage' is already configured in dataset '/Users/user/temp/sandbox/mydataset01']
action summary:
create-sibling-ria (error: 2)
Traceback (most recent call last):
File "/opt/miniconda3/envs/py39/dltest/bin/datalad", line 11, in <module>
sys.exit(main())
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 143, in main
_run(cmdlineargs)
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 167, in _run
ret = _run_with_debugger(namespace) \
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 181, in _run_with_debugger
return cmdlineargs.func(cmdlineargs)
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/exec.py", line 107, in call_from_parser
ret = list(ret)
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/interface/base.py", line 940, in _execute_command_
raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 2 failed:
[{'action': 'create-sibling-ria',
'message': ('a sibling %r is already configured in dataset %r',
'silverTemp',
'/Users/user/temp/sandbox/mydataset01'),
'name': 'silverTemp',
'path': '/Users/user/temp/sandbox/mydataset01',
'status': 'error',
'type': 'sibling'},
{'action': 'create-sibling-ria',
'message': ('a sibling %r is already configured in dataset %r',
'silverTemp-storage',
'/Users/user/temp/sandbox/mydataset01'),
'name': 'silverTemp-storage',
'path': '/Users/user/temp/sandbox/mydataset01',
'status': 'error',
'type': 'sibling'}]
> /opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/interface/base.py(940)_execute_command_()
-> raise IncompleteResultsError(
(Pdb)
terminal scratch. I guess it's because siblings is already set ... ?
[...] terminal scratch. I guess it's because siblings is already set ... ?
Yes, it would be good to call datalad create-sibling-ria
with a different name, e.g. -s silverTemp2
.
[...]
datalad wtf ... ## extensions ## git-annex ...
...
Thanks for the info. A quick first question: I am wondering why datalad-next
does not show up in "extensions"? Does a pip install --upgrade datalad-next
make a difference there?
ok, I did pip install --upgrade datalad-next
, and in fact, now we have the following:
datalad wtf
...
## extensions
- next:
- description: What is next in DataLad
- entrypoints:
- datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV:
- class: CreateSiblingWebDAV
- module: datalad_next.commands.create_sibling_webdav
- names:
- create-sibling-webdav
- datalad_next.commands.credentials.Credentials:
- class: Credentials
- module: datalad_next.commands.credentials
- names:
- datalad_next.commands.download.Download:
- class: Download
- module: datalad_next.commands.download
- names:
- download
- datalad_next.commands.ls_file_collection.LsFileCollection:
- class: LsFileCollection
- module: datalad_next.commands.ls_file_collection
- names:
- ls-file-collection
- datalad_next.commands.status.Status:
- class: Status
- module: datalad_next.commands.status
- names:
- next-status
- next_status
- datalad_next.commands.tree.TreeCommand:
- class: TreeCommand
- module: datalad_next.commands.tree
- names:
- tree
- module: datalad_next
- version: 1.5.0
## git-annex
...
but then
datalad --dbg create-sibling-ria -s silverTemp4 --new-store-ok ria+ssh://user@ssh-ubuntu3.ijclab.in2p3.fr/silver/temp/data_kev/data_store04/
[INFO ] create siblings 'silverTemp4' and 'silverTemp4-storage' ...
0.00 [00:01, ?/s]Traceback (most recent call last):
File "/opt/miniconda3/envs/py39/dltest/bin/datalad", line 11, in <module>
sys.exit(main())
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 143, in main
_run(cmdlineargs)
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 167, in _run
ret = _run_with_debugger(namespace) \
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/main.py", line 181, in _run_with_debugger
return cmdlineargs.func(cmdlineargs)
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/cli/exec.py", line 107, in call_from_parser
ret = list(ret)
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/interface/base.py", line 873, in _execute_command_
for r in _process_results(
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/interface/utils.py", line 319, in _process_results
for res in results:
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/distributed/create_sibling_ria.py", line 430, in __call__
yield from _create_sibling_ria(
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/distributed/create_sibling_ria.py", line 601, in _create_sibling_ria
create_ds_in_store(SSHRemoteIO(ssh_host) if ssh_host else LocalIO(),
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad/customremotes/ria_utils.py", line 227, in create_ds_in_store
_ensure_version(io, dsgit_dir, obj_version)
File "/opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad_next/patches/ria_utils.py", line 52, in ria_utils__ensure_version
version_file = io.url2transport_path(base_path / 'ria-layout-version')
AttributeError: 'SSHRemoteIO' object has no attribute 'url2transport_path'
> /opt/miniconda3/envs/py39/dltest/lib/python3.10/site-packages/datalad_next/patches/ria_utils.py(52)ria_utils__ensure_version()
-> version_file = io.url2transport_path(base_path / 'ria-layout-version')
(Pdb)
terminal scratch again. On the remote storage, the directory error-logs
and the file ria-layout-version
are created...
Hoping it helps
Thanks for all the info @kevinCassou. I found it, and I should have seen the problem earlier.
Please set the git config variable datalad.extensions.load
to next
, for example with the command:
git config --global --add datalad.extensions.load next
That should fix it.
Yes I confirm ! I'm still a newbie in datalad
and testing ! 🔰
Thanks for help 🙏🏼
Yes I confirm ! I'm still a newbie in
datalad
and testing ! 🔰Thanks for help 🙏🏼
you are very welcome.
Description
Dear dataladers,
We are trying to set an RIA to test a new sibling on a remote SSH server any idea about the following error ? We cannot install
datalad
or anything that require admin grant on the remote server :Many thanks for help.