search

datalad / datalad-deprecated

DataLad extension for functionality that has been phased out of the core package
Other
0 stars 3 forks source link

apparently publish doesn't provide --jobs to `annex copy` #40

Open yarikoptic opened 4 years ago

yarikoptic commented 4 years ago

For better (faster) or for worse (discovers bugs in datalad/git-annex, might trigger site connection limits) by default we parallelize get. But apparently we do not do that for publish (code: https://github.com/datalad/datalad/blob/master/datalad/support/annexrepo.py#L2929). I think we should be consistent and provide similar --jobs for annex copy as well

mih commented 4 years ago

Ping datalad/datalad#3415

yarikoptic commented 4 years ago

this issue is a twin-brother of datalad/datalad#4704 (push) which presumably (according to Closure above by @mih) was addressed in push. Since publish is still around, reopening this issue.

-J is simply seems not to be passed into git-annex copy invocation:

$ datalad publish --to=datalad-public -r -J4

results in 

yoh      2174564 45.6  0.0 1074080556 51508 pts/6 Sl+ 12:29   0:05  |               \_ /usr/lib/git-annex.linux/exe/git-annex --library-path /usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit:/usr/lib/git-annex.linux//etc/ld.so.conf.d:/usr/lib/git-annex.linux//lib64:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu:/usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git-annex/git-annex copy -c annex.dotfiles=true -c remote.datalad-public.annex-ssh-options=-o ControlMaster=auto -S /home/yoh/.cache/datalad/sockets/b0da704a -c annex.retry=3 --json --json-error-messages --json-progress --to=datalad-public --auto --fast

so there is no -J option provided. datalad 0.13.0.dev44 .

mih commented 2 years ago

Moving this to -deprecated