search

psychoinformatics-de / knowledge-base

Sources for the psyinf knowledge base
https://knowledge-base.psychoinformatics.de
Other
0 stars 3 forks source link

Weird inability to obtain file content for specific user #119

Closed adswa closed 5 months ago

adswa commented 7 months ago

Origin: Private message on matrix

Bianca (Sophie's group) wants to work with an fmriprep'ed ABCD dataset that resides as a dataset hierarchy on JUDAC. She has access to JUDAC and the RIA store in which the datasets are stored.

After successfully cloning the datasets, file content retrieval (with datalad get or git annex get stalls without an error message (she reportedly let it run over night for 4GB of data without any progress).

She has tried this with the same outcome on Leipzig computing infrastructure and on juseless. Adina has tried this on her laptop and succeeded without any issue. In a video call, everything in the dataset looked kosher (info below).

Relevant info and commands:

# clone command
datalad clone "ria+ssh://<judac-username>@judac.fz-juelich.de:/p/largedata/inm7/ria-ABCD/fmriprep_outputstore#~ABCD_fMRIprep"
# subdataset and content retrieval (cloning works, retrievals shows stalled progress bar at 0%)
cd fmriprep/sub-NDARINVZZFG6J5U && datalad get .

git annex -v -d get <file> shows chatter with the special remote that stalls midway.

Software versions seem up to date: datalad: 0.19.3 git-annex: 20221003

Screenshot from 2024-02-01 13-23-02

git annex info, and git-annex whereis show a correct, ssh-ria URL to JUDAC for the file (i.e., correct user name, correct protocol). 

whereis ../../sub-NDARINVZZFG6J5U.html (2 copies) 
    1671fa33-7b93-4bac-99bc-c8c87a7b4fe9 -- hoffstaedter1@jsfc142:/local/nvme/NDARINVZZFG6J5U.529024/ds/fmriprep/sub-NDARINVZZFG6J5U
    5f641b9b-e954-4bb5-857f-f05386518f24 -- [fmriprep_outputstore-storage]

  fmriprep_outputstore-storage: ssh://serio1@judac.fz-juelich.de: /p/largedata/inm7/ria-ABCD/fmriprep_outputstore/57b/b9b0a-2df1-49d8-9e74-2903483f9d90:2k/2Q/MD5E-s124560--6e7335a1fce9e571460e4bafa9bb6422.html/MD5E-s124560--6e7335a1fce9e571460e4bafa9bb6422.html
ok
$ git annex info 
trusted repositories: 0
semitrusted repositories: 5
    00000000-0000-0000-0000-000000000001 -- web
    00000000-0000-0000-0000-000000000002 -- bittorrent
    1671fa33-7b93-4bac-99bc-c8c87a7b4fe9 -- hoffstaedter1@jsfc142:/local/nvme/NDARINVZZFG6J5U.529024/ds/fmriprep/sub-NDARINVZZFG6J5U
    4b6f0af8-8164-4efb-9cbb-6644e5d29eb1 -- serio@comps08h06:/data/p_02667/development/ABCD/ABCD_fMRIprep/fmriprep/sub-NDARINVZZFG6J5U [here]
    5f641b9b-e954-4bb5-857f-f05386518f24 -- [fmriprep_outputstore-storage]
untrusted repositories: 0
transfers in progress: 
    downloading figures/sub-NDARINVZZFG6J5U_ses-baselineYear1Arm1_desc-about_T1w.html from fmriprep_outputstore-storage
available local disk space: 2.09 terabytes (+100 megabytes reserved)
local annex keys: 0
local annex size: 0 bytes
annexed files in working tree: 160
size of annexed files in working tree: 3.89 gigabytes
bloom filter size: 32 mebibytes (0% full)
backend usage: 
    MD5E: 160

Here is the content of remote.log:

5f641b9b-e954-4bb5-857f-f05386518f24 archive-id=57bb9b0a-2df1-49d8-9e74-2903483f9d90 autoenable=true encryption=none externaltype=ora name=fmriprep_outputstore-storage type=external url=ria+file:///p/fastdata/inm7/ria-ABCD/fmriprep_outputstore timestamp=1706704760.960927721s

TODO (not necessarily to be performed in this order)

adswa commented 7 months ago

special remote chatter:

[2024-02-01 14:12:13.971018402] (Utility.Process) process [50406] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--","sub-NDARINVZZFG6J5U.html"]
[2024-02-01 14:12:13.973673953] (Utility.Process) process [50407] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2024-02-01 14:12:13.974272468] (Utility.Process) process [50408] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2024-02-01 14:12:13.976526955] (Utility.Process) process [50409] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2024-02-01 14:12:14.026688918] (Utility.Process) process [50409] done ExitSuccess
[2024-02-01 14:12:14.027167261] (Utility.Process) process [50410] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2024-02-01 14:12:14.035242807] (Utility.Process) process [50410] done ExitSuccess
[2024-02-01 14:12:14.037299491] (Utility.Process) process [50411] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..2e9463fbdb91f6155a0eb05d868d73c0b5a6df6c","--pretty=%H","-n1"]
[2024-02-01 14:12:14.068443617] (Utility.Process) process [50411] done ExitSuccess
[2024-02-01 14:12:14.068866161] (Utility.Process) process [50412] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..4c4edc31714db7505d8d83143934c93765af7c76","--pretty=%H","-n1"]
[2024-02-01 14:12:14.078769003] (Utility.Process) process [50412] done ExitSuccess
[2024-02-01 14:12:14.104286149] (Utility.Process) process [50413] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
get sub-NDARINVZZFG6J5U.html [2024-02-01 14:12:14.124614834] (Utility.Process) process [50415] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
(from fmriprep_outputstore-storage...) 
[2024-02-01 14:12:14.349814231] (Utility.Process) process [50416] chat: /bin/git-annex-remote-ora []
[2024-02-01 14:12:14.705844119] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> VERSION 1
[2024-02-01 14:12:14.706320449] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- EXTENSIONS INFO GETGITREMOTENAME ASYNC
[2024-02-01 14:12:14.706630432] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> EXTENSIONS
[2024-02-01 14:12:14.706765628] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- PREPARE
[2024-02-01 14:12:14.706956808] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETGITDIR
[2024-02-01 14:12:14.707080755] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE .git
[2024-02-01 14:12:14.77050512] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETGITDIR
[2024-02-01 14:12:14.770623263] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE .git
[2024-02-01 14:12:14.770878796] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETCONFIG archive-id
[2024-02-01 14:12:14.771005542] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE 57bb9b0a-2df1-49d8-9e74-2903483f9d90
[2024-02-01 14:12:14.771212761] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETUUID
[2024-02-01 14:12:14.77154328] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE 5f641b9b-e954-4bb5-857f-f05386518f24
[2024-02-01 14:12:14.771767782] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETCONFIG url
[2024-02-01 14:12:14.771893404] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE ria+ssh://serio1@judac.fz-juelich.de:/p/largedata/inm7/ria-ABCD/fmriprep_outputstore
[2024-02-01 14:12:14.77211646] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETCONFIG push-url
[2024-02-01 14:12:14.772194361] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE 
[2024-02-01 14:12:14.772305838] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETCONFIG force-write
[2024-02-01 14:12:14.772381233] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE 
[2024-02-01 14:12:14.772475518] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETCONFIG name
[2024-02-01 14:12:14.772529986] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE fmriprep_outputstore-storage
[2024-02-01 14:12:14.778568222] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> GETUUID
[2024-02-01 14:12:14.778702709] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE 5f641b9b-e954-4bb5-857f-f05386518f24
[2024-02-01 14:12:15.190428647] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> PREPARE-SUCCESS
[2024-02-01 14:12:15.190786453] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- TRANSFER RETRIEVE MD5E-s124560--6e7335a1fce9e571460e4bafa9bb6422.html .git/annex/tmp/MD5E-s124560--6e7335a1fce9e571460e4bafa9bb6422.html
[2024-02-01 14:12:15.191469423] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] --> DIRHASH MD5E-s124560--6e7335a1fce9e571460e4bafa9bb6422.html
[2024-02-01 14:12:15.191630178] (Annex.ExternalAddonProcess) /bin/git-annex-remote-ora[1] <-- VALUE 2k/2Q/
adswa commented 7 months ago

JUDAC permissions in the ria store and her account:

[serio1@judac04 ~]$ groups
jusers inm7 jusuf_cpu icei-hbp-00000000004
[wagner11@judac04 ria-ABCD]$ ls -lah fmriprep_outputstore/ffd/ef768-1e0a-4f08-8ad1-6e93c34de1fd/
total 11K
drwxr-sr-x 6 hoffstaedter1 inm7 4.0K Dec 12 18:18 .
drwxr-sr-x 3 hoffstaedter1 inm7 4.0K Oct 17 04:12 ..
drwxr-sr-x 2 hoffstaedter1 inm7 4.0K Oct 19 02:29 archives
-rw-r--r-- 1 hoffstaedter1 inm7  134 Oct 17 04:12 config
-rw-r--r-- 1 hoffstaedter1 inm7   73 Oct 17 04:12 description
-rw-r--r-- 1 hoffstaedter1 inm7   23 Oct 17 04:12 HEAD
drwxr-sr-x 2 hoffstaedter1 inm7 4.0K Dec 12 18:18 info
drwxr-sr-x 4 hoffstaedter1 inm7 4.0K Dec 12 16:51 objects
-rw-r--r-- 1 hoffstaedter1 inm7  167 Dec 12 18:18 packed-refs
drwxr-sr-x 4 hoffstaedter1 inm7 4.0K Oct 17 04:12 refs
-rw-r--r-- 1 hoffstaedter1 inm7    2 Oct 17 04:12 ria-layout-version
adswa commented 7 months ago

I reached out to Felix to ask about a similar issue I recall he faced.

adswa commented 7 months ago

I'm waiting for confirmation, but it might be that her JUDAC account does not have 7z installed/on the PATH.

adswa commented 7 months ago

Installing 7z solved it indeed. This should probably make it into a KBI