`ls` doesn't work on files

yarikoptic commented 2 years ago

I have done datalad fusefs -d ~/datalad /mnt/datalad where ~/datalad is installation of http://datasets.datalad.org/.git

with some subdatasets installed

```shell $> datalad subdatasets -d ~/datalad --fulfilled yes -r subdataset(ok): abide (dataset) subdataset(ok): abide/Derivatives (dataset) subdataset(ok): abide/Derivatives/ccs_nofilt_global (dataset) subdataset(ok): allen-brain-observatory (dataset) subdataset(ok): allen-brain-observatory/visual-coding-neuropixels (dataset) subdataset(ok): allen-brain-observatory/visual-coding-neuropixels/raw-data (dataset) subdataset(ok): centerforopenneuroscience (dataset) subdataset(ok): centerforopenneuroscience/nuisance (dataset) subdataset(ok): centerforopenneuroscience/nuisance/data/dbic (dataset) subdataset(ok): centerforopenneuroscience/nuisance/presentations/2020-NNL (dataset) subdataset(ok): centerforopenneuroscience/nuisance/presentations/2020-NNL/reveal.js (dataset) subdataset(ok): conp-dataset (dataset) subdataset(ok): crcns (dataset) subdataset(ok): crcns/aa-1 (dataset) subdataset(ok): crcns/lgn-1 (dataset) subdataset(ok): dandi (dataset) subdataset(ok): dandi/dandisets (dataset) subdataset(ok): dandi/dandisets/000003 (dataset) subdataset(ok): dandi/dandisets/000004 (dataset) subdataset(ok): dandi/dandisets/000005 (dataset) subdataset(ok): dandi/dandisets/000006 (dataset) subdataset(ok): dandi/dandisets/000007 (dataset) subdataset(ok): dandi/dandisets/000008 (dataset) subdataset(ok): dandi/dandisets/000009 (dataset) subdataset(ok): dandi/dandisets/000010 (dataset) subdataset(ok): dandi/dandisets/000011 (dataset) subdataset(ok): dandi/dandisets/000012 (dataset) subdataset(ok): dandi/dandisets/000013 (dataset) subdataset(ok): dandi/dandisets/000015 (dataset) subdataset(ok): dandi/dandisets/000016 (dataset) subdataset(ok): dandi/dandisets/000017 (dataset) subdataset(ok): dandi/dandisets/000018 (dataset) subdataset(ok): dandi/dandisets/000019 (dataset) subdataset(ok): dandi/dandisets/000026 (dataset) subdataset(ok): dbic (dataset) subdataset(ok): dbic/QA (dataset) subdataset(ok): hcp-openaccess (dataset) subdataset(ok): hcp-openaccess/HCP1200/100206 (dataset) subdataset(ok): hcp-openaccess/HCP1200/100206/MNINonLinear (dataset) subdataset(ok): hcp-openaccess/HCP1200/100206/T1w (dataset) subdataset(ok): hcp-openaccess/HCP1200/100206/unprocessed (dataset) subdataset(ok): hcp-openaccess/HCP1200/198047 (dataset) subdataset(ok): hcp-openaccess/HCP1200/198047/MNINonLinear (dataset) subdataset(ok): hcp-openaccess/HCP1200/569965 (dataset) subdataset(ok): hcp-openaccess/HCP1200/569965/T1w (dataset) subdataset(ok): labs (dataset) subdataset(ok): labs/churchland (dataset) subdataset(ok): labs/churchland/najafi-2018-nwb (dataset) subdataset(ok): labs/hasson (dataset) subdataset(ok): labs/hasson/narratives (dataset) subdataset(ok): labs/hasson/narratives/derivatives/fmriprep (dataset) subdataset(ok): labs/haxby (dataset) subdataset(ok): labs/haxby/attention (dataset) subdataset(ok): labs/haxby/attention/derivatives/afni (dataset) subdataset(ok): labs/haxby/attention/derivatives/fmriprep (dataset) subdataset(ok): labs/haxby/attention/derivatives/freesurfer/fsaverage6 (dataset) subdataset(ok): labs/haxby/attention/derivatives/pymvpa (dataset) subdataset(ok): openfmri (dataset) subdataset(ok): openfmri/ds000001 (dataset) subdataset(ok): openfmri/ds000105 (dataset) subdataset(ok): openneuro (dataset) subdataset(ok): openneuro/ds000001 (dataset) subdataset(ok): openneuro/ds000002 (dataset) subdataset(ok): openneuro/ds000003 (dataset) subdataset(ok): openneuro/ds000005 (dataset) subdataset(ok): openneuro/ds000006 (dataset) subdataset(ok): openneuro/ds000007 (dataset) subdataset(ok): openneuro/ds000008 (dataset) subdataset(ok): openneuro/ds000009 (dataset) subdataset(ok): openneuro/ds000105 (dataset) subdataset(ok): openneuro/ds000138 (dataset) subdataset(ok): openneuro/ds001785 (dataset) subdataset(ok): openneuro/ds002721 (dataset) subdataset(ok): openneuro/ds003027 (dataset) ```

when I went to the subdataset's folder I get errors from trying to list those files:

lena:/mnt/datalad/openneuro
$> cd ds000001 
CHANGES                   sub-01/  sub-05/  sub-09/  sub-13/  task-balloonanalogrisktask_bold.json
README                    sub-02/  sub-06/  sub-10/  sub-14/
dataset_description.json  sub-03/  sub-07/  sub-11/  sub-15/
participants.tsv          sub-04/  sub-08/  sub-12/  sub-16/
1 27309.....................................:Thu 02 Dec 2021 11:13:30 AM EST:.
lena:/mnt/datalad/openneuro/ds000001
$> cd sub-01 
anat/  func/
1 27310.....................................:Thu 02 Dec 2021 11:13:33 AM EST:.
lena:/mnt/datalad/openneuro/ds000001/sub-01
$> cd anat 
ls: cannot access 'sub-01_T1w.nii.gz': Invalid argument
ls: cannot access 'sub-01_inplaneT2.nii.gz': Invalid argument
sub-01_T1w.nii.gz  sub-01_inplaneT2.nii.gz
1 27311.....................................:Thu 02 Dec 2021 11:13:35 AM EST:.
lena:/mnt/datalad/openneuro/ds000001/sub-01/anat
$> ls -l 
ls: cannot access 'sub-01_T1w.nii.gz': Invalid argument
ls: cannot access 'sub-01_inplaneT2.nii.gz': Invalid argument
total 0
?????????? ? ? ? ?            ? sub-01_T1w.nii.gz
?????????? ? ? ? ?            ? sub-01_inplaneT2.nii.gz

those files are indeed not present locally:

$> ls -Ll ~/datalad/openneuro/ds000001/sub-01/anat
ls: cannot access '/home/yoh/datalad/openneuro/ds000001/sub-01/anat/sub-01_T1w.nii.gz': No such file or directory
ls: cannot access '/home/yoh/datalad/openneuro/ds000001/sub-01/anat/sub-01_inplaneT2.nii.gz': No such file or directory
total 0
l????????? ? ? ? ?            ? sub-01_T1w.nii.gz
l????????? ? ? ? ?            ? sub-01_inplaneT2.nii.gz

but fuse should not "expose" that. Instead if should (and I believe did some time ago) get at least size from annex (if known from the key), date for now could be just the date of the last commit for that repository/dataset.

jwodder commented 2 years ago

@yarikoptic Could not reproduce. I cloned the dataset as follows:

datalad clone ///
cd datasets.datalad.org/
datalad install openneuro/
datalad install openneuro/ds000001

then FUSE-mounted it with datalad -l debug fusefs -d datasets.datalad.org mount -f, and then I did:

$ cd mount/openneuro/ds000001
$ ls -l sub-01/anat/
total 0
-rw-r--r-- 1 vagrant vagrant 5663237 Dec  2 11:30 sub-01_T1w.nii.gz
-rw-r--r-- 1 vagrant vagrant  669578 Dec  2 11:30 sub-01_inplaneT2.nii.gz

Did you perhaps leave out the -f option to the fusefs command? As stated here, fsspec's HTTPFileSystem doesn't work properly when backgrounded.

yarikoptic commented 2 years ago

doh... right -- without -f. Let's error out for now if ran without -f since we know that it doesn't work (yet).

datalad / datalad-fuse

`ls` doesn't work on files #20