Can't delete stuff on discovery

TheChymera commented 1 year ago

So @asmacdo and I have been running into some issues trying to delete intermediary outputs on discovery. Neither the chmod nor the datalad remove solutions seen on on the PR for datalad+shell issues worked:

With chmod:

(asmacdo-datalad2) [f006rq8@ndoli opfvta-replication-2023]$ chmod 775 -R outputs/20230803090623/
(asmacdo-datalad2) [f006rq8@ndoli opfvta-replication-2023]$ rm -rf outputs/20230803090623/
rm: cannot remove 'outputs/20230803090623/data/l2Manual/alias-block_filtered/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Manual/alias-block_other_controlled/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Manual/alias-block_other_controlled/zacq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Manual/alias-block_other_controlled/acq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Manual/alias-block_other/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Manual/alias-block_filtered_controlled/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Manual/alias-block_filtered_controlled/zacq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Manual/alias-block_filtered_controlled/acq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/vta_projection_cope.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/vta_projection_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/features_l2/z_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/features_l2/_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/features_l2/_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_filtered/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_control/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/task-CogBm/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/task-CogB/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/task-CogMwf/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/task-CogP/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-phasic_filtered/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_other_controlled/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_other_controlled/zacq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_other_controlled/acq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_other/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_filtered_controlled/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_filtered_controlled/zacq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-block_filtered_controlled/acq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/task-CogBl/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/task-CogBr/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-phasic_other/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/alias-phasic/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2/task-JPogP/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Onmibus/alias-phasic_filtered/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Onmibus/alias-phasic_other/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/vta_right.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Omnibus/alias-block_filtered/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Omnibus/alias-block_other_controlled/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Omnibus/alias-block_other_controlled/zacq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Omnibus/alias-block_other_controlled/acq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Omnibus/alias-block_other/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Omnibus/alias-block_filtered_controlled/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Omnibus/alias-block_filtered_controlled/zacq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/l2Omnibus/alias-block_filtered_controlled/acq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/seed_l2/alias-block_filtered/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/seed_l2/alias-block_control/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/seed_l2/alias-phasic_filtered/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/seed_l2/alias-block_other/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/seed_l2/alias-block_filtered_controlled/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/seed_l2/alias-block_filtered_controlled/zacq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/seed_l2/alias-block_filtered_controlled/acq-EPI_fstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/seed_l2/alias-phasic_other/acq-EPI_tstat.nii.gz': Permission denied
rm: cannot remove 'outputs/20230803090623/data/vta_projection_zstat.nii.gz': Permission denied
(asmacdo-datalad2) [f006rq8@ndoli opfvta-replication-2023]$ ls -lah outputs/20230803090623/
total 96K
drwxrwx--- 3 f006rq8 rc-DBIC  22 Aug  6 20:22 .
drwxrwx--- 3 f006rq8 rc-DBIC  62 Aug  3 09:06 ..
drwxrwx--- 8 f006rq8 rc-DBIC 322 Aug  6 20:22 data

With datalad --remove:

(asmacdo-datalad2) [f006rq8@ndoli opfvta-replication-2023]$ datalad remove --reckless modification outputs/20230803090623/
[ERROR  ] [Errno 13] Permission denied: 'acq-EPI_tstat.nii.gz'
(asmacdo-datalad2) [f006rq8@ndoli opfvta-replication-2023]$ datalad status
untracked: outputs/20230803090623/data/features_l2 (directory)
untracked: outputs/20230803090623/data/l2 (directory)
untracked: outputs/20230803090623/data/l2Manual (directory)
untracked: outputs/20230803090623/data/l2Omnibus (directory)
untracked: outputs/20230803090623/data/l2Onmibus (directory)
untracked: outputs/20230803090623/data/seed_l2 (directory)
untracked: outputs/20230803090623/data/vta_projection_cope.nii.gz (file)
untracked: outputs/20230803090623/data/vta_projection_tstat.nii.gz (file)
untracked: outputs/20230803090623/data/vta_projection_zstat.nii.gz (file)
untracked: outputs/20230803090623/data/vta_right.nii.gz (file)
    added: .reproman/jobs/discovery/20230803-084102-d9af/command-array (file)
    added: .reproman/jobs/discovery/20230803-084102-d9af/idmap (file)
    added: .reproman/jobs/discovery/20230803-084102-d9af/runscript (file)
    added: .reproman/jobs/discovery/20230803-084102-d9af/spec.yaml (file)
    added: .reproman/jobs/discovery/20230803-084102-d9af/status.0 (file)
    added: .reproman/jobs/discovery/20230803-084102-d9af/stderr.0 (file)
    added: .reproman/jobs/discovery/20230803-084102-d9af/stdout.0 (file)
    added: .reproman/jobs/discovery/20230803-084102-d9af/submit (file)
 modified: code/opfvta (dataset)
    added: outputs/20230803090623/data/CogB.tsv (file)
    added: outputs/20230803090623/data/CogBl.tsv (file)
    added: outputs/20230803090623/data/CogBm.tsv (file)
    added: outputs/20230803090623/data/CogBr.tsv (file)
    added: outputs/20230803090623/data/CogMwf.tsv (file)
    added: outputs/20230803090623/data/CogP.tsv (file)
    added: outputs/20230803090623/data/JPogP.tsv (file)
    added: outputs/20230803090623/data/JPogT.tsv (file)
    added: outputs/20230803090623/data/chr_beta1.txt (file)
    added: outputs/20230803090623/data/correlation_data.json (file)
    added: outputs/20230803090623/data/features_structural.csv (file)
    added: outputs/20230803090623/data/functional_significance.csv (file)
    added: outputs/20230803090623/data/functional_t.csv (file)
    added: outputs/20230803090623/data/groups.csv (file)
    added: outputs/20230803090623/data/implant_coordinates.csv (file)
    added: outputs/20230803090623/data/implant_coordinates_block.csv (file)
    added: outputs/20230803090623/data/implant_coordinates_phasic.csv (file)
    added: outputs/20230803090623/data/network_model.dot (file)

@yarikoptic any suggestions? We can retire the whole repo and clone a fresh one, but it will take a long time to get the data and images, discovery is apparently pretty slow with download :/ ~1MB/s

yarikoptic commented 1 year ago

well -- it is all under ACL, so you might need to use nfs4_setfacl etc for the files which were made "read only" that way. chmod would not be sufficient etc. But I do not think that was even the case here.

indeed best to use datalad commands for committed stuff
for not yet tracked files, ask why they are readonly - who made them such? or may be you are "the wrong" user ?

FWIW -- I do not see those files any longer:

[d31548v@discovery7 opfvta-replication-2023]$ pwd
/dartfs/rc/lab/D/DBIC/DBIC/CON/asmacdo/opfvta-replication-2023
[d31548v@discovery7 opfvta-replication-2023]$ ls -l outputs/20230803090623/data/features_l2
ls: cannot access outputs/20230803090623/data/features_l2: No such file or directory

re re-transfer -- indeed not to bother if you already have stuff there. But also you could cheat -- clone a new one, and them mv old .git/annex into that new one location I guess.

asmacdo commented 1 year ago

@yarikoptic This has happened to me twice /facepalm.

I have renamed the directories these former repos to broke-old and broke-old-2 ;)

No idea how to get rid of them. I think the workflow that put me in this state was by running datalad remove first, which didn't work because of untracked files. So I figured "oh they aren't tracked so I just delete them". But for this I get permission denied. I tried --reckless modification mode, but maybe only after I deleted the .git dir with an ill-advised rm -rf .

I'm hoping that next time, using reckless mode will get us out of this, but in the meantime I have no idea how to clean up those files.

yarikoptic commented 1 year ago

Let's troubleshoot that in an interactive session together morning at 10am?

asmacdo commented 1 year ago

(asmacdo-datalad2) [f006rq8@ndoli outputs]$ cd 20230803090623/data/
(asmacdo-datalad2) [f006rq8@ndoli data]$ ls
features_l2  l2Onmibus                    vta_projection_zstat.nii.gz
l2Manual     seed_l2                      vta_right.nii.gz
l2Omnibus    vta_projection_tstat.nii.gz
(asmacdo-datalad2) [f006rq8@ndoli data]$ rm vta_projection_zstat.nii.gz 
rm: remove write-protected regular file 'vta_projection_zstat.nii.gz'? y
rm: cannot remove 'vta_projection_zstat.nii.gz': Permission denied
(asmacdo-datalad2) [f006rq8@ndoli data]$ ls -l vta_projection_zstat.nii.gz 
-r-xr-x--- 2 f006rq8 rc-DBIC 240016 Aug  3 08:56 vta_projection_zstat.nii.gz
(asmacdo-datalad2) [f006rq8@ndoli data]$ nfs4_getfacl vta_projection_zstat.nii.gz 

# file: vta_projection_zstat.nii.gz
A::OWNER@:rxtncy
A:g:rc-DBIC-admin@KIEWIT.DARTMOUTH.EDU:rxtncCoy
A::d11124v@KIEWIT.DARTMOUTH.EDU:rxtncCoy
A::f002d6b@KIEWIT.DARTMOUTH.EDU:rxtncCoy
A::f00275v@KIEWIT.DARTMOUTH.EDU:rxtncCoy
A::d31548v@KIEWIT.DARTMOUTH.EDU:rxtncCoy
A::GROUP@:rxtncy
A::d26427b@KIEWIT.DARTMOUTH.EDU:rxtncCoy
A:g:rc-DartFSadmin@KIEWIT.DARTMOUTH.EDU:rxtncCoy
(asmacdo-datalad2) [f006rq8@ndoli data]$

yarikoptic commented 1 year ago

do we still have a sample of such data ? but as you see from ACL - write etc are not permitted. smth made it read-only. Could have been that freezing script we have for git-annex...

asmacdo commented 1 year ago

no, we deleted it all today I think

con / opfvta-reexecution

Can't delete stuff on discovery #33