DAV failure / errors with indiviudal files when pushing to ownCloud

lnnrtwttkhn commented 1 year ago

Hey all, I tried create_sibling_webdav for the first time together with GWDG's ownCloud.

I used the MoAEpilot dataset as an example, as described in the "Walk-through: Amazon S3 as a special remote" chapter in the DataLad Handbook.

When pushing the dataset to ownCloud for the first time, errors occurred related to .datalad/config and sub-01/func/sub-01_task-auditory_events.tsv (see below).

The file were transferred on a second push though.

So not a huge issue but just reporting this in case you see any opportunity to improve something on the DataLad side of things.

Thanks and keep up the good work!

Documentation

Configuring the WebDAV sibling

datalad create-sibling-webdav -d . --name owncloud-gwdg --mode filetree 'https://owncloud.gwdg.de/remote.php/nonshib-webdav/neuro-data-s3'
User name and password are required for WebDAV access at https://owncloud.gwdg.de/remote.php/nonshib-webdav/neuro-data-s3
user: wittkuhn@mpib-berlin.mpg.de

password: 
password (repeat):

Output:

create_sibling_webdav.storage(ok): . [owncloud-gwdg-storage: https://owncloud.gwdg.de/remote.php/nonshib-webdav/neuro-data-s3]
[INFO   ] Configure additional publication dependency on "owncloud-gwdg-storage" 
create_sibling_webdav(ok): . [owncloud-gwdg: datalad-annex::?type=webdav&encryption=none&exporttree=yes&url=https%3A//owncloud.gwdg.de/remote.php/nonshib-webdav/neuro-data-s3]
[WARNING] The entered credential will not be stored, a credential with the default name 'webdav-wittkuhn@mpib-berlin.mpg.de-https://owncloud.gwdg.de/ownCloud' already exists. Specify a credential name via the `credential` parameter  and/or configure a credential with the datalad-credentials command with a `realm=https://owncloud.gwdg.de/ownCloud` property

Pushing to WebDAV sibling:

datalad push --to owncloud-gwdg
Push to 'owncloud-gwdg':  25%|█████████████████████████████                                                                                       | 1.00/4.00 [00:00<00:00, 7.10k Steps/s]A credential is required for access
user: wittkuhn@mpib-berlin.mpg.de
password: 
password (repeat):

Output

copy(ok): .datalad/.gitattributes (dataset)                                                                                                                                               
copy(error): .datalad/config (dataset) [DAV failure: Status {statusCode = 403, statusMessage = "Forbidden"} "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n  <s:exception>Sabre\\DAV\\Exception\\Forbidden</s:exception>\n  <s:message/>\n</d:error>\n" HTTP request: "PUT" "/remote.php/nonshib-webdav/neuro-data-s3/.datalad/git-annex-webdav-tmp-GIT--456eb5f76b92337c8c401dd7aba2401c2bca439a"]                                                                                                              
copy(ok): .gitattributes (dataset)                                                                                                                                                        
copy(ok): CHANGES (dataset)                                                                                                                                                               
copy(ok): README (dataset)                                                                                                                                                                
copy(ok): dataset_description.json (dataset)                                                                                                                                              
copy(ok): sub-01/anat/sub-01_T1w.nii (dataset)                                                                                                                                            
copy(ok): sub-01/func/sub-01_task-auditory_bold.nii (dataset)                                                                                                                             
copy(error): sub-01/func/sub-01_task-auditory_events.tsv (dataset) [DAV failure: Status {statusCode = 403, statusMessage = "Forbidden"} "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n  <s:exception>Sabre\\DAV\\Exception\\Forbidden</s:exception>\n  <s:message/>\n</d:error>\n" HTTP request: "PUT" "/remote.php/nonshib-webdav/neuro-data-s3/sub-01/func/git-annex-webdav-tmp-MD5E-s144--23da36d995d1905cf4066e05853d8db2.tsv"                                                                              
DAV failure: Status {statusCode = 403, statusMessage = "Forbidden"} "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n  <s:exception>Sabre\\DAV\\Exception\\Forbidden</s:exception>\n  <s:message/>\n</d:error>\n" HTTP request: "PUT" "/remote.php/nonshib-webdav/neuro-data-s3/sub-01/func/git-annex-webdav-tmp-MD5E-s144--23da36d995d1905cf4066e05853d8db2.tsv"]
copy(ok): task-auditory_bold.json (dataset)                                                                                                                                               
Update availability for 'owncloud-gwdg':  75%|███████████████████████████████████████████████████████████████████████████                         | 3.00/4.00 [00:00<00:00, 13.4k Steps/s]A credential is required for accessrage':  50%|██████████████████████████████████████████████████                                                  | 2.00/4.00 [00:18<00:18, 9.29s/ Steps]
user: wittkuhn@mpib-berlin.mpg.de

password: 
password (repeat): 
A credential is required for access
user: wittkuhn@mpib-berlin.mpg.de

password: 
password (repeat): 
publish(ok): . (dataset) [refs/heads/main->owncloud-gwdg:refs/heads/main [new branch]]                                                                                                    
publish(ok): . (dataset) [refs/heads/git-annex->owncloud-gwdg:refs/heads/git-annex [new branch]]                                                                                          
                                                                                                                                                                                         action summary:                                                                                                                                                                            
  copy (error: 2, ok: 8)
  publish (ok: 2)

adswa commented 1 year ago

huh, it also looks like there's something weird happening to the credential, as well. Which datalad-next version is this?

lnnrtwttkhn commented 1 year ago

yes, sorry, forgot about the version details:

datalad 0.18.3
datalad-next 0.6.3
git-annex 10.20221212
git 2.37.1 (Apple Git-137.1)

All on macOS 12.6.3 (Monterey) with M1.

adswa commented 1 year ago

Thanks! I'm trying to reproduce this, and set up an account. Initially, I got an "insufficient storage" error because it seems that I need to configure an external storage location for each folder.

What kind of storage do you have configured?

lnnrtwttkhn commented 1 year ago

I followed @mih's (great!) video instructions:

Get WebDAV address (in my case: https://owncloud.gwdg.de/remote.php/nonshib-webdav/ - usually at the bottom left of the web interface)
Create webdav sibling (append <dataset_name> to webdav address):

datalad create-sibling-webdav \
  --dataset . \
  --name owncloud-gwdg \
  --mode filetree \
  'https://owncloud.gwdg.de/remote.php/nonshib-webdav/<dataset-name>'

Provide access credentials
datalad push --to owncloud-gwdg

That's all. So I didn't configure any storage location manually via the web interface.

adswa commented 1 year ago

huh, interesting! Thanks, I'll try get some storage space, I have none :D

mslw commented 1 year ago

Hey, 2 cents from my side:

I'm also slightly surprised by the repeated prompts for credentials. May be interesting to see what credentials (if any) were saved. These are reported by datalad credentials. The default name template is something like webdav-username-address.
The 403 error on copy looks odd, pretty much random (especially since it's gone on next repetition). And why 403?
If problems persisted, you could try replacing the push with a low-level git annex export HEAD --to owncloud-gwdg-storage --json --json-error-messages to see what exactly git-annex does. But since everything worked in the end, there is no need.

I had similar issues with another hosting provider #202 - but there it was a spurious but persistent 404 error for a DELETE request. So I waved it of with "weird server behaviour". Not even sure if related, just linking so that we can keep track of.

From reading the issue I have no idea what may have caused the intermittent 403 error, but probably the credentials are worth looking into.

datalad / datalad-next