gdcc / dvwebloader

A web tool for uploading folders of files to a Dataverse dataset
Apache License 2.0
1 stars 6 forks source link

File upload doesn't start #2

Closed philippconzett closed 1 year ago

philippconzett commented 1 year ago

I tried to start file upload by loading the html file into my Chrome browser using this address:

file:///C:/Users/pco000/Desktop/Files/DVWebloader/dvwebloader.html?siteUrl=https://test-docker.dataverse.no/&datasetPid=doi:10.21337/F3YZSW&key=36be7c2a-8f3b-429a-a9bc-b0abeecc809f

As far as I can see, the address has the same structure as the one I used when testing the previous version of DVWebloader, before Jim put it on GitHub.

When I click Enter, I see "Select a Directory" in the upper left corner of the screen (it looks somewhat different from last time, not like a button anymore, just text).

In Chrome Inspect > Console, I see the following:

dvwebloader.css:1 Failed to load resource: net::ERR_FILE_NOT_FOUND fileupload2.js:1 Failed to load resource: net::ERR_FILE_NOT_FOUND dvwebloader.html?siteUrl=https://test-docker.dataverse.no/&datasetPid=doi:10.21337/F3YZSW&key=36be7c2a-8f3b-429a-a9bc-b0abeecc809f:18 Uncaught ReferenceError: queueFileForDirectUpload is not defined at input.onchange (dvwebloader.html?siteUrl=https://test-docker.dataverse.no/&datasetPid=doi:10.21337/F3YZSW&key=36be7c2a-8f3b-429a-a9bc-b0abeecc809f:18:7)

qqmyers commented 1 year ago

@philippconzett - the version on github has two changes related to the fix for multipart uploads and use as an external tool:

With those two changes you should still be able to run locally and by constructing the Url yourself. That said, now that it works as an external tool, registering it to run from github.io is probably easier, and makes sure you get updates automatically.

philippconzett commented 1 year ago

We have tried to installed the tool as an external tool as suggested in README.md, but we haven't been able to run it. Could you provide some more instructions, preferably with screenshots, on how to install and run it? Thanks!

qqmyers commented 1 year ago

There isn't much to do. One guess would be that I didn't state that dataset-level tools only appear if you have a file(s) in the dataset, so you have to add at least one file before the tool will be listed in the access menu. I've updated the instructions to provide a cut/paste install mechanism and noted the need to add a file.

philippconzett commented 1 year ago

Thanks! We'll give it another try.

philippconzett commented 1 year ago

I have run another test. This time, the files to be uploaded are displayed on the DVWebloader popup window, but after that, no uploading is happening. See the error log below. It seems there is an authorization issue (on our side)?

fileupload2.js:43 https://test-docker.dataverse.no fileupload2.js:45 PID: doi:10.21337/1SC5N1 fileupload2.js:47 null /favicon.ico:1 Failed to load resource: the server responded with a status of 404 () test-docker.dataverse.no/api/datasets/:persistentId/uploadurls?persistentId=doi:10.21337/1SC5N1&size=24&=1664886757537:1 Failed to load resource: the server responded with a status of 401 (Unauthorized) fileupload2.js:180 Failure: 401 fileupload2.js:181 Failure: Unauthorized fileupload2.js:677 Upload error:undefined upid=[object File], Error 401: Unauthorized fileupload2.js:549 35 : 1 : 0 : 4 test-docker.dataverse.no/api/datasets/:persistentId/uploadurls?persistentId=doi:10.21337/1SC5N1&size=179611&=1664886757538:1 Failed to load resource: the server responded with a status of 401 (Unauthorized) fileupload2.js:180 Failure: 401 fileupload2.js:181 Failure: Unauthorized fileupload2.js:677 Upload error:undefined upid=[object File], Error 401: Unauthorized fileupload2.js:549 35 : 2 : 0 : 4 test-docker.dataverse.no/api/datasets/:persistentId/uploadurls?persistentId=doi:10.21337/1SC5N1&size=4184&=1664886757539:1 Failed to load resource: the server responded with a status of 401 (Unauthorized) fileupload2.js:180 Failure: 401 fileupload2.js:181 Failure: Unauthorized fileupload2.js:677 Upload error:undefined upid=[object File], Error 401: Unauthorized fileupload2.js:549 35 : 3 : 0 : 4 test-docker.dataverse.no/api/datasets/:persistentId/uploadurls?persistentId=doi:10.21337/1SC5N1&size=4263&=1664886757540:1 Failed to load resource: the server responded with a status of 401 (Unauthorized) fileupload2.js:180 Failure: 401 fileupload2.js:181 Failure: Unauthorized fileupload2.js:677 Upload error:undefined upid=[object File], Error 401: Unauthorized fileupload2.js:549 35 : 4 : 0 : 4

qqmyers commented 1 year ago

There should be a &key= in the URL when the tool launches - is there? That is only true of you are logged in though and the tool would appear even when you aren't logged in. If that's not it, seeing the whole URL address for the pop-up window might help in figuring it out.

philippconzett commented 1 year ago

No, there is no $key= element in the URL, although I'm logged in (as a super user):

https://gdcc.github.io/dvwebloader/src/dvwebloader.html?siteUrl=https://test-docker.dataverse.no&datasetPid=doi:10.21337/1SC5N1

Could there be a firewall issue in our Azure test environment?

qqmyers commented 1 year ago

Is there a draft version of that dataset? It may be that using this as an explore tool is not sending the key unless it's draft. We could try changing to a config tool which might send the key in all cases. If having a draft fixes this, let me know and I'll check out the config tool idea later today.

I'd be surprised if the firewall would be smart enough to strip some parameters from the URL and not all of them.

As a workaround, you should be able to just add &key=<your api key> to the URL.

philippconzett commented 1 year ago

Yes, there was only a draft version, no published version. And now I just gave it another try after I had published the first version and added some metadata to create a new draft, but I still get the Unauthorized error message in the log.

Could you add a correct URL example including the &key= element?

qqmyers commented 1 year ago

https://gdcc.github.io/dvwebloader/src/dvwebloader.html?siteUrl=https://dv.dev-aws.qdr.org&datasetPid=doi:10.33564/FK2QQIQNF&key=20828f8c-4f21-4300-9c63-f63f3917750e is a good example

philippconzett commented 1 year ago

Thanks, @qqmyers. On my way to work today, I realized what the reason for the error could be: Before I shared an earlier error log with you, I revoked my API token. I now have created a new one, and file upload with DVWebloader started working. However, after a while it stopped uploading. I have pasted the content of the log file below. I see there are still some "401 (Unauthorized)" and "server responded with a status of 404" errors (and maybe other errors I have overlooked). I'm also pinging my colleagues from our cloud migration team (@4tikhonov @Louis-wr @oodu @rolfande); maybe the error is on our side.

Log file:

fileupload2.js:43 https://test-docker.dataverse.no fileupload2.js:45 PID: doi:10.21337/1SC5N1 fileupload2.js:47 bbf1ccc4-fb80-4bae-aee0-b6dcdcda51ec /favicon.ico:1 Failed to load resource: the server responded with a status of 404 () fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :4 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65945d8-caee93481695?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041332Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=51c77802a456c32e284753e1e20d8e934115e5ab1352854517e11309789fdb92","partSize":6553600} fileupload2.js:365 S3 Upload complete for TEST.txt : S3://2002-green-dataversenotest1:183a65945d8-caee93481695 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :3 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65946fe-16615fe3f3b3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041332Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=04690e65995932a816e4383c4fa17f211bf80dcd81bd56cd20d7cd6f7f93beea","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest-3-hexapep.out : S3://2002-green-dataversenotest1:183a65946fe-16615fe3f3b3 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :2 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65947d9-1a7074a60aa7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041332Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=ca3125778661078d537413e5a935c68c23a195e03366e9d17c427a611e399162","partSize":6553600} fileupload2.js:431 handling2 fileupload2.js:549 32 : 1 : 3 : 1 fileupload2.js:365 S3 Upload complete for cyclic-phe-pro-gly-arg-gly-asp-start.xyz : S3://2002-green-dataversenotest1:183a65947d9-1a7074a60aa7 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :2 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65948c3-e3d9c0de36b2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041332Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=1c5bbef75a34af0d1f12d72577eca30f862d9ef08bca866a5950a9d505f09c75","partSize":6553600} fileupload2.js:431 handling2 fileupload2.js:549 31 : 2 : 4 : 1 fileupload2.js:431 handling2 fileupload2.js:549 31 : 3 : 4 : 2 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :3 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6594971-bd37a5bcb6a8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041332Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=18ef98f064f57dc7e183c0bda92a9e438a2f0b19b77c48eeb9f87ddcc89a25fb","partSize":6553600} fileupload2.js:365 S3 Upload complete for wbo : S3://2002-green-dataversenotest1:183a6594971-bd37a5bcb6a8 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 30 : 4 : 5 : 2 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :3 fileupload2.js:260 Num parts: 3 fileupload2.js:177 {"urls":{"1":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a659498e-dc508d926540?uploadId=MWRjZTMwMTAyMTY2NDk0MzIxMzA3MQ&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=5e0f52c55cc15319a3b7b4ff6d65860e7c047d6bbad48646855171f7b6b5d1fb","2":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a659498e-dc508d926540?uploadId=MWRjZTMwMTAyMTY2NDk0MzIxMzA3MQ&partNumber=2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=03ed14603403847576bab85b427d02eecb1a6e1deb1255b43beb7207fd3aa3ac","3":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a659498e-dc508d926540?uploadId=MWRjZTMwMTAyMTY2NDk0MzIxMzA3MQ&partNumber=3&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=f0e9d579dcdb4b3097ea51b2c8e9f923e9e002e3bd6026de8232dae2c7aa8043"},"abort":"/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MWRjZTMwMTAyMTY2NDk0MzIxMzA3MQ&storageidentifier=S3://2002-green-dataversenotest1:183a659498e-dc508d926540","complete":"/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MWRjZTMwMTAyMTY2NDk0MzIxMzA3MQ&storageidentifier=S3://2002-green-dataversenotest1:183a659498e-dc508d926540","partSize":6553600} fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :2 fileupload2.js:260 Num parts: 3 fileupload2.js:177 {"urls":{"1":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6594a06-e505376fb459?uploadId=MWUyZWUwMTA0MTY2NDk0MzIxMzEzOQ&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=8abc4d30c3105aec936865be25a9d6d711a1d8fac297e7daee05a78add29c0d1","2":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6594a06-e505376fb459?uploadId=MWUyZWUwMTA0MTY2NDk0MzIxMzEzOQ&partNumber=2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=4e6e651bc716977ccd6b402b4e1704f1210d84c83a2f52dee595a4c77ebaaecb","3":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6594a06-e505376fb459?uploadId=MWUyZWUwMTA0MTY2NDk0MzIxMzEzOQ&partNumber=3&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=aef0d7f9005a5355b6a072e16e49225394010016dd4f5d13a70b58becb82f0c5"},"abort":"/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MWUyZWUwMTA0MTY2NDk0MzIxMzEzOQ&storageidentifier=S3://2002-green-dataversenotest1:183a6594a06-e505376fb459","complete":"/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MWUyZWUwMTA0MTY2NDk0MzIxMzEzOQ&storageidentifier=S3://2002-green-dataversenotest1:183a6594a06-e505376fb459","partSize":6553600} fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:260 Num parts: 2 fileupload2.js:177 {"urls":{"1":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6594a86-80ed858f655a?uploadId=MTFkNTEzMDEwMzE2NjQ5NDMyMTMyNzM&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=451f5c426b82dda924700de4f40a5655cf29ed02aca16e779028ed42f77b85ad","2":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6594a86-80ed858f655a?uploadId=MTFkNTEzMDEwMzE2NjQ5NDMyMTMyNzM&partNumber=2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=ad1a9d8b573624b9f5c5c1d218ee09ceb4c3bbfdcd2819e010dff62a8ec1f90f"},"abort":"/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MTFkNTEzMDEwMzE2NjQ5NDMyMTMyNzM&storageidentifier=S3://2002-green-dataversenotest1:183a6594a86-80ed858f655a","complete":"/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MTFkNTEzMDEwMzE2NjQ5NDMyMTMyNzM&storageidentifier=S3://2002-green-dataversenotest1:183a6594a86-80ed858f655a","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest_rotamers_0.xyz : S3://2002-green-dataversenotest1:183a65948c3-e3d9c0de36b2 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 27 : 5 : 8 : 0 fileupload2.js:292 Successful upload of part 2 of 2 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6594d75-926ae41ded60?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041333Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=3574a47bb588289cba81a51c191e9459eddeb64fdf5bc69443e935cba6fcdf85","partSize":6553600} fileupload2.js:292 Successful upload of part 3 of 3 fileupload2.js:292 Successful upload of part 3 of 3 fileupload2.js:365 S3 Upload complete for crest_rotamers_4.xyz : S3://2002-green-dataversenotest1:183a6594d75-926ae41ded60 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 26 : 6 : 9 : 0 fileupload2.js:292 Successful upload of part 1 of 3 fileupload2.js:292 Successful upload of part 2 of 3 fileupload2.js:340 reporting file crest_rotamers_1.xyz fileupload2.js:292 Successful upload of part 1 of 3 fileupload2.js:292 Successful upload of part 2 of 3 fileupload2.js:340 reporting file crest_rotamers_2.xyz fileupload2.js:292 Successful upload of part 1 of 2 fileupload2.js:340 reporting file crest_rotamers_3.xyz test-docker.dataverse.no/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MWUyZWUwMTA0MTY2NDk0MzIxMzEzOQ&storageidentifier=S3://2002-green-dataversenotest1:183a6594a06-e505376fb459:1 Failed to load resource: the server responded with a status of 401 (Unauthorized) fileupload2.js:418 Failure: 401 fileupload2.js:419 Failure: Unauthorized test-docker.dataverse.no/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MWRjZTMwMTAyMTY2NDk0MzIxMzA3MQ&storageidentifier=S3://2002-green-dataversenotest1:183a659498e-dc508d926540:1 Failed to load resource: the server responded with a status of 401 (Unauthorized) fileupload2.js:418 Failure: 401 fileupload2.js:419 Failure: Unauthorized fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65954bb-b61f0d7a9ec2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041335Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=8b6ba0d006d5fe7a081c043d234971752b681ab3399aa5b7e88fac2a282d3b31","partSize":6553600} test-docker.dataverse.no/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MTFkNTEzMDEwMzE2NjQ5NDMyMTMyNzM&storageidentifier=S3://2002-green-dataversenotest1:183a6594a86-80ed858f655a:1 Failed to load resource: the server responded with a status of 401 (Unauthorized) fileupload2.js:418 Failure: 401 fileupload2.js:419 Failure: Unauthorized fileupload2.js:365 S3 Upload complete for crest_rotamers_5.xyz : S3://2002-green-dataversenotest1:183a65954bb-b61f0d7a9ec2 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 25 : 7 : 10 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6595ce2-d13d3bbe532f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041337Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=222a89c4b4fda81c502c0a3455d92adeaea0d4e8569f3611e15d549aa948271c","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest_rotamers_6.xyz : S3://2002-green-dataversenotest1:183a6595ce2-d13d3bbe532f fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 24 : 8 : 11 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6596357-851ff19ff259?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041339Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=ff802a84f668702a4f3378b6e985f6e47d27c5e0696fe13da0da0bfd8914ff6e","partSize":6553600} fileupload2.js:365 S3 Upload complete for coord : S3://2002-green-dataversenotest1:183a6596357-851ff19ff259 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 23 : 9 : 12 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6596479-861f93c781b4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041339Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=39f89dcb9c743a58607665b83f8d6890d10e97de14f84b2f8f095e93912ee35c","partSize":6553600} fileupload2.js:365 S3 Upload complete for coord.original : S3://2002-green-dataversenotest1:183a6596479-861f93c781b4 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 22 : 10 : 13 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65965ad-c7123ec5029c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041340Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=c2ac917043fe1d59558f5108a7ded6366b1b2c2af10003520b36dc2482251df6","partSize":6553600} fileupload2.js:365 S3 Upload complete for cregen_0.tmp : S3://2002-green-dataversenotest1:183a65965ad-c7123ec5029c fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 21 : 11 : 14 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65966c3-c5f4c4e1376e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041340Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=02a1d52a851d1c4b3383d9b071773e5b457215af7e120ff9d5359c3de4f05c52","partSize":6553600} fileupload2.js:365 S3 Upload complete for cregen_1.tmp : S3://2002-green-dataversenotest1:183a65966c3-c5f4c4e1376e fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 20 : 12 : 15 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65967e1-1e648b9a4bb9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041340Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=af80305c9ba4edf87a93dfc209ef0f14daca2a2f2fca3df7cb43323828707e49","partSize":6553600} fileupload2.js:365 S3 Upload complete for cregen_2.tmp : S3://2002-green-dataversenotest1:183a65967e1-1e648b9a4bb9 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 19 : 13 : 16 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65968fd-1c211037004d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041341Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=2aa9643f4e246172bd9f95df98d3390a0798061d2e93ae3f834c6da5e085c562","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest_best.xyz : S3://2002-green-dataversenotest1:183a65968fd-1c211037004d fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 18 : 14 : 17 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6596a43-bc63e2a6f1d8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041341Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=2ad58167f25f530c856dc837eb2146f588025ee7d84d6a80ba5ea34bd0073cc2","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest_conformers.xyz : S3://2002-green-dataversenotest1:183a6596a43-bc63e2a6f1d8 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 17 : 15 : 18 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6596fda-c0dc6d5c8adb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041342Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=09c94d7e79a00d5930fa89a1367e56df29d526936ae7025cc2a6edf415c4691a","partSize":6553600} fileupload2.js:365 S3 Upload complete for coord : S3://2002-green-dataversenotest1:183a6596fda-c0dc6d5c8adb fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 16 : 16 : 19 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6597103-b65a64f82f52?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041343Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=a0bab0497d42c506f5e32b53b52f672f24314b4b336a6013d822e948773675dc","partSize":6553600} fileupload2.js:365 S3 Upload complete for coord.original : S3://2002-green-dataversenotest1:183a6597103-b65a64f82f52 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 15 : 17 : 20 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a659722d-22b607231fee?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041343Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=b1fa1cf95835e65237f8f1843ee87ab396b035d211b2e65db78acf8cecfdba19","partSize":6553600} fileupload2.js:365 S3 Upload complete for cregen_0.tmp : S3://2002-green-dataversenotest1:183a659722d-22b607231fee fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 14 : 18 : 21 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6597365-36819bd0ec8e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041343Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=4b021d027fbcffde2de50f4c5d7fdb9b3d2198a1995ca83e3c867b1c76542b13","partSize":6553600} fileupload2.js:365 S3 Upload complete for cregen_1.tmp : S3://2002-green-dataversenotest1:183a6597365-36819bd0ec8e fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 13 : 19 : 22 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6597486-0baf952cf118?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041344Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=00770684519e1897aaf2d2251935462f7c811a86db44b11931ec6f1cfa449672","partSize":6553600} fileupload2.js:365 S3 Upload complete for cregen_2.tmp : S3://2002-green-dataversenotest1:183a6597486-0baf952cf118 fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 12 : 20 : 23 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65975d4-298edba5681a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041344Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=a348f18441e7967583480fc84b7f8a31ea3f894f5baf9fdbd1cb3f9b7a35efbf","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest-3-hexapep.out : S3://2002-green-dataversenotest1:183a65975d4-298edba5681a fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 11 : 21 : 24 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6597747-c83c6f45cb2a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041344Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=11b034631a6af3030002c2f009f4529844d4fb01b527d847ba1096b9bb2bffbc","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest_best.xyz : S3://2002-green-dataversenotest1:183a6597747-c83c6f45cb2a fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 10 : 22 : 25 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a659786c-40dbcef5648b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041345Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=8259ecfd2c087985ce205b75022f7778c40e2d39fb073198b7a85e5c35d6e688","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest_conformers.xyz : S3://2002-green-dataversenotest1:183a659786c-40dbcef5648b fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 9 : 23 : 26 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:177 {"url":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a6597e31-b7349a55ec0e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041346Z&X-Amz-SignedHeaders=host%3Bx-amz-tagging&X-Amz-Expires=7200&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=805846666469b11f53a4a53e1f2035cd71181381103abce7f65a137676779388","partSize":6553600} fileupload2.js:365 S3 Upload complete for crest_rotamers_0.xyz : S3://2002-green-dataversenotest1:183a6597e31-b7349a55ec0e fileupload2.js:375 md5 done fileupload2.js:427 handling fileupload2.js:431 handling2 fileupload2.js:549 8 : 24 : 27 : 0 fileupload2.js:169 Object fileupload2.js:171 Object fileupload2.js:206 Decrementing fip from :1 fileupload2.js:260 Num parts: 3 fileupload2.js:177 {"urls":{"1":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65982cf-49e4e15323d5?uploadId=MTFkNTAzMDEwMDE2NjQ5NDMyMjc2ODI&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041347Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=8a1f6824acba024b5496839b631546f0e0e6666492c3f423a738904646534901","2":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65982cf-49e4e15323d5?uploadId=MTFkNTAzMDEwMDE2NjQ5NDMyMjc2ODI&partNumber=2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041347Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=280a0f4b7e0c1010dbf70d576531f7f90a20675c182cb349859d69eb6f3c7a9a","3":"https://2002-green-dataversenotest1.s3-oslo.educloud.no/10.21337/1SC5N1/183a65982cf-49e4e15323d5?uploadId=MTFkNTAzMDEwMDE2NjQ5NDMyMjc2ODI&partNumber=3&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20221005T041347Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7199&X-Amz-Credential=6c812c2bd143d45b86fb%2F20221005%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Signature=498b33f4265d660cde3a30c81e12fbc3fe4ebf288d507dea4293f30cfa4320a9"},"abort":"/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MTFkNTAzMDEwMDE2NjQ5NDMyMjc2ODI&storageidentifier=S3://2002-green-dataversenotest1:183a65982cf-49e4e15323d5","complete":"/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MTFkNTAzMDEwMDE2NjQ5NDMyMjc2ODI&storageidentifier=S3://2002-green-dataversenotest1:183a65982cf-49e4e15323d5","partSize":6553600} fileupload2.js:292 Successful upload of part 3 of 3 fileupload2.js:292 Successful upload of part 2 of 3 fileupload2.js:292 Successful upload of part 1 of 3 fileupload2.js:340 reporting file crest_rotamers_1.xyz test-docker.dataverse.no/api/datasets/mpupload?globalid=doi:10.21337/1SC5N1&uploadid=MTFkNTAzMDEwMDE2NjQ5NDMyMjc2ODI&storageidentifier=S3://2002-green-dataversenotest1:183a65982cf-49e4e15323d5:1 Failed to load resource: the server responded with a status of 401 (Unauthorized) fileupload2.js:418 Failure: 401 fileupload2.js:419 Failure: Unauthorized

qqmyers commented 1 year ago

Well, some progress. I think I see the issue - the calls to finish a multipart upload aren't sending the key header. Not quite sure how this worked before - perhaps being logged into Dataverse was enough when I tested with a co-located dvwebloader and it only fails when configured as a tool on github. I made an update which should go live to the github.io site within a few minutes. See if it works for you after that.

philippconzett commented 1 year ago

Thanks! I have now been able to upload a larger collection of several ~5 GB files organized in several folders. So far, I have been using Chrome on Windows 10.

I have some other questions, but will add them into a new issue.

qqmyers commented 1 year ago

Cool! FYI - testing on important browsers is a good idea - the basic select a folder widget is non-standard and appears to be supported by most modern browsers, but there could be limitations/differences across browsers.