Closed ross-spencer closed 5 years ago
Triaged for 1.9.1. as this issue is blocking upgrades for customers who use DSpace + automation tools
When we configure automation tools to transfer dspace files (not directories) we see the following problem with approval:
INFO 2019-03-19 18:38:37 transfer.py:444 Failed transfer approval, try 1 of 3
INFO 2019-03-19 18:38:37 transfer.py:461 Approving uuid_test_2.zip
DEBUG 2019-03-19 18:38:43 utils.py:36 URL: http://127.0.0.1:62080/api/transfer/unapproved; params: {u'username': 'test', u'api_key': 'test'}; method: GET
DEBUG 2019-03-19 18:38:43 utils.py:48 Response: <Response [200]>
DEBUG 2019-03-19 18:38:43 utils.py:49 type(response.text): <type 'unicode'>
DEBUG 2019-03-19 18:38:43 utils.py:51 Response content-type: application/json
INFO 2019-03-19 18:38:43 transfer.py:481 Found waiting transfer: uuid_test_2.zip
DEBUG 2019-03-19 18:38:43 utils.py:36 URL: http://127.0.0.1:62080/api/transfer/approve/; params: {u'directory': 'uuid_test_2.zip', u'type': u'dspace'}; method: POST
DEBUG 2019-03-19 18:38:43 utils.py:48 Response: <Response [500]>
DEBUG 2019-03-19 18:38:43 utils.py:49 type(response.text): <type 'unicode'>
DEBUG 2019-03-19 18:38:43 utils.py:51 Response content-type: application/json
WARNING 2019-03-19 18:38:43 utils.py:58 POST Request to http://127.0.0.1:62080/api/transfer/approve/ returned 500 INTERNAL SERVER ERROR
DEBUG 2019-03-19 18:38:43 utils.py:59 Response: {"message": "Unable to start the transfer.", "error": true}
ERROR 2019-03-19 18:38:43 transfer.py:491 Error approving transfer: Invalid response from server, check amclient log
INFO 2019-03-19 18:38:43 transfer.py:444 Failed transfer approval, try 2 of 3
INFO 2019-03-19 18:38:43 transfer.py:461 Approving uuid_test_2.zip
DEBUG 2019-03-19 18:38:49 utils.py:36 URL: http://127.0.0.1:62080/api/transfer/unapproved; params: {u'username': 'test', u'api_key': 'test'}; method: GET
DEBUG 2019-03-19 18:38:49 utils.py:48 Response: <Response [200]>
DEBUG 2019-03-19 18:38:49 utils.py:49 type(response.text): <type 'unicode'>
DEBUG 2019-03-19 18:38:49 utils.py:51 Response content-type: application/json
INFO 2019-03-19 18:38:49 transfer.py:481 Found waiting transfer: uuid_test_2.zip
DEBUG 2019-03-19 18:38:49 utils.py:36 URL: http://127.0.0.1:62080/api/transfer/approve/; params: {u'directory': 'uuid_test_2.zip', u'type': u'dspace'}; method: POST
DEBUG 2019-03-19 18:38:49 utils.py:48 Response: <Response [500]>
DEBUG 2019-03-19 18:38:49 utils.py:49 type(response.text): <type 'unicode'>
DEBUG 2019-03-19 18:38:49 utils.py:51 Response content-type: application/json
WARNING 2019-03-19 18:38:49 utils.py:58 POST Request to http://127.0.0.1:62080/api/transfer/approve/ returned 500 INTERNAL SERVER ERROR
DEBUG 2019-03-19 18:38:49 utils.py:59 Response: {"message": "Unable to start the transfer.", "error": true}
ERROR 2019-03-19 18:38:49 transfer.py:491 Error approving transfer: Invalid response from server, check amclient log
INFO 2019-03-19 18:38:49 transfer.py:444 Failed transfer approval, try 3 of 3
WARNING 2019-03-19 18:38:49 transfer.py:448 Transfer not approved: uuid_test_2.zip
The change was introduced here: https://github.com/artefactual/archivematica/commit/145df0c1669e49888a7db9d667766ed1e0323f60#diff-60bfea212a0451303313ec7bc09c8770
If we check the following: if transfer is dspace and the object is a file
then we should be able to mitigate this issue. Regular transfers and dspace transfers that are directory based should still be handled regularly even if this check doesn't pass.
A PR is already submitted, I just forgot to press go on this comment before submitting it! ;D
Notes from testing on am191bionic
:
1000 Dspace ZIP files create in a DSpace transfer source:
-rw-r--r-- 1 root root 2146 Apr 3 18:37 dspace-992-aip_1314.zip
-rw-r--r-- 1 root root 2146 Apr 3 18:37 dspace-993-aip_1314.zip
-rw-r--r-- 1 root root 2146 Apr 3 18:37 dspace-994-aip_1314.zip
-rw-r--r-- 1 root root 2146 Apr 3 18:37 dspace-995-aip_1314.zip
-rw-r--r-- 1 root root 2146 Apr 3 18:37 dspace-996-aip_1314.zip
-rw-r--r-- 1 root root 2146 Apr 3 18:37 dspace-997-aip_1314.zip
-rw-r--r-- 1 root root 2146 Apr 3 18:37 dspace-998-aip_1314.zip
-rw-r--r-- 1 root root 2146 Apr 3 18:37 dspace-999-aip_1314.zip
(venv) root@am191bionic:/home/artefactual/archivematica-sampledata/custom-transfers/dspace# ls -la | wc -l
1004
Processing configuration setup for DSpace production environments, i.e. to skip quarantine.
With the cron task set to run every minute, the transfers all start as expected:
dspace-104-aip_1314-06bd9a92-c065-4a1f-8ed1-b46c89d6f840
├── bag-info.txt
├── bagit.txt
├── data
│ ├── logs
│ │ ├── fileFormatIdentification.log
│ │ ├── filenameCleanup.log
│ │ └── transfers
│ │ └── dspace-104-aip_1314-6bbd2c1b-dcad-4601-b060-64a5e350f878
│ │ └── logs
│ │ ├── extractContents.log
│ │ ├── fileFormatIdentification.log
│ │ ├── filenameCleanup.log
│ │ └── verifyChecksumsInFileSecOfDSpaceMETSFiles.log
│ ├── METS.06bd9a92-c065-4a1f-8ed1-b46c89d6f840.xml
│ ├── objects
│ │ ├── dspace-104-aip_1314.zip-2019-04-03T19_10_36.542685_00_00
│ │ │ └── mets.xml
│ │ ├── metadata
│ │ │ └── transfers
│ │ │ └── dspace-104-aip_1314-6bbd2c1b-dcad-4601-b060-64a5e350f878
│ │ │ └── directory_tree.txt
│ │ └── submissionDocumentation
│ │ └── transfer-dspace-104-aip_1314-6bbd2c1b-dcad-4601-b060-64a5e350f878
│ │ └── METS.xml
│ └── README.html
├── manifest-sha256.txt
└── tagmanifest-md5.txt
12 directories, 15 files
Expected behaviour
When I start a DSpace transfer from the API and that transfer is file-only transfer I expect to be able to approve it via the API as well.
Current behaviour
This code introduced in
stable/1.8
means that a slash is appended to the file-path when a transfer is started if it isn't azipped_bag
. Dspace transfers that might just be a zip file will always have a slash appended.Tools using the API might look for a path like:
/var/archivematica/sharedDirectory/watchedDirectories/activeTransfers/Dspace/api_test_2.zip
But Archivematica will look for:
/var/archivematica/sharedDirectory/watchedDirectories/activeTransfers/Dspace/api_test_2.zip/
when attempting to approve the transfer. It will not be possible to approve the transfer.
Steps to reproduce
Attempt to start and approve a DSpace transfer from just a zip file in Archivematica using the API.
Your environment (version of Archivematica, OS version, etc)
qa/1.x
https://github.com/artefactual/archivematica/commits/qa/1.x but this is also the anticipated behavior instable/1.8
.For Artefactual use: Please make sure these steps are taken before moving this issue from Review to Verified in Waffle: