search

bodleian / ora_data_model

Documentation and crosswalks relating to the ORA data model
1 stars 1 forks source link

Repotool2 sending incorrect headers to Sword2 for file deposits #151

Open tomwrobel opened 4 years ago

tomwrobel commented 4 years ago

Related to #14

When repotool deposits a file, it is sending the 'in progress' flag incorrectly for file and file metadata actions. For object-level actions its behaviour is correct.

The in_progress flag should indicate that more activity is set to take place on an object. When set to False, it indicates that deposit has finished and that the receiving server (Hyrax) should act on the object.

See the scenarios below, in each case, the in_progress header was set correctly for object creation (set to True), and incorrectly for file creation and metadata update (set to False for intermediate actions and to True for the final action).

Log file results are in the comment below.

tomwrobel commented 4 years ago

Note that files were added to the UI in number order (one.txt, two.txt, three.txt) etc., and were received in number order. It is not the case that the order of file save has thrown anything off

Depositing a single file to 1012334 (uuid_f99fda65-345b-498b-a040-260df3

fbc7dc)

Action url (after sword/collections/default/works/) method in_progress value
Create new work POST uuid_f99fda65-345b-498b-a040-260df3fbc7dc True
Deposit new file (eleven.txt) POST uuid_f99fda65-345b-498b-a040-260df3fbc7dc/file_sets False
Set (actually update) binary file metadata for file PUT uuid_f99fda65-345b-498b-a040-260df3fbc7dc/file_sets/sympc247ds08k True
Harvest information from Hyrax GET uuid_f99fda65-345b-498b-a040-260df3fbc7dc nil

Depositing two files to 1012335 (uuid_f99fda65-345b-498b-a040-260df3

fbc7dc)

Action url (after sword/collections/default/works/) method in_progress value
Create new work POST uuid_8b710238-a33d-43ab-85e2-b2dab44d08db True
Deposit first new file (one.txt) POST uuid_8b710238-a33d-43ab-85e2-b2dab44d08db/file_sets False
Set (actually update) binary file metadata for file one.txt PUT uuid_8b710238-a33d-43ab-85e2-b2dab44d08db/file_sets/symppn89d656c False
Deposit second new file (two.txt) POST uuid_8b710238-a33d-43ab-85e2-b2dab44d08db/file_sets False
Set (actually update) binary file metadata for file two.txt PUT uuid_8b710238-a33d-43ab-85e2-b2dab44d08db/file_sets/symp1544bp080 True
Harvest information from Hyrax GET uuid_8b710238-a33d-43ab-85e2-b2dab44d08db nil

Depositing three files to 1012336 (uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a)

Action url (after sword/collections/default/works/) method in_progress value
Create new work POST uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a True
Deposit first new file (one.txt) POST uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets False
Set (actually update) binary file metadata for file one.txt PUT uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets/sympx059c732j False
Deposit second new file (two.txt) POST uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets False
Set (actually update) binary file metadata for file two.txt PUT uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets/symphh63sv88g False
Deposit third new file (three.txt) POST uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets False
Set (actually update) binary file metadata for file three.txt PUT uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46a/file_sets/sympxg94hp52h True
Harvest information from Hyrax GET uuid_d6d4f045-5fb7-4cc4-846e-f612b058b46aqq nil

Redepositing a single file - add eleven.txt to 1012310 (uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4)

Action url (after sword/collections/default/works/) method in_progress value
Deposit new file (eleven.txt) POST uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets False
Set (actually update) binary file metadata for file PUT uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets/symp8s45q877q True
Harvest information from Hyrax GET uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4 nil

Redepositing multiple files - and nine.txt and ten.txt to 1012310 (uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4)

Action url (after sword/collections/default/works/) method in_progress value
Deposit first new file (nine.txt) POST uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets False
Set (actually update) binary file metadata for first file PUT uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets/symp1831cj92d False
Deposit second new file (ten.txt) POST uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets False
Set (actually update) binary file metadata for second file PUT uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4/file_sets/sympbg257f046 True
Harvest information from Hyrax GET uuid_1ef0f790-ab00-4703-b22b-d8d4a48fcba4 nil

Note that files were added to the UI in number order (one.txt, two.txt, three.txt) etc. It is not the case that the order of file save has thrown anything off.

tomwrobel commented 4 years ago

@AndrewBennet there is a workaround for this: I can invert the meanings of the in_progress flag for filesets to match Repotool2's current behaviour. We have to be VERY careful to undo this once it's fixed though. Thoughts?

The workaround is to invert the significance of in_progress = true and in_progress=false in app/controllers/concerns/integrator/hyrax/file_sets_behaviour.rb. A comment has been left in that file to link to this ticket

tomwrobel commented 4 years ago

Workaround is in place: github.com/tomwrobel/willow_sword feature/ora_customizations commit commit 995c7ef4e3a24d901e3956672a56a8528a4420df

AndrewBennet commented 4 years ago

You are quite right - I've made the classic programming error of inverting my booleans (when dealing with the files only).

We should be able to fix in a 5.18 patch. I'll let you know which version this gets in to, so you can remove your workaround when upgraded.

tomwrobel commented 4 years ago

Brilliant! I'll move to post-release!

AndrewBennet commented 4 years ago

Just to let you know: we released this in the patch to 5.18 out today (and also included in a 5.19 patch). You'll need to coordinate your workaround-removal with the Elements upgrade, of course :)

tomwrobel commented 4 years ago

There is now a configuration flag in willow_sword that is set in Hyrax and will be set by ansible. If config.in_progress_workaround = true the workaround will be used, once fixed, this value can be removed or set to false