Pilot project related works not updating status or related work

[briri]

Describe the bug When you go into a DMP in the DMP Upload section of the site, and go to the 'Related Works' page, the system does not record the changes you make when you 'approve' or 'reject' that the item is related to the DMP.

To Reproduce Steps to reproduce the behavior:

1. Go to 'DMP Upload'
2. Click on a DMP that shows that it has 'n related works' under the title
3. Navigate to the 'Related Works' page
4. Click on one of the work
5. Make sure that the 'Current Status' is 'Pending'
6. Click either the 'Mark as related' or 'Mark as unrelated' buttons
7. Click 'Save'
8. Return to the 'Related Works' page an open the same related work. The'Current Status' still shows as 'Pending'

Expected behavior The status changes to 'Approved' or 'Rejected'

Screenshots

Additional context

A review of the logs in CloudWatch shows that the PutDmp API is receiving 'pending' for the status. It appears as though the React page is not sending the updated status when 'Save' is clicked. It is sending 'pending' for the status.

[jupiter007]

When running the dmptool locally, I was able to successfully save the updated status of the related work.

However, when attempting the same thing on Stage, I get the 401 Unauthorized error when attempting to save the updates.

Here is a copy of one of the fetch urls that failed: https://dmptool-stg.cdlib.org/api/v3/dmps/doi.org/10.48321/D14F10A7F8

[jupiter007]

Brian helped me figure out how to get around the "401 Unauthorized" error by inserting my user_id into the "drafts" table record for the plan that has related works. So now I can update "related works" on Stage.

What I see on Stage:

1. When I make the related works update, I see the correct Payload being sent in the request in the Network tab of my browser, and I see the correct, updated "status" of the related work in the request response in the Network tab.
2. When I checked the DynamoDB table for Stage, I see the updated value in the "VERSION#latest" record for the associated PK
3. However, when I load the related works page, the "GET" response returns the old value.

The above leads me to believe there could be some caching happening on Stage.

This is not happening in the "dev" environment. I noticed that the "dev" Response Header has "Cache-control" set to "no-store", whereas the "stage" Response Header does not have that. I'm wondering if that is the cause.

I checked the AWS Console, and it looks like caching has been provisioned on Stage for the Lamba function for GET requests:

There is no cache provisioned for Dev in the API Gateway.

Lastly, I made some requests from outside of the app using Postman, just to confirm that this has nothing to do with the app itself. I made a PUT request and then a GET request, and I saw the same issue with the correct data in the payload and response of the PUT, but the wrong data returned for the GET request

[briri]

Looked at the uc3-dmp-id gem's updater class and it looks like the _process_harvester_mods function is failing for some reason. The DMP record has the dmphub_modifications block which it should not, it should be overwriting the HARVESTER_MODS record instead

[briri]

• Need to update the Harvester to include "domain": "blah blah" for each related_work
• Need to update the uc3-dmp-id gem's Updater class _process_harvester_mods function to compare against the harvester mod's identifier an domain together instead of the key