search

ebi-ait / dcp-ingest-central

Central point of access for the Ingestion Service of the HCA DCP
Apache License 2.0
0 stars 0 forks source link

update MA project metadata on submission #1038

Closed idazucchi closed 1 month ago

idazucchi commented 2 months ago

As wrangler I want to update managed access projects' metadata when uploading a submission so that adding project metadata is faster and more accurate

Acceptance Criteria / Definition of Done

timeline by mid October

idazucchi commented 2 months ago

Testing the updated version of the lambda function

  1. no additional parameters, just submission upload

    aws lambda invoke --function-name dcp-secure-spreadsheet-upload-auth --payload '{"folder_name": "51a98b2a-a1a9-473d-bf65-d91c9332afbb", "file_name": "RiskVariantsAF_ontologies.xlsx" }' response.json
$LATEST 200
idaz@U-13Q0EYKSQCS37:~$ cat response.json 
{"statusCode": 500, "body": "{\"errorType\": \"Exception\", \"errorMessage\": \"Error for project UUID: 50242e39-8373-4e53-8d96-bb829a27a625: prepare_notification() got an unexpected keyword argument 'is_update'\", \"projectUUID\": \"50242e39-8373-4e53-8d96-bb829a27a625\", \"bucketName\": \"hca-util-upload-area\", \"objectKey\": \"51a98b2a-a1a9-473d-bf65-d91c9332afbb/RiskVariantsAF_ontologies.xlsx\"}"}

    despite the error message the submission is created - it's in draft status but this is an unrelated issue to the lambda function project metadata remains the same

  2. add "isUpdate", "updateProject" set to true to update project metadata

    aws lambda invoke --function-name dcp-secure-spreadsheet-upload-auth --payload '{"folder_name": "51a98b2a-a1a9-473d-bf65-d91c9332afbb", "file_name": "RiskVariantsAF_ontologies.xlsx", "isUpdate": true, "updateProject": true }' response.json
$LATEST 200
idaz@U-13Q0EYKSQCS37:~$ cat response.json 
{"statusCode": 500, "body": "{\"errorType\": \"Exception\", \"errorMessage\": \"Error for project UUID: 50242e39-8373-4e53-8d96-bb829a27a625: prepare_notification() got an unexpected keyword argument 'is_update'\", \"projectUUID\": \"50242e39-8373-4e53-8d96-bb829a27a625\", \"bucketName\": \"hca-util-upload-area\", \"objectKey\": \"51a98b2a-a1a9-473d-bf65-d91c9332afbb/RiskVariantsAF_ontologies.xlsx\"}"}

    no submission is created project metadata remains the same

  3. add "isUpdate", "updateProject" set to false to keep project metadata

    aws lambda invoke --function-name dcp-secure-spreadsheet-upload-auth --payload '{"folder_name": "51a98b2a-a1a9-473d-bf65-d91c9332afbb", "file_name": "RiskVariantsAF_ontologies.xlsx", "isUpdate": false, "updateProject": false }' response.json
$LATEST 200
idaz@U-13Q0EYKSQCS37:~$ cat response.json 
{"statusCode": 500, "body": "{\"errorType\": \"Exception\", \"errorMessage\": \"Error for project UUID: 50242e39-8373-4e53-8d96-bb829a27a625: prepare_notification() got an unexpected keyword argument 'is_update'\", \"projectUUID\": \"50242e39-8373-4e53-8d96-bb829a27a625\", \"bucketName\": \"hca-util-upload-area\", \"objectKey\": \"51a98b2a-a1a9-473d-bf65-d91c9332afbb/RiskVariantsAF_ontologies.xlsx\"}"}

    despite the error message the submission is created - it's in draft status but this is an unrelated issue to the lambda function project metadata remains the same

KociOrges commented 2 months ago

I've deployed an updated version of the Lambda function with the bug resolved. The is_update and update_project parameters are now optional in the notification message and when calling the Lambda.

idazucchi commented 1 month ago

Testing the update

  1. no additional parameters, just submission upload

    aws lambda invoke --function-name dcp-secure-spreadsheet-upload-auth --payload '{"folder_name": "51a98b2a-a1a9-473d-bf65-d91c9332afbb", "file_name": "RiskVariantsAF_ontologies.xlsx" }' response.json
$LATEST 200
idaz@U-13Q0EYKSQCS37:~$ cat response.json 
{"statusCode": 200, "body": "\"Notification sent successfully for project UUID: 50242e39-8373-4e53-8d96-bb829a27a625 and submission ID: 66e3181c86816b2568b2c411.\""}

    The error message is gone and the submission is created project metadata remains the same

  2. add "isUpdate", "updateProject" set to true to update project metadata

    aws lambda invoke --function-name dcp-secure-spreadsheet-upload-auth --payload '{"folder_name": "51a98b2a-a1a9-473d-bf65-d91c9332afbb", "file_name": "RiskVariantsAF_ontologies.xlsx", "isUpdate": true, "updateProject": true }' response.json
$LATEST 200
idaz@U-13Q0EYKSQCS37:~$ cat response.json 
{"statusCode": 200, "body": "\"Notification sent successfully for project UUID: 50242e39-8373-4e53-8d96-bb829a27a625 and submission ID: 66e31ca986816b2568b2c5bc.\""}

    no submission is created but no error message is sent - odd project metadata remains the same

  3. add "isUpdate", "updateProject" set to false to keep project metadata

    aws lambda invoke --function-name dcp-secure-spreadsheet-upload-auth --payload '{"folder_name": "51a98b2a-a1a9-473d-bf65-d91c9332afbb", "file_name": "RiskVariantsAF_ontologies.xlsx", "isUpdate": false, "updateProject": false }' response.json
$LATEST 200
idaz@U-13Q0EYKSQCS37:~$ cat response.json 
{"statusCode": 200, "body": "\"Notification sent successfully for project UUID: 50242e39-8373-4e53-8d96-bb829a27a625 and submission ID: 66e31da386816b2568b2c76a.\""}

    the submission is created and the error is gone - the submission uuid stays the same for all tests, but doesn't match the uuid visible in the ui project metadata remains the same

idazucchi commented 1 month ago

project update is working!! "isUpdate" set to false because it's a new submisssion, "updateProject" set to true to update project metadata

aws lambda invoke --function-name dcp-secure-spreadsheet-upload-auth --payload '{"folder_name": "51a98b2a-a1a9-473d-bf65-d91c9332afbb", "file_name": "RiskVariantsAF_ontologies.xlsx", "isUpdate": false, "updateProject": true }' response.json
$LATEST 200
idaz@U-13Q0EYKSQCS37:~$ cat response.json 
{"statusCode": 200, "body": "\"Notification sent successfully for project UUID: 50242e39-8373-4e53-8d96-bb829a27a625 and submission ID: 66e9581086816b2568b2dbaa.\""}
idazucchi commented 1 month ago

project + submission update also works however it requires to download the submission spreadsheet with the uuids from ingest so we've agreed not to do it for reference: "isUpdate" set to true to update the submisssion, "updateProject" set to true to update project metadata

aws lambda invoke --function-name dcp-secure-spreadsheet-upload-auth --payload '{"folder_name": "51a98b2a-a1a9-473d-bf65-d91c9332afbb", "file_name": "42f9e276-0971-43e6-acf0-eab5fdcf00d2_20240917-103152.xlsx", "isUpdate": true, "updateProject": true }' response.json
$LATEST 200
idaz@U-13Q0EYKSQCS37:~$ cat response.json 
{"statusCode": 200, "body": "\"Notification sent successfully for project UUID: 50242e39-8373-4e53-8d96-bb829a27a625 and submission ID: 66e9817786816b2568b2e070.\""}
KociOrges commented 1 month ago

This has been now pushed to production, with isUpdate and updateProject parameters available. Relevant wiki guide has been updated accordingly.