search

HumanCellAtlas / upload-service

DCP Ingestion File Upload Service
MIT License
4 stars 4 forks source link

Ingest receives 404 when trying to stage some metadata files. Need to investigate #278

Closed parthshahva closed 5 years ago

parthshahva commented 5 years ago 
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/5b822e19-1858-4e50-a0a6-7d38bb7ad03a/links_76758683-91bd-44ca-93d1-36cd2c789a54.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/6d0a3148-af17-46d6-9860-1b0abde41f0a/analysis_file_3bc988cb-dd8c-4085-a700-38a7f388cb34.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/fae6b803-5e10-4c2b-af25-354c31b92564/analysis_protocol_9ffae6a9-ad26-42b1-a4ba-85aee8b1224f.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/a84fcca8-3352-4dd3-8b03-9420528c0e2c/links_de72edf5-e062-4bb7-80a3-2371bdf6ca5d.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/5d593bf2-0522-4b71-be9e-d659c6344cc6/analysis_file_c77ee77e-96be-4f57-a667-7099d35c6ae7.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/d6e35e8e-4d8d-4578-8211-7ac70e6bf4a0/analysis_file_4cb26077-c30c-45a0-a14b-31dea37c6fd8.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/b7aa9d4f-9d72-4450-a0dd-63da4c5dd7b0/analysis_file_d97bc766-03fd-4805-884a-2f3211ec34d2.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/e1339117-d33f-4328-9576-11d3cc52451a/analysis_file_7a152000-11c0-42af-9295-0a91cbdcabe5.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/d397aed0-e425-40d0-b2fc-087b9b34961b/sequence_file_d1d9da6f-a0ec-4e59-b960-ba42bd492aa1.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/37837f2c-2329-4820-b921-c4f5306a6d4b/analysis_file_4c5a006f-7fa3-423d-a5d9-bf94ec101ba7.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/923506c5-41c4-4f9c-aa28-2172cfd65cb5/analysis_file_0d836299-b5f9-4bc8-8e51-d6cacea653d3.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/34e43fa6-d547-49ab-93dc-53ddb3dd945d/analysis_file_ebc20507-2548-4451-88e7-e2df94334457.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/73292dca-e032-4d87-a689-9ed9aead2a50/analysis_file_5b6129f9-e680-4de8-9b2a-cfa743987952.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/5cc59542-adac-4e10-8491-bdcb72a8fdf7/analysis_file_042c875c-c05d-4ec8-8ea9-7f7f50423f1c.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/b02c40f0-1a02-4e11-8ab8-add8587afc7d/analysis_file_c8d6d30a-c169-497c-a249-823719395a5d.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/d397aed0-e425-40d0-b2fc-087b9b34961b/process_2fddf176-ff6b-4676-bd5b-76a3e9fe1029.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/c2dde83b-d7b9-4865-8bf6-c5ba2ee4b539/analysis_file_c34ef34b-20ae-4e5d-9050-f574a0d89362.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/dbb337f0-94ee-4600-8ff9-b7a89fdf159f/analysis_file_b713b69c-9cbb-4ffe-a265-7c39e49d191b.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/99a14b8f-3b50-47c9-b8b9-e80599f2fdd0/analysis_file_831b736e-3385-40fc-b9cd-0147802683bd.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/be5c3148-ee82-4204-aca4-8ade53c45cf1/analysis_file_39cf4197-3300-48af-93ef-c5fcb3441d03.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/cdf16524-4c39-4c20-a77e-7bc9075cd209/analysis_file_daad5410-2faf-4644-9924-eeda66caff42.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/f0ffb5b4-b39c-4a90-8896-94e1b29de218/analysis_file_5e35daa5-6060-47dd-9d46-6dc07fd3c139.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/c771bdf1-2945-4d4c-bc59-d49910182775/analysis_file_d9effcbb-315f-4d31-ab82-26f9f087ab24.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/002f6700-47db-460e-aedd-0324c13302fe/analysis_file_68f799c2-00f4-41e8-b023-695b6ece0129.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/eccd695c-6db0-48d9-b8cd-5b3312480c98/analysis_file_9b33a534-2ea1-42d7-ba32-5f3a3f369399.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/9e17ffc1-5529-4f37-aaf7-695c76eff16b/analysis_file_1f577dff-299f-4cc6-9bc9-afc294a74b96.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/f24ac1b9-dd68-4b4c-afe5-2f12f0979999/analysis_file_ab637bf9-ede7-4818-ac66-ce9744dfa13e.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/78b17f9e-a79c-4cfb-a438-d10855505fb2/analysis_file_f5e3c14f-96df-4045-90b5-790548ba6739.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/167adec2-9cdd-40a4-b522-1791fed4d3e1/analysis_file_501089ab-49cb-4ced-b4cd-401e6be05adb.json
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/14802326-44ee-41be-9358-bd076f3f0422/analysis_file_d8dde61f-144f-43d7-8e04-8d1c48d290c4.json
sampierson commented 5 years ago

Would it be possible to get a timeframe for one of these error message? For example the first one - when was this message received?

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/5b822e19-1858-4e50-a0a6-7d38bb7ad03a/links_76758683-91bd-44ca-93d1-36cd2c789a54.json

And to confirm - this was generated from a PUT /v1/<uuid>/<filename>?

sampierson commented 5 years ago

Ingest reported:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://upload.integration.data.humancellatlas.org/v1/area/5b822e19-1858-4e50-a0a6-7d38bb7ad03a/links_76758683-91bd-44ca-93d1-36cd2c789a54.json

scripts/dcpdig -d integration @upload area=5b822e19-1858-4e50-a0a6-7d38bb7ad03a —show all

The file links_76758683-91bd-44ca-93d1-36cd2c789a54.json that generated the error does not have a file record.

Kibana says:

Searching from 2019-01-28 13:00:00 to 2019-01-28 15:59:59 for 5b822e19-1858-4e50-a0a6-7d38bb7ad03a/links_76758683-91bd-44ca-93d1-36cd2c789a54.json:

4 hits total describing 2 requests:

@log_group:API-Gateway-Execution-Logs_iz05wpatc9/integration
@message: (eb91e82c-2304-11e9-af96-81bdf247f3fe) HTTP Method: GET, Resource Path: /v1/area/5b822e19-1858-4e50-a0a6-7d38bb7ad03a/links_76758683-91bd-44ca-93d1-36cd2c789a54.json
@timestamp:January 28th 2019, 13:59:25.060

@log_Group: API-Gateway-Execution-Logs_iz05wpatc9/integration
@message: (eb9a4ccf-2304-11e9-ba66-4d36ef5d7711) HTTP Method: PUT, Resource Path: /v1/area/5b822e19-1858-4e50-a0a6-7d38bb7ad03a/links_76758683-91bd-44ca-93d1-36cd2c789a54.json
@timestamp: January 28th 2019, 13:59:25.113

Interestingly that a GET happened before a PUT.

Searching for just links_76758683-91bd-44ca-93d1-36cd2c789a54.json gets 10 hits, including stuff from the Lambda log:

@message:[INFO] 2019-01-28T13:59:25.128Z    559d00ca-b8df-48af-87b9-a65af3fc4633    Running <function put_file at 0x7f0bb7d81d90> with args=() kwargs={'upload_area_uuid': '5b822e19-1858-4e50-a0a6-7d38bb7ad03a', 'filename': 'links_76758683-91bd-44ca-93d1-36cd2c789a54.json', 'body':

Searching for that Lambda execution ID we see:

@message:[ERROR]    2019-01-28T13:59:25.177Z    559d00ca-b8df-48af-87b9-a65af3fc4633    Returning rfc7807 error response: status=404, title=No such file, detail=No such file in that upload area
@log_group:/aws/lambda/upload-api-integration
@timestamp:January 28th 2019, 13:59:25.177

This error can be generated from two places:

I think… eventual consistency. put_file calls UploadedFile.create() which does a s3object.put() to create thre acutal S3 object, then immediately calls UploadedFile.init() which attempts to read back from that object. If the write and read calls, that happen within milliseconds of each other hit different S3 handlers, I could see a case where the object wasn’t yet consistent across all the S3 handers.

Recommendation: we add an @retry around UploadedFile._s3_load

parthshahva commented 5 years ago

I second this recommendation. I think this is the cleanest solution to account for the eventual consistency