Examine WV03_Pan_L1B granule errors for year 2017 in more detail

[krisstanton]

Examine WV03_Pan_L1B granule errors for year 2017 in more detail

There were over 64,000 errors on this ingest run.

We are attempting to account for this by parsing log files, aggregating them and examining them closer.

• [ ] Use a custom version of log aggregator tools to examine the number of retries per error (This process can retry up to 5 times)
• [ ] Clean up MCP bucket temp area when finished
• • [ ] Delete file: s3://csdap-maxar-delivery/utils/jayanthi/ERROR_WV03_2017_log.out.gz
• • [ ] Delete file: s3://csdap-maxar-delivery/utils/jayanthi/WV03_2017_log.out.gz

[krisstanton]

Related (parent) Ticket https://github.com/NASA-IMPACT/csdap-cumulus/issues/259

[krisstanton]

I had to write a specific customization to the Log Aggregator to process this error log file.

Here is the report after running it and tweaking it a bit.

Edit: Note: The first two errors are actually the same error, shown two different ways in the log (Launchpad token errors)

################################################################################
# (number_of_occurences)     |    (unique_string)
# 325612    |    HTTPError: Response code 401 (Unauthorized)
# 390710    |    Failed to ingest, statusCode: 401, statusMessage: Unauthorized, CMR error message: ["Cannot get info for Launchpad token 
# 2618    |    HTTPError: Response code 422 (Unprocessable Entity)
# 4050    |    Failed to ingest, statusCode: 422, statusMessage: Unprocessable Entity, CMR error message: [{"path":["RelatedUrls"],"errors":["Related Urls must be unique. This contains duplicates named
# 168    |    HTTPError: Response code 409 (Conflict)
# 165    |    Failed to ingest, statusCode: 409, statusMessage: Conflict, CMR error message: ["The Granule Ur [null] must be unique. The following concepts with the same granule ur were found: []."]
# 46391    |    HTTPError: Response code 500 (Server Error)
# 56339    |    Failed to ingest, statusCode: 500, statusMessage: Server Error, CMR error message: ["An Internal Error has occurred."]
# 1182    |    HTTPError: Response code 504 (Gateway Time-out)
# 1223    |    Failed to ingest, statusCode: 504, statusMessage: Gateway Time-out
# 3    |    RequestError: Client network socket disconnected before secure TLS connection was established
# 3    |    Failed to ingest, statusCode: ECONNRESET, statusMessage: Client network socket disconnected before secure TLS connection was established
# 2    |    RequestError: socket hang up
# 2    |    Failed to ingest, statusCode: ECONNRESET, statusMessage: socket hang up
# 3    |    Failed to ingest, statusCode: 409, statusMessage: Conflict, CMR error message: ["A concept with 
# 2    |    Failed to ingest, statusCode: 409, statusMessage: Conflict, CMR error message: ["Conflict with existing 
################################################################################

Total Lines: 828473