Currently, failed files persist in the processing bucket (such as an invalid XML file). This means every time the processing lambda runs (triggered whenever a new file appears in the bucket), it tries to re-processes all the failed files as well as the new file. The impact is the Lambda logs and the Google app engine logs for the crossword microapp are polluted with lots of failures every time a file is processed, making debugging tricky and confusing. It will also be costing us more in compute time.
This pr proposes moving files that have failed to process into a separate failed bucket to allow for debugging.
How to test
Deploy to CODE and add a malformed XML file to the crossword-files-for-processing-code bucket (there are plenty of malformed XML files in the PROD processing bucket to choose from). The uploader lambda should be triggered when a new file is detected and all the malformed XML files in the processing bucket should appear in the failed bucket
What does this change?
Currently, failed files persist in the processing bucket (such as an invalid XML file). This means every time the processing lambda runs (triggered whenever a new file appears in the bucket), it tries to re-processes all the failed files as well as the new file. The impact is the Lambda logs and the Google app engine logs for the crossword microapp are polluted with lots of failures every time a file is processed, making debugging tricky and confusing. It will also be costing us more in compute time.
This pr proposes moving files that have failed to process into a separate failed bucket to allow for debugging.
How to test
Deploy to CODE and add a malformed XML file to the crossword-files-for-processing-code bucket (there are plenty of malformed XML files in the PROD processing bucket to choose from). The uploader lambda should be triggered when a new file is detected and all the malformed XML files in the processing bucket should appear in the failed bucket