TDP currently uses python's logging utility to capture debug messages sent by the application to the terminal. These logs are captured by Cloud.gov, Prometheus, and Sentry.
These logs possibly leak sensitive data
Hard to dig back through logs to file errors associated with a particular upload
No visibility into differences in logs between parse runs for the same file
2562 explored and created a tech memo for saving all the log output to a file, allowing end-users to have visibility into past parser runs for a particular datafile submission.
Following the design outlined in #2562
Create and implement the singleton ParseLogContextManager class
Upload the resulting log file to s3, using the same directory as the uploaded datafile
Associate the log file to the submission in the Django Admin Console #3295
Add compression to the log file #3296
Acceptance Criteria:Create a list of functional outcomes that must be achieved to complete this issue
[ ] For every datafile submitted, a log file is created and stored in the same s3 directory
[ ] The log file contains all backend app logs generated from the parser_task (and all subroutines)
[ ] Testing Checklist has been run and all tests pass
[ ] README is updated, if necessary
Tasks:Create a list of granular, specific work items that must be completed to deliver the desired outcomes of this issue
[ ] Create and implement the singleton ParseLogContextManager class (Method/Design steps 1-4 from the tech memo)
[ ] Upload the resulting log file to s3, using the same directory as the uploaded datafile (Method/Design step 5 from the tech memo)
[ ] Run Testing Checklist and confirm all tests pass
Notes:Add additional useful information, such as related issues and functionality that isn't covered by this specific issue, and other considerations that will be helpful for anyone reading this
Note 1
Note 2
Note 3
Supporting Documentation:Please include any relevant log snippets/files/screen shots
Doc 1
Doc 2
Open Questions:Please include any questions or decisions that must be made before beginning work or to confidently call this issue complete
Description:
TDP currently uses python's logging utility to capture debug messages sent by the application to the terminal. These logs are captured by Cloud.gov, Prometheus, and Sentry.
2562 explored and created a tech memo for saving all the log output to a file, allowing end-users to have visibility into past parser runs for a particular datafile submission.
Following the design outlined in #2562
ParseLogContextManagerclassAssociate the log file to the submission in the Django Admin Console#3295Add compression to the log file#3296Acceptance Criteria: Create a list of functional outcomes that must be achieved to complete this issue
parser_task(and all subroutines)Tasks: Create a list of granular, specific work items that must be completed to deliver the desired outcomes of this issue
ParseLogContextManagerclass (Method/Design steps 1-4 from the tech memo)Notes: Add additional useful information, such as related issues and functionality that isn't covered by this specific issue, and other considerations that will be helpful for anyone reading this
Supporting Documentation: Please include any relevant log snippets/files/screen shots
Open Questions: Please include any questions or decisions that must be made before beginning work or to confidently call this issue complete