Updates the S3 uploading/archiving process to account for the new directory structure and overall processing pipeline.
Updates the S3 downloading process (whether done via make data_init or via python -m data.update --scan=download) to also account for the new directory structure. It also moves all the logic in one place inside make data_init, and has it hinge on the defined scanners for parent domains and subdomains, so that there's less duplication of code and the S3 logic is defined in one place.
There's also a tweak to the production and staging memory sizes, to account for increased vuln scanning to the production site that may have been leading to increased crashes.
This does two main things:
Updates the S3 uploading/archiving process to account for the new directory structure and overall processing pipeline.
Updates the S3 downloading process (whether done via
make data_init
or viapython -m data.update --scan=download
) to also account for the new directory structure. It also moves all the logic in one place insidemake data_init
, and has it hinge on the defined scanners for parent domains and subdomains, so that there's less duplication of code and the S3 logic is defined in one place.There's also a tweak to the production and staging memory sizes, to account for increased vuln scanning to the production site that may have been leading to increased crashes.