brexhq / substation

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.
https://substation.readme.io
MIT License
330 stars 21 forks source link

chore(scripts): Add Concurrency to S3 Rehydration #172

Closed jshlbrd closed 6 months ago

jshlbrd commented 6 months ago

Description

Motivation and Context

This makes S3 rehydration go considerably faster and fixes an issue when paging through more than 1,000 objects. Unless the client has connectivity issues with SNS, there should no error checking required on the threads (the SNS service has its own message retry policy). Futures are stored in a list in case we want to make error checking an option later.

How Has This Been Tested?

Tested with a production deployment (similar to the S3/SNS example in the repo).

Types of changes

Checklist: