parallelize on Amazon Web Services using Simple Queue Service (SQS)

simsong / bulk_extractor

This is the development tree. Production downloads are at:

https://github.com/simsong/bulk_extractor/releases

Other

1.04k stars 183 forks source link

parallelize on Amazon Web Services using Simple Queue Service (SQS) #101

Open simsong opened 6 years ago

simsong commented 6 years ago

Make bulk_extractor run with disk images stored in EBS or on Amazon S3. Distribute work using SQS. Currently AWS is limited to m5.24xlarge nodes with 96 vCPUs. With this addition, you could have 1000 CPU cores running...

simsong commented 4 years ago

Or even better, run on Lambda.