Load to RDS - cloudified

[rufuspollock]

Have load from S3 to RDS performed "online" rather than locally so that it can be automated and is not dependent on a particular person.

At present we:

• Download CSV from S3 to local disk (they are exported there by Redshift)
• We then load them into tables in our RDS

Options

AWS Lambda

• Get 500MB scratch space so we can download
• Probably very fast
• Problems:
  • Timeouts could be an issue and ultimately space for downloading the CSVs could be a problem
  • Configuration - have to do the environment var hack - ie. load from s3 or dump in the lambda function before upload

ECS - basically means dockerizing this app

• Advantages
  • Easy way to wrap up everything
  • Very few changes
• Disadvantages
  • A little bulky

Choice

Probably go the ECS way as super simple.

Note: this is such a small item that rather than than build standalone we can just integrate it straight into the "Orchestrator"

[rufuspollock]

Note: this is such a small item that rather than than build standalone we can just integrate it straight into the "Orchestrator"

[rufuspollock]

WONTFIX / INVALID. Plan going forward is:

1. to run the aggregator and load to rds from the orchestrator code.
2. to run orchestrator in docker / ECS - see https://github.com/cybergreen-net/infrastructure/issues/13

=> running load to RDS will happen in ECS since orchestrator runs there (basically we won't be running this code from local machine any more).