@chorsley @kxyne Scanned data, that we use for both - production and development - is still remaining in private-bits-cybergreen-net/dev/clean/... directory on amazon S3 and only
We need to move/copy data that belongs to production in private-bits-cybergreen-net/prod/.
Without this it is hard to run aggregation and load script - On each load someone has to manually change environmental variables in the middle of the process...
Eg: Right now, to load data into Redshift, I have to manually point source path to private-bits-cybergreen-net/dev/ to get the raw data and set everything else prod to aggregate, unload, and load into RDS production database... and make data live on http://stats.cybergreen.net/
@chorsley @kxyne Scanned data, that we use for both - production and development - is still remaining in
private-bits-cybergreen-net/dev/clean/...directory on amazon S3 and onlyWe need to move/copy data that belongs to production in
private-bits-cybergreen-net/prod/. Without this it is hard to run aggregation and load script - On each load someone has to manually change environmental variables in the middle of the process...Eg: Right now, to load data into Redshift, I have to manually point source path to
private-bits-cybergreen-net/dev/to get the raw data and set everything elseprodto aggregate, unload, and load into RDS production database... and make data live on http://stats.cybergreen.net/