As it stands, I just ensure that the scheduling of the job lines up with the DATE_SUB(NOW(),INTERVAL 1 HOUR) interval where parameters. This breaks down in a couple places.
1) If a job runs slightly ahead late, then we miss whatever happened in the interlude the interval before.
2) If the interval the lambda is scheduled for doesn't line up with the query interval, we either capture too much or not enough.
I see this being a bit more difficult than just storing a date, because as it stands the individual job settings sit inside a JSON payload in each CloudWatch rule. What's really going to need to happen is we have a new table 'Scheduled Jobs' where all the parameters currently stored in CloudWatch are defined. We'll then just pass the Id of the job into the CloudWatch rule. Here we'll also be able to store the last run date, as well as a foreign key to a log table that we want to build in #34. I'm also going to open up #57 to address this larger issue of a separate Table being required.
As it stands, I just ensure that the scheduling of the job lines up with the
DATE_SUB(NOW(),INTERVAL 1 HOUR)interval where parameters. This breaks down in a couple places.1) If a job runs slightly ahead late, then we miss whatever happened in the interlude the interval before. 2) If the interval the lambda is scheduled for doesn't line up with the query interval, we either capture too much or not enough.
I see this being a bit more difficult than just storing a date, because as it stands the individual job settings sit inside a JSON payload in each CloudWatch rule. What's really going to need to happen is we have a new table 'Scheduled Jobs' where all the parameters currently stored in CloudWatch are defined. We'll then just pass the Id of the job into the CloudWatch rule. Here we'll also be able to store the last run date, as well as a foreign key to a log table that we want to build in #34. I'm also going to open up #57 to address this larger issue of a separate Table being required.