search

m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.
Apache License 2.0
13 stars 5 forks source link

Dailyonly option #352

Closed gfr10598 closed 1 year ago

gfr10598 commented 2 years ago

This introduces a dailyOnly option, that suppresses reprocessing of old data for a datatype. It will, of course, cause alerts to fire.

This is currently built on top of a change to provide faster build times, but that will be merged first. Although it has sandbox in the name, deployment is currently being suppressed, as another branch that incorporates this change and others is currently running in sandbox.

This change is Reviewable

coveralls commented 2 years ago

Coverage Status

Coverage decreased (-0.06%) to 60.379% when pulling 23818a81b6b004b3e3a62b294e07f2fb3b441d97 on sandbox-gfr-dailyonly-option into 3bd6e68f160ad4ab730088ceb3801c4813873f04 on master.

gfr10598 commented 2 years ago

Ok - deploying to sandbox with one extra commit to make ndt7 dailyOnly. Unfortunately, we won't see if it works correctly until tomorrow morning.

The branch now has just the code changes, without config change.

gfr10598 commented 2 years ago

Argh - looks like I screwed that up and lost the code changes when I removed the last commit. Fixing now.

stephen-soltesz commented 2 years ago

@gfr10598 will you have a chance to complete this today? several PRs would be helped by this change #359 , #350 and more to come.

stephen-soltesz commented 2 years ago

Hmm, I think I discovered that this approach won't quite work as I imagined for https://github.com/m-lab/etl-gardener/pull/350 -- the job keys are used as the actual jobs by other parts of the gardener system, so some state is lost. Modifications are getting harder to reason about and coordinate on without an architectural design. I'd like to help create that.

stephen-soltesz commented 1 year ago

Obsolete by: