search

m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.
Apache License 2.0
13 stars 5 forks source link

Add local writer option for persistence of Gardener job status #365

Closed stephen-soltesz closed 2 years ago

stephen-soltesz commented 2 years ago

This change introduces two new flags and preserves default behavior. These flags allow selecting a new "local saver" to persist Gardener state to local disk instead of using Datastore for the v2 system. The v1 "legacy" mode continues to use Datastore.

This change is necessary because we recently encountered a limitation in Datastore object sizes when adding a new datatype configuration #364.

Because default behavior is preserved, this functionality must be enabled explicitly using a flag. The -saver.dir must already exist.

This change is Reviewable

coveralls commented 2 years ago

Coverage Status

Coverage increased (+0.6%) to 61.132% when pulling e7d94898478e9cb3bf8d1f79f2d6691ed5cc8fe8 on sandbox-soltesz-localwriter-2 into 62d6f77709aefc0e4547d4bf1004f9abf00bcbee on master.