github-actions[bot]
commented
11 months ago If you are interested in working on this issue, please leave a comment below and we will be happy to assign the issue to you. If this is the first time you are contributing a Pull Request to Cube.js, please check our contribution guidelines. You can also post any questions while contributing in the #contributors channel in the Cube.js Slack.
Is your feature request related to a problem? Please describe. Older version (v0.30) deployed to an environment that was upgraded from Athena V2 to V3. V3 includes a precision on timestamps
timestamp(3)when describing the data type on columns from the information schema. A fix was deployed for this creating errors https://github.com/cube-js/cube/pull/6324.As a consequence of this error, pre-aggregations were being built and then deleted, creating a tremendous amount of activity in memory, on CPU, and on the FS. The volume of attempted (and failing work) remained steady until our monitoring systems alerted us to the anomaly.
Describe the solution you'd like Some kind of exponential backoff or throttling in place for repeating identical queries.
Describe alternatives you've considered By upgrading the deployed versions we're now compatible with the new Athena version, so the problem creating the need for an exponential back off is resolved.
Additional context The underlying problem of Athena compatibility has been fixed, but this incident highlighted an opportunity to improve Cube's ability to mitigate risks by adding throttling or backoff on repeated failure. I can see two current issues raised for search term backoff: https://github.com/cube-js/cube/issues/2506 and https://github.com/cube-js/cube/issues/3904. In both cases, addressing the bug would reduce the need for backoff - but backoff is a nice insurance policy against unknown bugs or incompatibilities generating a lot of resource use.