EXPORT causes problems at a high scale. We've been recommending for a while changefeeds over export. Unfortunately, we're still seeing frequent escalations, such as this one. It would take a large investment in EXPORTs to make them more stable, inclusing adding memory monitoring and porting many of the checkpointing, compression, obs logic over from changefeeds. We are not going to make this investment.
We should do one or several of:
1) Deprecate EXPORT - put a deprecation notice in the docs and when you try to use export, notify you
2) When an EXPORT is initiated, check the number of ranges, disallow over a certain threshold (5000 ranges is the suggested threshold, could also be a cluster setting.
blathers-crl[bot]
commented
1 year ago
EXPORT causes problems at a high scale. We've been recommending for a while changefeeds over export. Unfortunately, we're still seeing frequent escalations, such as this one. It would take a large investment in EXPORTs to make them more stable, inclusing adding memory monitoring and porting many of the checkpointing, compression, obs logic over from changefeeds. We are not going to make this investment.
We should do one or several of: 1) Deprecate EXPORT - put a deprecation notice in the docs and when you try to use export, notify you 2) When an EXPORT is initiated, check the number of ranges, disallow over a certain threshold (5000 ranges is the suggested threshold, could also be a cluster setting.
My recommendation is we do both
Jira issue: CRDB-28969