search

estuary / ui

A web based UI to assist in working with Estuary Flow
https://dashboard.estuary.dev/
Other
12 stars 1 forks source link

Enable task shard configuration in UI workflows #510

Open dyaffe opened 1 year ago

dyaffe commented 1 year ago

Bug If a user manually adds shard settings to a materialization via the CLI, then edits that materialization via the UI, we delete them.

Expected We shouldn't blow away those shard settings on deletion

Proposal We could either manually blow those settings away or add maxTxnDuration and minTxnDuration as advanced settings in the UI. The latter would be preferable if possible and the names in the UI would ideally be:

Max Sync Frequency and Min Sync Frequency

jgraettinger commented 1 year ago

Max Sync Frequency and Min Sync Frequency

🤔 @dyaffe This is potentially confusing. It happens to be accurate in the specific case of snowflake, bigquery, or redshift given how those connectors work. It's not generally true of materializations, though, and we will still sync data to other systems continuously and only close transactions (if that's even something the connector considers to be meaningful) at those intervals.

I'm not aware of a more technically-accurate naming than Minimum / Maximum Transaction Duration in terms of what's actually going on.

dyaffe commented 1 year ago

Those names mean kind of nothing to me -- definitely less than the ones I proposed @jgraettinger and I'd imagine they would mean the same too users too

jgraettinger commented 1 year ago

we will still sync data to other systems continuously and only close transactions (if that's even something the connector considers to be meaningful) at those intervals.

🤦 This is inaccurate. Inside of a materialization transaction, we'll stream loads as needed but we do hold back documents to store until the end of the transaction, so calling it a sync frequency actually is pretty reasonable.

kiahna-tucker commented 1 year ago

The defect identified in this issue should be resolved by PR 512. The scope of this defect was larger than originally described, affecting all optional specification properties for materializations and captures alike. As of now, the UI should only set task specification properties that can be edited via a UI form (i.e., endpoint.connector and the resource bindings object property) when the form is submitted (via the Next CTA).

Converting this defect issue into a feature issue for the idea proposed in the initial comment. Adjusted the title to align with the proposal. Please scope the feature request as needed by adjusting the title (e.g., enable task shard configuration in materialization workflows, enable task shard configuration in edit workflows, etc.) and/or adding additional definition in a comment.