Closed sploiselle closed 4 weeks ago
@rjobanp I tinkered with this a little bit and, unfortunately, not quite as easy I had thought. Wanted to ping you lest it feel like I suggest you go on an interminable yak shave.
In the PG source, we've essentially hard-coded that the output_index
of messages lines up with their position in PostgresSourcePublicationDetails
's tables
field. This means there's no way to demux tables input into multiple shards. Assume this is similar for the MySQL source.
My current hunch is that this is most easily fixed by adding a kind of "key" to SourceExport
for each type of source which explains which upstream object this GlobalId
corresponds to.
We can then maybe remove the output_index
field on SourceExport
and rely on enumerating over source_exports
to determine the output position.
#[derive(Clone, Debug, Serialize, Deserialize, Eq, PartialEq)]
enum SouceExportConfig {
/// PG subsources use the table OID
Pg(u32),
/// Kafka only exports a single collection,
Kafka,
/// Kafka exports are correlated to the upstream table's name
MySql(UnresolvedItemName),
}
#[derive(Clone, Debug, Serialize, Deserialize, Eq, PartialEq)]
pub struct SourceExport<S = ()> {
/// The collection metadata needed to write the exported data
pub storage_metadata: S,
pub config: SouceExportConfig,
}
Just a quick sketch––we might want to do this by making things a trait or something. Not suggesting this is what the final code will or should look like, just that this is a general direction that would get us out of the restriction we currently have.
Awesome thanks for the context!
In the PG source, we've essentially hard-coded that the output_index of messages lines up with their position in PostgresSourcePublicationDetails's tables field. This means there's no way to demux tables input into multiple shards. Assume this is similar for the MySQL source.
Yes right now I implemented this for MySQL very similar to Postgres: We assume that the order of tables within the MySqlSourceConnection
tables
field is the same as the output_index
ordering for each sub-source.
This was definitely one of the most brittle things I noticed when working through the MySQL source so I like the idea of us removing this restriction and using the source-specific 'key' to map GlobalIds to their corresponding output (and therefore supporting 1:N table mappings).
It still seems that we would need to store the output_index
on the SourceExports
struct, right? Since we store a BTreeMap<GlobalId, SourceExport<G>>
on both the RawSourceCreationConfig
and IngestionDescription
I'm not sure that we can rely on the iteration order to determine position
BTreeMap
offers stable and ordered iteration (arguably the feature of a b-tree vs. hash map). Or maybe it's important to note that we have to rebuild the entire dataflow if the exports change. Either way. I have this working locally using enumerate
while iterating over the b-tree map to determine output indexes.
The complex change, though, is swapping out output_index
in SourceExport
for something that lets us correlate source exports to an upstream table. Will see what everyone thinks later this week probably.
BTreeMap offers stable and ordered iteration
I thought this was ordered by-key though, right? Unless you're saying that GlobalId
s are ordered too such that we can assume a new entry in the map would always sort to the end of the keys?
The complex change, though, is swapping out output_index in SourceExport
Yep I briefly looked up references to SourceExport
and it looks fairly non-trivial to refactor all the uses :)
Yep it is ordered by key. The output index, though, just needs to be consistent between the ingestion dataflow and where we correlate output indexes to persist shards. That gets re-built every time we restart the dataflow, so supports things like ALTER SOURCE...ADD SUBSOURCE
changing the source exports by just re-figuring it out. i.e. the output index is just a transient mechanism to support multi-output ingestions, which we have to orchestrate in two separate spots.
Ahh that makes sense, thanks for explaining!
@petrosagg suggests that this feature should not violate the contract that SHOW CREATE SOURCE
shows you how to recreate an environment in exactly its current state. The only way to solve this is to allow creating subsources with projections from the input table (maybe they must be prefixes, maybe the allow reordering). https://github.com/MaterializeInc/materialize/pull/26871#pullrequestreview-2045572908 Assigning to him to ensure the scope of that work is to the spec he envisions.
Closing this issue in favor of tracking in https://github.com/MaterializeInc/materialize/issues/28430
Feature request
Materialize currently:
This means that if a user does want to start ingesting new columns from a table, their only option is to drop the subsource and re-add it. This introduces downtime, which a user can avoid only by doing a blue-green deployment on the entire source.
However, this issue can be largely resolved by supporting 1:N mappings between upstream database tables and persist shards. Users could then:
Conceptually
For each table, we can produce not a
Vec<Cast>
, but aVec<Vec<Cast>>
. Then, when producing the data from the ingestion, we produce a row for each element in the outer collection.This is somewhat wasteful because the prefix of the casts are likely to be the same. If we find that this impedes performance, we can introduce a more exotic structure here that produces data for the common prefix, and then the distinct suffixes.
PG implementation
Mechanically, this requires something akin to this.