CDC off materialised views possible ?

[gedw99]

Can you let me know if it’s possible to do cdc off materialised views please ?

It’s not mentioned in the docs , but maybe it’s just assumed

[avikivity]

I think not, but why not use CDC off the main table? The same information is returned.

[piodul]

@gedw99 No, it's not possible to enable CDC on a materialized view, you can only do it for tables. Enabling CDC for materialized views is explicitly forbidden by Scylla:

cqlsh> create materialized view ks.tbl_mv as select * from ks.tbl where v is not null and ck1 is not null and ck2 is not null primary key (pk, ck1, ck2) with cdc = {'enabled': true};
InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot enable CDC for a materialized view"
cqlsh> 

[gedw99]

thanks all. good to know.

@avikivity Want to have consumers subscribe to changes based on a query. I think i will have to put in a system that listens to the CDC off the tables, and then updates "consumer" tables and then CDC off those consumer tables.

[avikivity]

That sounds like a very roundabout (and slow) way. Why not just CDC the main tables? Any filtering can be done with a WHERE clause or on the client side.

[gedw99]

Thanks. I will try it out

[haaawk]

@gedw99 it is forbidden to use CDC with materialised views because updates to materialised views are asynchronous and CDC is synchronous in nature. @piodul please close this issue.

[gedw99]

@avikivity

That sounds like a very roundabout (and slow) way. Why not just CDC the main tables? Any filtering can be done with a WHERE clause or on the client side.

Yes that a good workaround i could live with, and i am leaning that way now, so that things just work. I will need to write a parser on the client side. What golang scylladb pkg to use to write the sql parser for the client side ?

[gedw99]

@gedw99 it is forbidden to use CDC with materialised views because updates to materialised views are asynchronous and CDC is synchronous in nature. @piodul please close this issue.

@haaawk Helpful. thankyou.. If there a way to get a hook on when a synchronous event occurs on the Materialized VIew, so that i can manage the consumer side cache / buffer ?

[haaawk]

Materialized Views are asynchronous. Are you asking for a hook when asynchronous MV write is done? I don't think there's anything like that but I'm not an expert in MV. @psarna do you know?

[psarna]

No such thing is available at the moment. Some materialized views are synchronous though. E.g. views which have an identical partition key definition as the base table will also be collocated on the same nodes, which will also make all the operations synchronous - the write will only be acknowledged after the data is stored for both the base table and its accompanying views. If that's by any chance your case, observing CDC of the base table is enough to know that the view updates got applied as well.

[gedw99]

No such thing is available at the moment. Some materialized views are synchronous though. E.g. views which have an identical partition key definition as the base table will also be collocated on the same nodes, which will also make all the operations synchronous - the write will only be acknowledged after the data is stored for both the base table and its accompanying views. If that's by any chance your case, observing CDC of the base table is enough to know that the view updates got applied as well.

@psarna

so then i can listen for base table event, and know the views associated with it have updated, so then this is going to work out ok i think.

Are there any plans to extend the simple examples for various use cases ?

[dkropachev]

No such thing is available at the moment. Some materialized views are synchronous though. E.g. views which have an identical partition key definition as the base table will also be collocated on the same nodes, which will also make all the operations synchronous - the write will only be acknowledged after the data is stored for both the base table and its accompanying views. If that's by any chance your case, observing CDC of the base table is enough to know that the view updates got applied as well.

@psarna

so then i can listen for base table event, and know the views associated with it have updated, so then this is going to work out ok i think.

Are there any plans to extend the simple examples for various use cases ?

Do you have any particular examples in mind ?

[dkropachev]

@gedw99 , let's do the following, I am closing this issue, since initial question was clarified, if you feel we have a gap in examples we are encourage you can create another issue on that. Thanks for your contribution.