cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.19k stars 3.82k forks source link

changefeedccl: core changefeed fails in high data volume throughput test #134425

Open asg0451 opened 3 weeks ago

asg0451 commented 3 weeks ago

I observed and reproduced a core changefeed fail with a batch timestamp error while testing something unrelated.

Reproduction:

Given ./cockroach start-single-node --insecure running.

In 2 terminals, A and B:

A (in ./cockroach sql --insecure):

create table stuff2 (a int primary key);

B:

./cockroach sql --insecure -e "EXPERIMENTAL CHANGEFEED FOR TABLE stuff2 WITH OPTIONS (updated, resolved)" | grep -v stuff2 # start a changefeed but discard the data so it doesnt spam your terminal

A:

insert into stuff2 (a)  select i from generate_series(0, 10000) as g(i);
insert into stuff2 (a)  select i from generate_series(10001, 50000) as g(i);
insert into stuff2 (a)  select i from generate_series(50001, 100000) as g(i);
insert into stuff2 (a)  select i from generate_series(100001, 500000) as g(i);
insert into stuff2 (a)  select i from generate_series(500001, 1000000) as g(i);
insert into stuff2 (a)  select i from generate_series(1000001, 5000000) as g(i);
insert into stuff2 (a)  select i from generate_series(5000001, 10000000) as g(i);

..And then wait.. after a minute or so, the core changefeed in B will fail with:

(error encountered after some results were delivered)
ERROR: batch timestamp 1730910286.954309999,0 must be after replica GC threshold 1730910298.890121000,0 (r140: /Table/105/4/{3384343-5848963})

(table 105 here is stuff2; you can check that with SHOW RANGES FROM DATABASE defaultdb WITH TABLES)

blathers-crl[bot] commented 3 weeks ago

Hi @asg0451, please add branch-* labels to identify which branch(es) this C-bug affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] commented 3 weeks ago

cc @cockroachdb/cdc