alter-table-after-source.td:313:1: error: expected error containing "table was dropped", got "Source error: u155: incompatible schema change: column f1 no longer present in table alter_table_rename"

[def-]

What version of Materialize are you using?

937b9e3b0ca1

What is the issue?

Seen in https://buildkite.com/materialize/test/builds/80985#018ef6a4-f623-42bb-a362-49dc4fdbc421 with MySQL CDC

Docker compose failed: docker compose -f/dev/fd/3 --project-directory /var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-0096197445f7a703d-1/materialize/test/test/mysql-cdc run testdrive --var=ssl-ca=<CERTIFICATE>  --var=ssl-client-cert=<CERTIFICATE>  --var=ssl-client-key=<RSA PRIVATE KEY>  --var=ssl-wrong-ca=<CERTIFICATE>  --var=ssl-wrong-client-cert=<CERTIFICATE>  --var=ssl-wrong-client-key=<PRIVATE KEY>  --var=mysql-root-password=p@ssw0rd --var=mysql-user-password=us3rp4ssw0rd --var=default-replica-size=4-4 --var=default-storage-size=4-1 *.td
^^^ +++
alter-table-after-source.td:313:1: error: expected error containing "table was dropped", got "Source error: u155: incompatible schema change: column f1 no longer present in table alter_table_rename"
     |
  13 | $ postgres-execute c ... [rest of line truncated for security]
  20 | > CREATE SECRET mysq ... [rest of line truncated for security]
  24 |     PASSWORD SECRET  ... [rest of line truncated for security]
  27 | $ mysql-connect name ... [rest of line truncated for security]
 312 | 
 313 | ! SELECT * FROM alter_table_rename;
     | ^
+++ !!! Error Report
1 errors were encountered during execution
files involved: alter-table-after-source.td

ci-regexp: alter-table-after-source.td:

[rjobanp]

Haven't seen this re-occur in a few months - am inclined to close this. Thoughts?

[def-]

Sure, let's close and reopen if it happens again.

[def-]

We have a similar issue in CI being flaky again: https://buildkite.com/materialize/test/builds/84628#019052ac-d24f-427e-914e-5909d8cd5680

>> ALTER TABLE alter_decimal MODIFY COLUMN f1 DECIMAL(6,1);
>> INSERT INTO alter_decimal VALUES (12345.6);
> SELECT * FROM alter_decimal;
query error didn't match; sleeping to see if dataflow produces error shortly 50ms 75ms 113ms 169ms 253ms 380ms 570ms 854ms 1s 2s 3s 4s 6s 10s 15s 16s^^^ +++
alter-table-after-source.td:297:1: error: query succeeded, but expected error containing "altered"
     |
  18 | > CREATE SECRET mysq ... [rest of line truncated for security]
  22 |     PASSWORD SECRET  ... [rest of line truncated for security]
  25 | $ mysql-connect name ... [rest of line truncated for security]
 296 | 
 297 | ! SELECT * FROM alter_decimal;
     | ^
--- alter-table-irrelevant.td

[def-]

And another one, reported by Frank: https://materializeinc.slack.com/archives/C01LKF361MZ/p1720812756750439

[rjobanp]

Still unclear why this is flaky but it is worth noting that this test file alter-table-after-source.td has these overrides which are not in many of the other tests

$ set-sql-timeout duration=1s
$ set-max-tries max-tries=20

It's possible that we are just not giving enough time for these errors to be detected and propagated

[def-]

Let's see if that helps! https://github.com/MaterializeInc/materialize/pull/28247

[def-]

Tried that, didn't help, still seeing the same error.

[def-]

This is still occurring, I'll disable the test for now: https://buildkite.com/materialize/test/builds/86989#0190fd81-49e2-4588-b657-6e9f69d80677