cloud canary test: canary-sources.td:43:1: error: non-matching rows with FORMAT BYTES

[def-]

What version of Materialize are you using?

4460d47dab

What is the issue?

Started with 4460d47dab, was still good in b24ade2e0c Potential changes: https://github.com/MaterializeInc/materialize/compare/b24ade2e0c...4460d47dab

https://buildkite.com/materialize/nightlies/builds/4173#018ab6ad-8f0a-47fd-9b35-b0562bf0ec39

> CREATE SOURCE bytes FROM KAFKA CONNECTION kafka_conn (TOPIC 'testdrive-bytes-1') FORMAT BYTES ENVELOPE NONE WITH (SIZE '3xsmall');
rows match; continuing at ts 1695282233.0195313
> CREATE MATERIALIZED VIEW bytes_view AS SELECT COUNT(*) AS cnt FROM bytes;
rows match; continuing at ts 1695282233.5970292
> CREATE DEFAULT INDEX ON bytes_view;
rows match; continuing at ts 1695282234.1382542
> SELECT cnt > 0 from bytes_view
rows didn't match; sleeping to see if dataflow catches up 50ms 75ms 113ms 169ms 253ms 380ms 570ms 854ms 1s
--- canary-tables.td
^^^ +++
canary-sources.td:43:1: error: non-matching rows: expected:
[["true"]]
got:
[["false"]]
Poor diff:
+ false
- true

@petrosagg https://github.com/MaterializeInc/materialize/pull/21732 sounds suspicious to me, so I'll assign this to you for now. I will try to figure out if the change is actually responsible or which one is today.

This is included in the v0.70.0 SHA (f064bf51bc9df536222a558c064c378551f6967d), so I am considering this a release blocker until someone from storage team can claim this is a false alarm and not that bad.

[def-]

Confirmed it's #21732 with https://buildkite.com/materialize/nightlies/builds/4224 (before) and https://buildkite.com/materialize/nightlies/builds/4225 (after). I tried reproducing it locally with a td file but failed:

$ kafka-create-topic topic=bytes

$ kafka-ingest format=bytes key-terminator=: key-format=bytes topic=bytes repeat=100
abc:abc

> CREATE CONNECTION kafka_conn TO KAFKA (
  BROKER '${testdrive.kafka-addr}'
  );

> CREATE SOURCE bytes
  FROM KAFKA CONNECTION kafka_conn (TOPIC 'testdrive-bytes-${testdrive.seed}')
  FORMAT BYTES
  ENVELOPE NONE
  WITH (SIZE '1');

> CREATE MATERIALIZED VIEW bytes_view AS SELECT COUNT(*) AS cnt FROM bytes;

> CREATE DEFAULT INDEX ON bytes_view;

> SELECT cnt > 0 from bytes_view
true

So my best guess is that we have different flags on canary than we set in CI and this has an effect. I'll try to check LaunchDarkly. Edit: I tried most staging flags, but still the same.

[def-]

Duplicate of https://github.com/MaterializeInc/materialize/issues/21860