search

electric-sql / electric

Sync little subsets of your Postgres data into local apps and services.
https://electric-sql.com
Apache License 2.0
6.22k stars 147 forks source link

Infinite loop of error log on ALTER TABLE migration serialization #909

Closed JosselinTILLAY closed 2 months ago

JosselinTILLAY commented 8 months ago

I have a POC project for testing electric SQL and I want to check the comportement of migrations.

Everything work as expected for new table creation and electrification.

But when I want to add a column to an existing table, it doesn't work. The migration on server seems to work as intendeed (I use django and the new column is accessible), but the device enter an infinite loop of errors.

The loop of logs is infinite, so I don't really know where to stop and where to begin to copy / paste it, so here is my try, I can generate more log if needed...

Service log :

poc-electricsql-electric-1  | 15:27:16.937 pid=<0.4294.0> client_id=4ba8fb8e-d89f-4198-9f0a-17875452d7d8 instance_id=4d03eed2-392d-4b7a-b603-17923b7575fe user_id=1234567890 [debug] msg from producer: [{%Electric.Replication.Changes.Transaction{xid: 797, changes: [], commit_timestamp: ~U[2024-02-01 15:09:37.989195Z], origin: "postgres_1", publication: "electric_publication", lsn: #Lsn<0/19D1B80>, ack_fn: nil, origin_type: :postgresql}, 27073408}, {%Electric.Replication.Changes.Transaction{xid: 798, changes: [%Electric.Replication.Changes.NewRecord{relation: {"electric", "ddl_commands"}, record: %{"id" => "5", "query" => "ALTER TABLE \"synchro_neighbour\" ADD COLUMN \"sublabel\" text NULL", "txid" => "798", "txts" => "1706801226376675"}, tags: []}], commit_timestamp: ~U[2024-02-01 15:27:06.458476Z], origin: "postgres_1", publication: "electric_publication", lsn: #Lsn<0/1A16138>, ack_fn: nil, origin_type: :postgresql}, 27353400}, {%Electric.Replication.Changes.Transaction{xid: 799, changes: [], commit_timestamp: ~U[2024-02-01 15:27:06.493983Z], origin: "postgres_1", publication: "electric_publication", lsn: #Lsn<0/1A18708>, ack_fn: nil, origin_type: :postgresql}, 27363080}]
poc-electricsql-electric-1  | 15:27:16.938 pid=<0.4294.0> client_id=4ba8fb8e-d89f-4198-9f0a-17875452d7d8 instance_id=4d03eed2-392d-4b7a-b603-17923b7575fe user_id=1234567890 [debug] Filtering transaction %Electric.Replication.Changes.Transaction{xid: 797, changes: [], commit_timestamp: ~U[2024-02-01 15:09:37.989195Z], origin: "postgres_1", publication: "electric_publication", lsn: #Lsn<0/19D1B80>, ack_fn: nil, origin_type: :postgresql} for user 1234567890
poc-electricsql-electric-1  | 15:27:16.938 pid=<0.4294.0> client_id=4ba8fb8e-d89f-4198-9f0a-17875452d7d8 instance_id=4d03eed2-392d-4b7a-b603-17923b7575fe user_id=1234567890 [debug] trans: %Electric.Replication.Changes.Transaction{xid: 798, changes: [%Electric.Replication.Changes.NewRecord{relation: {"electric", "ddl_commands"}, record: %{"id" => "5", "query" => "ALTER TABLE \"synchro_neighbour\" ADD COLUMN \"sublabel\" text NULL", "txid" => "798", "txts" => "1706801226376675"}, tags: []}], commit_timestamp: ~U[2024-02-01 15:27:06.458476Z], origin: "postgres_1", publication: "electric_publication", lsn: #Lsn<0/1A16138>, ack_fn: nil, origin_type: :postgresql} with offset 27353400
poc-electricsql-electric-1  | 15:27:16.938 pid=<0.4294.0> client_id=4ba8fb8e-d89f-4198-9f0a-17875452d7d8 instance_id=4d03eed2-392d-4b7a-b603-17923b7575fe user_id=1234567890 [info] Serializing migration "20240201152706_376": "ALTER TABLE \"synchro_neighbour\" ADD COLUMN \"sublabel\" text NULL"
poc-electricsql-electric-1  | 15:27:16.942 pid=<0.4294.0> client_id=4ba8fb8e-d89f-4198-9f0a-17875452d7d8 instance_id=4d03eed2-392d-4b7a-b603-17923b7575fe user_id=1234567890 [error] GenServer #PID<0.4294.0> terminating
poc-electricsql-electric-1  | ** (FunctionClauseError) no function clause matching in Electric.Postgres.Dialect.SQLite.map_constraint/2
poc-electricsql-electric-1  |     (electric 0.9.1) lib/electric/postgres/dialect/sqlite.ex:165: Electric.Postgres.Dialect.SQLite.map_constraint(nil, :add_column)
poc-electricsql-electric-1  |     (elixir 1.15.4) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
poc-electricsql-electric-1  |     (electric 0.9.1) lib/electric/postgres/dialect/sqlite.ex:137: Electric.Postgres.Dialect.SQLite.alter_table_cmd/3
poc-electricsql-electric-1  |     (elixir 1.15.4) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
poc-electricsql-electric-1  |     (electric 0.9.1) lib/electric/postgres/dialect/sqlite.ex:90: Electric.Postgres.Dialect.SQLite.to_sql/2
poc-electricsql-electric-1  |     (electric 0.9.1) lib/electric/postgres/replication.ex:136: anonymous fn/2 in Electric.Postgres.Replication.build_replication_msg/3
poc-electricsql-electric-1  |     (elixir 1.15.4) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
poc-electricsql-electric-1  |     (electric 0.9.1) lib/electric/postgres/replication.ex:132: Electric.Postgres.Replication.build_replication_msg/3

Client log :

 LOG  [rpc] send: #SatInStartReplicationReq{lsn: MjcwNTI3MzY=, subscriptions: [ddbee208-0d64-41f9-a814-4f8b11fed981]}
 LOG  [proto] send: #SatRpcRequest{method: startReplication, requestId: 173}
 LOG  [proto] recv: #SatRpcResponse{method: startReplication, requestId: 173}
 LOG  [rpc] recv: #SatInStartReplicationResp{}
 LOG  [proto] recv: #SatRpcRequest{method: startReplication, requestId: 0}
 INFO  Server sent a replication request to start from 0, and options []
 LOG  [proto] send: #SatRpcResponse{method: startReplication, requestId: 0}
 LOG  Connectivity state changed: connected
 WARN  an error occurred in satellite: socket closed
 LOG  Connectivity state changed: disconnected
 WARN  Client disconnected with a non fatal error, reconnecting

My migration is generated from Django, so I don't really know what the SQL is, but the logs seems to point the failure on

ALTER TABLE "synchro_neighbour" ADD COLUMN "sublabel" text NULL
linear[bot] commented 8 months ago

VAX-1610 Infinite loop of error log on ALTER TABLE migration serialization

JosselinTILLAY commented 7 months ago

I have a investigated. The migration serialization work with :

ALTER TABLE "synchro_neighbour" ADD COLUMN "sublabel" text

but not with

ALTER TABLE "synchro_neighbour" ADD COLUMN "sublabel" text NULL
KyleAMathews commented 2 months ago

👋 we've been working the last month on a rebuild of the Electric server over at a temporary repo https://github.com/electric-sql/electric-next/

You can read more about why we made the decision at https://next.electric-sql.com/about

We're really excited about all the new possibilities the new server brings and we hope you'll check it out soon and give us your feedback.

We're now moving the temporary repo back here. As part of that migration we're closing all the old issues and PRs. We really appreciate you taking the time to investigate and report this issue!