dimitri / pgloader

Migrate to PostgreSQL in a single command!
http://pgloader.io
Other
5.45k stars 548 forks source link

Cannot import SQLite array columns for various types into PostgreSQL (`TEXT[]`, `NUMERIC[]`, `BYTE[]`) #1552

Open lgo opened 10 months ago

lgo commented 10 months ago

It seems like the pgloader grammer simply cannot handle <TYPE>[], leading to inability to parse these files at all. I was just trying to import a misc app into Postgres, so I'm not exactly sure what the type is (JSON? Something else?) but the application bootstrapped the Postgres tables with columns that had the exact same types so PostgreSQL clearly supports the same thing.

Specifically, the SQLite's column types were text[] defaut '{}'. The postgres bootstrapped tables appear to be text[] NULL [{}] (I didn't check if the former syntax is compatible).


# Generating the SQL file
sqlite3 /tmp/test.db \
  "CREATE TABLE foo(id TEXT PRIMARY KEY, broken TEXT[] DEFAULT '{}');" \
  "INSERT INTO foo VALUES ('bar', '{}');" \
  "CREATE TABLE foo2(id TEXT PRIMARY KEY, broken NUMERIC[] DEFAULT '{}');" \
  "INSERT INTO foo2 VALUES ('bar', '{}');" 
# pgloader command (plain ol' import)
pgloader /tmp/test.db postgresql://localhost/testdb
2023-12-29T17:52:56.016000Z LOG pgloader version "3.6.7~devel"
2023-12-29T17:52:56.120000Z LOG Migrating from #<SQLITE-CONNECTION sqlite:///tmp/test.db {1005D913E3}>
2023-12-29T17:52:56.120000Z LOG Migrating into #<PGSQL-CONNECTION pgsql://localhost/testdb {1005F57353}>
2023-12-29T17:52:56.156000Z ERROR sqlite: At

  text[]
      ^ (Line 1, Column 4, Position 4)

In context SQLITE-TYPE-NAME:

While parsing SQLITE-TYPE-NAME. Problem:

  The production

    #\]

  does not satisfy the predicate ALPHA-CHAR-P.

Expected:

     the character _ (LOW_LINE)
  or any character satisfying ALPHA-CHAR-P
  or <end of input>
2023-12-29T17:52:56.156000Z LOG report summary reset
       table name     errors       rows      bytes      total time
-----------------  ---------  ---------  ---------  --------------
            fetch          0          0                     0.000s
  fetch meta data          0          0                     0.000s
-----------------  ---------  ---------  ---------  --------------
-----------------  ---------  ---------  ---------  --------------