sync command error: panic: Unsupported PostgreSQL type: geometry

[ngnnah]

When run this bemidb --pg-database-url postgresql://..... sync I GOT this error and the sync failed and stopped

2024/11/12 11:42:27 [INFO] Syncing public.table_with_geometry_column...
panic: Unsupported PostgreSQL type: geometry

goroutine 1 [running]:
main.(*PgSchemaColumn).toParquetSchemaField(0x140036a5400?)
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/pg_schema_column.go:251 +0x6f4
main.PgSchemaColumn.ToStringParquetSchemaField({{0x140033244e8, 0x8}, {0x140033244f0, 0x8}, {0x140033244f8, 0x3}, {0x104b3a0bf, 0x1}, {0x104b3a0bb, 0x1}, ...})
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/pg_schema_column.go:49 +0x68
main.(*StorageBase).WriteParquetFile(0x1400007c558, {0x105332018, 0x14000bf03c0}, {0x14000f9e908, 0x5, 0x0?}, 0x14000bf0390)
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/storage_base.go:29 +0x194
main.(*StorageLocal).CreateParquet(0x140003fa0f0, {0x1400004cd20, 0x69}, {0x14000f9e908, 0x5, 0x8}, 0x14000bf0390)
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/storage_local.go:92 +0x210
main.(*IcebergWriter).Write(0x1400000e558, {{0x140003dc018?, 0x140003dc018?}, {0x14000042360?, 0x14000042360?}}, {0x14000f9e908, 0x5, 0x8}, 0x14000bf0390)
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/iceberg_writer.go:338 +0xc8
main.(*Syncer).syncFromPgTable(0x1400000e588, 0x1400040b200, {{0x140003dc018?, 0x6?}, {0x14000042360?, 0x0?}})
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/syncer.go:108 +0x304
main.(*Syncer).SyncFromPostgres(0x1400000e588)
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/syncer.go:46 +0x1a8
main.syncFromPg(0x1059bf940)
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/main.go:60 +0x24
main.main()
        /Users/exaspark/Documents/Code/BemiHQ/BemiDB/src/main.go:25 +0xb0

[exAspArk]

Hey @ngnnah — thanks for submitting this issue. BemiDB current doesn't work with geometric data types, but we'll be happy to add support for them.

Would you be able to share information about your schema? For example, by querying it like:

SELECT * FROM information_schema.columns WHERE table_name = 'table_with_geometry_column';

[ngnnah]

this is standard postgis geometry column type, here's the schema: columns_202411121156.csv columns_202411121156.csv The sync command doesn't need to support geometry/user-defined type right away but instead of failing and breaking, i think it would help to just CAST all unsupported dtypes to string, either via ST_AsText (for geometry type) or simple CAST()

[exAspArk]

Great, thank you for sharing this.

Would you be able to install the latest version and try again?

User-defined types should now be automatically converted to strings.

[EricSoroos]

Similarly:

./bemidb --pg-database-url $DB_URL --iceberg-path /data/.iceberg sync
2024/11/12 17:09:32 [INFO] Syncing "public"."00c4a58f-9cf1-407e-bf7a-4b67bee5b081"...
panic: Unsupported PostgreSQL type: tsvector

Tsvectors should probably be ignored.

[exAspArk]

Thanks, @EricSoroos. "tsvector" should also be fixed now. Let us know if there are any other similar issues

[EricSoroos]

Thanks. That was fast!