aljazerzen / connector_arrow

Apache Arrow database client for many databases.
https://docs.rs/connector_arrow
MIT License
37 stars 2 forks source link

can PostgreSQL and SQLite store `LargeUtf8`? #12

Open aljazerzen opened 6 months ago

aljazerzen commented 6 months ago

In other words, do they support storing text/blob that has length larger than 2^32 (4GiB)?

If yes, then the type of TEXT should always be LargeUtf8 (and Uft8 is coerced into LargeUft8). If no, then the type of TEXT should always be Utf8 (and coercion is reversed). Similar for binary types.

DuckDB kindly answers this question by returning a schema that contain Utf8 when you declare a column as VARCHAR or TEXT.

This means that it might not be possible to store an Arrow in a database. We need an error for that. And we need to indicate that in coerce_type.