search

crate / sqlalchemy-cratedb

SQLAlchemy dialect for CrateDB.
https://cratedb.com/docs/sqlalchemy-cratedb/
Apache License 2.0
3 stars 2 forks source link

Add utility support wrapping CrateDB's special table options #143

Open amotl opened 2 months ago

amotl commented 2 months ago

About

@wierdvanderhaar suggested to improve the ctk load table interface such that the target table can be created with CrateDB-specific features like partitioned table, and friends. Thanks.

This patch adds relevant support code, mostly for providing it to downstream libraries and applications. The first candidate is CrateDB Toolkit's I/O subsystem.

References

See also

seut commented 2 months ago

@amotl Honestly, I do not like this approach as it would force use to maintain table options not only at CrateDB but also at this dialect. I'd really prefer to not add concrete options here. Why not just double quote CrateDB specific (all settings prefixed with crate_ when they get converted into the create table stmt?

amotl commented 2 months ago

Response

Why not just double quote CrateDB specific (all settings prefixed with crate_ when they get converted into the create table stmt?

This is probably about that issue?

This patch's aim is different, and beyond the UX improvements we would like to tackle there.

Idea

It is to have a map for shorthand versions of special table options to be conveyed via URL query parameters, like this:

export CRATEDB_SQLALCHEMY_URL='crate://crate@localhost:4200/testdrive/demo?if-exists=replace&partitioned-by=time&clustered-by="A"&replicas=0-2&shards=2&durability=async&max-columns=42&column-policy=dynamic'

The patch is exploring this area to have a proper metadata knowledge data structure in Python, to be used for different purposes. Currently, it is targeted for the ctk load table interface of CrateDB Toolkit.

It would force use to maintain table options not only at CrateDB but also at this dialect.

I hear you. About the maintenance details, it does not necessarily have to be the source of truth here. In a future iteration, both the documentation and API wrappers would be able to derive the information from CrateDB itself, either its sources, or by inquiring at runtime?