search

Qbeast-io / qbeast-spark

Qbeast-spark: DataSource enabling multi-dimensional indexing and efficient data sampling. Big Data, free from the unnecessary!
https://qbeast.io/qbeast-our-tech/
Apache License 2.0
215 stars 19 forks source link

Unify Create Qbeast Table in one step #371

Open osopardo1 opened 4 months ago

osopardo1 commented 4 months ago

Right now, the code to create a Qbeast Table without data (no SAVE AS), is done in two steps:

  1. Create an empty data frame and save it in the Location.
  2. Convert the Delta Table to the Qbeast Table with ConvertToQbeastCommand.

It is executed like that because we do not have access to methods in the CreateDeltaTableCommand, nor can we delegate the creation to that object. Otherwise (if we call directly CreateDeltaTableCommand), the table would be created in the Catalog as "delta", and the whole operation would fail.

The idea is to do both in the same transaction. Perhaps it is a good moment to build a CreateQbeastTableCommand to centralize the process.

fpj commented 1 month ago

We need a design document for this issue to understand clearly the implications of the change.