search

vendekagon-labs / unify

An engine for automating data integration & harmonization via schema inference.
Apache License 2.0
8 stars 0 forks source link

unify transact task should not require schema-directory argument #21

Closed benkamphaus closed 10 months ago

benkamphaus commented 10 months ago

Currently this looks like it's an unintended side effect of using the same machinery as prepare for cross-checking database vs schema versions. Instead, some schema metadata should be written to the import working directory (if it's not already there) and parsed by transact at import time to ensure schema compatibility.

benkamphaus commented 10 months ago

The issue here is a little less trivial. Ensure-db was wired into the old logic, which expected that pret (now the Unify CLI) would have a full schema directory along with it in resources, so that if the schema was compatible, it could update it during the ensure-db step. To enable this behavior, though, we need a snapshot of the schema directory at the time prepare was created. It probably makes the most sense to just copy the schema directory contents into the working directory when the prepare task executes.