search

datahq / dataflows

DataFlows is a simple, intuitive lightweight framework for building data processing flows in python.
https://dataflows.org
MIT License
193 stars 39 forks source link

Updating SQL tables with indexes in dump_to_sql #170

Open pwalsh opened 2 years ago

pwalsh commented 2 years ago

Relates to https://github.com/datahq/dataflows/pull/169

With support for index fields, the question arises regarding updating existing tables. There is no migration process now, so if one added indexes and wanted to apply them to an existing table when updating data, there is no possibility via the sql dumper.

Update runs could probably use Alembic and specifically https://alembic.sqlalchemy.org/en/latest/autogenerate.html to handle this.

akariv commented 2 years ago

Yes, using alembic would be pretty cool - perhaps as an improvement to the current tableschema-sql inner workings.