pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.

https://pydiversepipedag.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

15 stars 2 forks source link

Enable the creation of external TableReferences #139

Closed nicolasmueller closed 5 months ago

nicolasmueller commented 6 months ago

Enable the creation of external TableReferences that reference tables in external schemas. This is useful to tell pipedag about tables that already exist in the database and should not be copied. It avoid manually reflecting them, which can sometimes be cumbersome as one might want to call a Task with a pipedag Table and sometimes with an external table.

Checklist

[x] Added a docs/source/changelog.md entry

NicolasMuellerQC commented 6 months ago

Enabling external views for DuckDB depends on https://github.com/duckdb/duckdb/issues/10322

windiana42 commented 6 months ago

Enabling external views for DuckDB depends on https://github.com/duckdb/duckdb/issues/10322

We could implement a workaround since this problem is quite common among some SQLAlchemy backends. We have such a workaround in other places of the codebase.

nicolasmueller commented 6 months ago

We could implement a workaround since this problem is quite common among some SQLAlchemy backends. We have such a workaround in other places of the codebase.

Done.

NicolasMuellerQC commented 6 months ago

@windiana42 Let me know if it looks good now 😃

windiana42 commented 5 months ago

@nicolasmueller good catch! did you find something else?

NicolasMuellerQC commented 5 months ago

@nicolasmueller good catch! did you find something else?

@windiana42 No, I am fine with merging then 👍