issues
search
pydiverse
/
pydiverse.pipedag
A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15
stars
2
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Automatic extraction of pipeline subgraph for PROD execution
#217
windiana42
opened
1 week ago
0
feat(): Task links
#216
DominikZuercherQC
opened
1 week ago
2
Fix pytest assert enrichment problem when running pytest with `--workers` option
#215
windiana42
opened
2 weeks ago
0
fix: Only warn about ignore_position_hashes if it is True
#214
nicolasmueller
closed
3 weeks ago
0
Test in CI for macOS and Windows as well
#213
pavelzw
opened
3 weeks ago
1
Fix polars >= 1
#212
nicolasmueller
closed
3 weeks ago
0
Implemented ExternalTableReference at flow wiring time.
#211
windiana42
closed
3 weeks ago
0
Fix primary_key generation when writing pandas dataframe to postgres database
#210
windiana42
closed
1 month ago
0
Make `MAX_LENGTH_PK` and `MAX_LENGTH_INDEX` configurable
#209
DominikZuercherQC
opened
1 month ago
0
Support more table stores (i.e. for parquet files or apache arrow dataframes in-memory)
#208
adzcai
opened
1 month ago
1
Switch from poetry (pypi) to pixi (conda-forge)
#207
windiana42
opened
1 month ago
2
SQLAlchemy >= 2.0.0, Depedency for IBM-DB2
#206
DelongChenQC
opened
1 month ago
0
Batched processing of dataframe based task: reduce memory consumption and parallelize
#205
windiana42
opened
1 month ago
3
Complex tasks with imperative materialization break caching
#204
DominikZuercherQC
closed
1 month ago
0
bug(): Complex tasks using imperative materialization break caching + release `v 0.9.4`
#203
DominikZuercherQC
closed
1 month ago
2
feat(): Added ignore_position_hash option for SQLTableStore
#202
DominikZuercherQC
closed
1 month ago
2
feat(): add truncation of primary keys and indexes
#201
DominikZuercherQC
closed
1 month ago
0
Allow custom identifier names
#200
DominikZuercherQC
closed
1 month ago
0
Mixed per-user and team-shared pipeline runs
#199
windiana42
opened
2 months ago
0
Solve connectorx failure
#198
windiana42
closed
2 months ago
0
fix(): add previous_hook_replace argument for register_table
#197
DominikZuercherQC
closed
2 months ago
0
feat(): add upload_table and download_table functions to PandasTableHook
#196
DominikZuercherQC
closed
2 months ago
0
Persist manually versioned intermediate state / decoupling development on two parts of the pipeline
#195
windiana42
opened
3 months ago
0
Support calling subgraph even if position hashes of inputs changed
#194
windiana42
closed
1 month ago
0
Improve error message for input not found when running subgraph
#193
windiana42
opened
3 months ago
0
Allow running tasks on active schema (Big DANGER mode disclaimer)
#192
windiana42
opened
3 months ago
0
DuckDB: Fix Table rename with primary key/index
#191
windiana42
opened
3 months ago
0
Move primary key generation for mssql backend after completely filling a table.
#190
windiana42
closed
4 months ago
0
Implement bulk load to snowflake database (the driver has a mechanism to load parquet files via S3)
#189
windiana42
opened
4 months ago
0
Implement @input_stage_version decorator which provides inputs from two versions of the same stage before schema swapping
#188
windiana42
closed
3 months ago
0
Better error message when executing subflow and input tasks not found in cache
#187
windiana42
opened
4 months ago
0
Implementation of snowflake backend.
#186
windiana42
closed
4 months ago
2
Do not require connectorx for polars dematerialization
#185
nicolasmueller
closed
4 months ago
0
Support Snowflake backend
#184
windiana42
closed
4 months ago
1
Watch duckdb 0.10.1 and sqlalchemy 2.0.29 compatibility issues
#183
windiana42
opened
4 months ago
1
Add group nodes for structuring flows, as task ordering barrier, and for clusting tasks in visualization
#182
windiana42
closed
4 months ago
5
Add validation tasks to a Stage which get access to multiple schemas at once (last committed schema of this instance or another instance for this Stage)
#181
windiana42
closed
3 months ago
8
Group tasks or stages in visualization
#180
windiana42
closed
4 months ago
5
Implement imperative materialization.
#179
windiana42
closed
4 months ago
0
Improve insert_into_in_query to treat comments in TSQL select statements
#178
windiana42
opened
4 months ago
1
fix(): Column called from triggers bug
#177
DominikZuercherQC
closed
4 months ago
0
add autoincrement option
#176
DominikZuercherQC
opened
5 months ago
0
Provide a lot more examples for readthedocs.io documentation
#175
windiana42
closed
5 months ago
1
Provide pydiverse.pipedag.__version__
#174
windiana42
opened
5 months ago
0
Move some or all combinatorial tests for cache_validation options to nightly tests.
#173
windiana42
opened
5 months ago
0
Add identity_insert materialisation option for MSSQL tablestores
#172
DominikZuercherQC
opened
5 months ago
0
Setting the isolation level for MSSQL table store to READ UNCOMMITED
#171
DominikZuercherQC
closed
5 months ago
0
Propose redesign of cache validation options in configuration
#170
windiana42
closed
5 months ago
5
mssql: Fast materialization/dematerialization based on bcpandas and similar bulk load techniques
#169
windiana42
opened
5 months ago
0
Emergency Release 0.7.2: Implemented config for max_copy_operations, pool_size, pool_timeout.
#168
windiana42
closed
5 months ago
0
Next