pydiverse pydiverse.pipedag issues

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.

https://pydiversepipedag.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

12 stars 2 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

SQLAlchemy >= 2.0.0, Depedency for IBM-DB2

#206 DelongChenQC opened 1 day ago
0
Batched processing of dataframe based task: reduce memory consumption and parallelize

#205 windiana42 opened 3 days ago
3
Complex tasks with imperative materialization break caching

#204 DominikZuercherQC opened 5 days ago
0
bug(): Complex tasks using imperative materialization break caching

#203 DominikZuercherQC opened 5 days ago
1
feat(): Added ignore_position_hash option for SQLTableStore

#202 DominikZuercherQC opened 2 weeks ago
0
feat(): add truncation of primary keys and indexes

#201 DominikZuercherQC opened 2 weeks ago
0
Allow custom identifier names

#200 DominikZuercherQC opened 3 weeks ago
0
Mixed per-user and team-shared pipeline runs

#199 windiana42 opened 3 weeks ago
0
Solve connectorx failure

#198 windiana42 closed 4 weeks ago
0
fix(): add previous_hook_replace argument for register_table

#197 DominikZuercherQC closed 3 weeks ago
0
feat(): add upload_table and download_table functions to PandasTableHook

#196 DominikZuercherQC closed 1 month ago
0
Persist manually versioned intermediate state / decoupling development on two parts of the pipeline

#195 windiana42 opened 1 month ago
0
Support calling subgraph even if position hashes of inputs changed

#194 windiana42 opened 1 month ago
0
Improve error message for input not found when running subgraph

#193 windiana42 opened 1 month ago
0
Allow running tasks on active schema (Big DANGER mode disclaimer)

#192 windiana42 opened 2 months ago
0
DuckDB: Fix Table rename with primary key/index

#191 windiana42 opened 2 months ago
0
Move primary key generation for mssql backend after completely filling a table.

#190 windiana42 closed 2 months ago
0
Implement bulk load to snowflake database (the driver has a mechanism to load parquet files via S3)

#189 windiana42 opened 2 months ago
0
Implement @input_stage_version decorator which provides inputs from two versions of the same stage before schema swapping

#188 windiana42 closed 2 months ago
0
Better error message when executing subflow and input tasks not found in cache

#187 windiana42 opened 2 months ago
0
Implementation of snowflake backend.

#186 windiana42 closed 2 months ago
2
Do not require connectorx for polars dematerialization

#185 nicolasmueller closed 2 months ago
0
Support Snowflake backend

#184 windiana42 closed 2 months ago
1
Watch duckdb 0.10.1 and sqlalchemy 2.0.29 compatibility issues

#183 windiana42 opened 2 months ago
1
Add group nodes for structuring flows, as task ordering barrier, and for clusting tasks in visualization

#182 windiana42 closed 2 months ago
5
Add validation tasks to a Stage which get access to multiple schemas at once (last committed schema of this instance or another instance for this Stage)

#181 windiana42 closed 2 months ago
8
Group tasks or stages in visualization

#180 windiana42 closed 2 months ago
5
Implement imperative materialization.

#179 windiana42 closed 2 months ago
0
Improve insert_into_in_query to treat comments in TSQL select statements

#178 windiana42 opened 3 months ago
1
fix(): Column called from triggers bug

#177 DominikZuercherQC closed 3 months ago
0
add autoincrement option

#176 DominikZuercherQC opened 3 months ago
0
Provide a lot more examples for readthedocs.io documentation

#175 windiana42 closed 3 months ago
1
Provide pydiverse.pipedag.__version__

#174 windiana42 opened 3 months ago
0
Move some or all combinatorial tests for cache_validation options to nightly tests.

#173 windiana42 opened 3 months ago
0
Add identity_insert materialisation option for MSSQL tablestores

#172 DominikZuercherQC opened 3 months ago
0
Setting the isolation level for MSSQL table store to READ UNCOMMITED

#171 DominikZuercherQC closed 3 months ago
0
Propose redesign of cache validation options in configuration

#170 windiana42 closed 3 months ago
5
mssql: Fast materialization/dematerialization based on bcpandas and similar bulk load techniques

#169 windiana42 opened 3 months ago
0
Emergency Release 0.7.2: Implemented config for max_copy_operations, pool_size, pool_timeout.

#168 windiana42 closed 3 months ago
0
Retry of producing a stage output

#167 windiana42 opened 3 months ago
0
Add option to `Flow.run` to force execution of all tasks

#166 nicolasmueller closed 3 months ago
5
Disable cache validation for a specific run

#165 nicolasmueller closed 3 months ago
0
Imperative Materialization

#164 windiana42 closed 2 months ago
2
Implement inline feature in pipedag.Table() to prevent materialization to database if not needed

#163 windiana42 opened 3 months ago
2
Fix #127 regarding UNLOGGED postgres tables during copy table.

#162 windiana42 closed 3 months ago
1
Implement systematic retry logic especially for `sa.Table(autoload_with=)`

#161 windiana42 opened 3 months ago
1
Support nullable and non_nullable parameters in pipedag.Table() and refactor ddl/table-store/hook logic

#160 windiana42 closed 3 months ago
2
Support setting nullable / non-nullable constraints in pipedag.Table()

#159 windiana42 closed 3 months ago
6
Refactor DB2 dependency in generic code.

#158 windiana42 closed 3 months ago
2
Fix duplicate table bug

#157 nicolasmueller closed 3 months ago
0