issues
search
pydiverse
/
pydiverse.pipedag
A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15
stars
2
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Add a function to ad-hoc materialize a table for interactive debugging
#117
nicolasmueller
closed
9 months ago
0
Simplify debugging of materialization write-back
#116
windiana42
opened
10 months ago
1
Create "virtual" stacktrace which combines wiring time and task execution time
#115
windiana42
opened
11 months ago
2
pipedag-manage delete-schemas: support clearing unused schemas
#114
windiana42
opened
11 months ago
0
for DB2 database: allow configuring table creation properties globally and/or per schema or table
#113
windiana42
closed
8 months ago
1
for DB2 database: RUNSTAT for every table after it was filled and indexes created
#112
windiana42
closed
8 months ago
2
Bump actions/checkout from 3 to 4
#111
dependabot[bot]
closed
8 months ago
1
Apply database default case to table names during materialization
#110
windiana42
opened
11 months ago
1
Disable warning that parquet table cache cannot dematerialize sa.Table
#109
windiana42
opened
11 months ago
1
Improve error message when polars frame is returned but connectorx is not installed
#108
windiana42
closed
1 year ago
0
Making `ignore_fresh_input` more explicit (or the docs)
#107
MarcAntoineSchmidtQC
closed
1 year ago
1
version 0.6.2 writes pandas tables with lowercase names to DB2
#106
windiana42
closed
1 year ago
2
pipedag-manage command to list all instances
#105
windiana42
opened
1 year ago
2
DB2 does not allow order_by CLOB column
#104
windiana42
closed
1 year ago
0
Implement Stage.tbls
#103
windiana42
opened
1 year ago
2
Optimize error messages for SQL errors
#102
windiana42
opened
1 year ago
0
fix prefect 2 engine
#101
windiana42
closed
1 year ago
1
polars dataframe needs write_parquet not to_parquet
#100
windiana42
closed
1 year ago
2
support empty stage
#99
windiana42
closed
1 year ago
0
make default string length in DB2 dataframe materialization configurable
#98
windiana42
opened
1 year ago
0
fix type annotation errors in clear_metadata.py
#97
windiana42
closed
1 year ago
0
sqlalchemy does not have a type called Double
#96
QuantCo-mr-T
closed
1 year ago
1
DatabaseLockManager requires schema creation permission
#95
QuantCo-mr-T
closed
1 year ago
0
Get rid of `ibm_db_sa_fix_name` hack
#94
NMAC427
closed
1 year ago
0
Refactor TableHooks for SQLTableStore + Support for pyarrow backed pandas DataFrames
#93
NMAC427
closed
1 year ago
0
Reflect on global communication with tasks in multi-node setup
#92
windiana42
opened
1 year ago
1
Optimize Log initialization for parallel execution and prefect integration
#91
windiana42
opened
1 year ago
1
Add support for DuckDB
#90
NMAC427
closed
1 year ago
0
Allow execution of a Subflow
#89
NMAC427
closed
1 year ago
0
Feat: Implement DatabaseLockManager
#88
NMAC427
closed
1 year ago
0
Create helm chart to make it easier to setup Linux cloud VM for testing pipedag with docker
#87
windiana42
opened
1 year ago
1
Fix tidypolars test.
#86
windiana42
closed
1 year ago
0
Log exceptions produced by tasks, even if `fail_fast=False`
#85
NMAC427
closed
1 year ago
0
Refactor deferred table store operations and properly handle renaming of tables
#84
NMAC427
closed
1 year ago
0
Encrypt and authenticate IPC Server + Client communication
#83
NMAC427
closed
1 year ago
0
Update public interface
#82
NMAC427
closed
1 year ago
0
Restructure Dependencies (#80)
#81
NMAC427
closed
1 year ago
0
Reorganize dependencies
#80
windiana42
closed
1 year ago
2
consider creating a utils library in pydiverse
#79
windiana42
opened
1 year ago
2
publish documentation per release version
#78
windiana42
closed
1 year ago
2
Implement local table cache that stores inputs/outputs of dataframe tasks in parquet
#77
windiana42
closed
1 year ago
1
Forward dataframes between tasks in parallel to SQL writeback
#76
windiana42
opened
1 year ago
1
Support <n> cache slots
#75
windiana42
opened
1 year ago
1
Store categorical variables with automatic shadow table in relational DB
#74
windiana42
opened
1 year ago
1
Local file caching for dataframe/eager tasks
#73
windiana42
closed
1 year ago
7
implemented ignore_task_version option
#72
windiana42
closed
1 year ago
0
Add option that disables explicit version checking for some instances
#71
windiana42
closed
1 year ago
0
fix ibm_db_sa bug when copying dataframes from cache: uppercase table names by default
#70
windiana42
closed
1 year ago
2
Pandas <-> SQL type mapping and chunksize=100k
#69
windiana42
closed
1 year ago
1
Add polars and tidypolars support for pipedag
#68
windiana42
closed
1 year ago
1
Previous
Next