-
We need to develop a robust and scalable data ingest/ETL (Extract, Transform, Load) pipeline to facilitate the reading of eQTL (expression Quantitative Trait Loci) data from FTP sources, indexing it i…
-
Blocked by #73
Context
-------
1. Set up an Airflow environment within Docker.
2. Define DAG for fetching and updating the data in DB tables - reuse ETL code from #73 for fetching the data from…
-
**Is your feature request related to a problem? Please describe.**
Currently our ETL job runs every 30 minutes and inserts a file into S3, triggering OpenSearch ingestion pipeline. Due to varying ETL…
-
We're looking to enhance Elsa's data processing capabilities. To do this, we're drawing ideas from Azure Data Factory. Here's a breakdown of the new concepts we're considering for integration:
1. *…
-
task; https://n-abledev.atlassian.net/browse/DA-2724?atlOrigin=eyJpIjoiNWQ4NDhmYzIwNGViNDJlOWIxYjhiZmJlMzgwYjMwZTMiLCJwIjoiaiJ9
-
Please update footer structure and links based on the following suggested links.
| Product | Resources | Company |
|-…
-
### Mage version
0.9.73
### Describe the bug
I have issues with the pipeline schedules that are not triggered, pipelines stuck in `Ready` and `Running` after I downgraded my database spec (CP…
-
Hi Lampros,
We're deploying the PublicationsRetriever in an ETL pipeline that is not as stable as we would like, so sometimes jobs are retried. I would like to know if it would be possible to imple…
-
## Description
In ETL pipelines, loading transformed data into various data warehouses is a critical requirement. Currently, the `ibis.TableDataset ` connector in Kedro does not support data insertio…
-
Enhance the ETL process to include a concept of handlers.
These handlers will take data from the source and allow the developer to manipulate the incoming data before sending it on to the destination…