-
_Idea_:
Moving to a new repository positions the build as a more generic solution in the k8s landscape. This will end-up with the same repo under this new umbrella org, but with a different name.
…
-
The following SHP files here could not be copied to SQL Server
http://www3.stats.govt.nz/digitalboundaries/annual/ESRI_Shapefile_Digital_Boundaries_2015_Generalised_Clipped.zip
The error log for the…
-
I'm trying to change the IRIs of pipelines that I created locally to be deployed on a remote LinkedPipes instance. I have created a script that executes the commands in [this Wiki entry](https://githu…
-
**- Title**
_Azure + Python = Data Analytics ♥_
**- Brief description about the content to be covered**
_Learn how Azure and python are tightly integrated in Data engineering & Data Science sce…
-
## Description
Most of my usage requires running the same pipeline independently on each of a set of partitions. For example, an ETL pipeline that selects 1 week's worth of data, applies a sequence o…
-
**Components:**
- Mirror node importer (publishes to pubsub)
- Dataflow jobs
- Dedupe job
**Resources needed:**
1. BigQuery tables : transactions, errors, dedupe_state
2. PubSub topic for transaction…
-
I've been following Dagster for a month or so as we're looking to revamp our data pipelines at $company. We'll be using Spark for the majority of our ETLs, but using Databricks to manage our infrastru…
-
I love the work your team is doing here, and I'd love to help out in any way that i can. I'm a product manager with technical skillsets for wrangling data, but i also do a fair amount of technical wr…
-
**Feature Request**
If the inputs, container, command, etc. for a workflow step are all identical to a step performed in a previous workflow, the step should be skipped and the output from the prev…
-
This example code on my system, I assume should run without error:
```python
from data_integration.commands.bash import RunBash
from data_integration.pipelines import Pipeline, Task
from data_in…