-
It must be possible to distinguish between new annotations applied to old unannotated data in the etl pipeline and annotations circa data collection.
Consider making the annotation include: the SHA…
-
As a user I would like to be able to upload large datasets directly from different data sources without converting them to GeoJSON. It would be great if I could use a GeoPackage, shapefile, FGDB, SQLi…
-
### Describe the feature
Support KeepJobFlowAliveWhenNoSteps Or Auto-termination (after idle) in stepfunction creates EMR cluster.
### Use Case
Our team is using Stepfunction EMR (EmrCreateCluste…
-
As the operations are nicely organized from overpass requests to result plotting, we may undertake a data pipeline formalization, for instance through a Python ETL like `Luigi` (see [doc](https://luig…
-
# Summary
After a fresh install, attempted to run arthur:
```
./bin/run_arthur.sh
++ pwd
+ docker run --rm --interactive --tty {volumes omitted} --env DATA_WAREHOUSE_CONFIG=/opt/data-warehou…
-
The avail-map-conflation-platform MUST be adhere to AVAIL's Data Management System's standards.
This would REQUIRE:
- Using standard Data Management Metadata (DMMetadata) for input processing.
…
-
### What's the task?
It's common for projects to require background jobs (i.e. for ETL pipelines, processing file uploads, etc). How could a team using the template create a background job? What is…
-
By design, there is one root pipeline where all other pipelines are added to. This might make sence when you just have one pipeline which just does one task, but I have several pipelines which I don't…
ghost updated
3 years ago
-
→ Not displayed here : I think roulette timing should be displayed in seconds, so that would be 0.5s, 1s, 2s etc instead of 50, 100, 200 ? It took me a long while to understand what it was x)
-
Over the last two days, I've seen the transient timeout failures resurface in ophys_etl_pipelines. See for instance:
https://app.circleci.com/pipelines/github/AllenInstitute/ophys_etl_pipelines/2417/…