-
○ Time: 1 week
○ Tools Required: Azure Data Factory
○ Steps:
1. Design ETL processes to extract data from data providers.
2. Transform the data into suitable formats for analysis.
…
zepor updated
3 weeks ago
-
**Description:**
The Extract-Transform-Load (ETL) design pattern is crucial for data integration and data warehousing processes. It involves extracting data from various sources, transforming it to fi…
-
The question this issue is two fold. To start a discussion on how to transform existing datasets into a form that can work in a format that Citygram expects, and second to recommend a few tools that a…
-
Apache NiFi
Apache Airflow
pandas in Python
-
The data files in Participant Data are tricky to work with in Python requiring unclear expressions to access the data, eg
`age = data['data'][0,0]['individual'][0,0]['age'][0][0]`
In order…
-
https://www.youtube.com/watch?v=ad1GmW_TmYg
-
### Discussed in https://github.com/apache/airflow/discussions/33556
Originally posted by **ntnhaatj** August 20, 2023
### Description
Hi, as [my issue was raised here](https://github.com/a…
-
## User Story
In order to identify memory leaks in the harvester, datagovteam wants to conduct a formal analysis using industry standard memory profiling tools.
## Acceptance Criteria
[ACs sh…
-
This file format is used for datasets that are aligned against a combined reference (i.e. hg19 + mm10). It contains additional columns with per-species counts and a call that indicate which species th…
-
As part of our pilot program with STLT's we need to understand the challenges that come with putting data into Maven or Clinisys assume our data is in JSON format the question we should answer is
1. W…