-
- [x] Update config.YAML
- [x] Update params.YAML
- [x] Research or analyis
- [x] Update the entity
- [x] Update the configuration manager in src config.
- [x] Update the components
- [x] Updat…
-
- [ ] First we have to dump the dataset into **MySQL** dataset
- [ ] Create / Establish a connection with database .
- [ ] Load the data and crate **dataframe**
- [ ] Drop the unnecessary columns…
-
During our test, we found that the osctrl logs the size of each successful request to the database. I would suggest switching to Prometheus metrics with a good overview and much less operational costs…
-
Tasks:
- [x] Land nodes ingest step
- [x] Land edges ingest step (already scaffolded, but needs the nodes to be completed)
- [ ] Food groups ingest (file `UniqueFG1_FG2.csv` seems to include the …
-
MVP data:
[MVP](https://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/study.cgi?study_id=phs002453.v1.p1) is an ongoing prospective cohort study and mega‐biobank in the Department of Veterans Affairs Heal…
-
## Expectation
As a user, I require access to the following datasets which I will access via the API:
1. Ground Observations (TAHMO stations) - csv
2. Ground Observations (TAHMO stations) - API
3. CB…
-
-
I'm looking into the AIND format option for inputting data into the pipeline. The readme states
> aind: data ingestion used at AIND. The input folder must contain an ecephys subfolder which in turn…
-
**Title**: Ingest LLM Data into Editable Table
**Description**: Develop a data ingestion pipeline to store new headlines and associated metadata in a structured, easily editable table.
**Tasks**:
…
-
Given now we support incremental models with splits, it will be good to also allow users to set a TTL on their data.
A common use case will be to load data for last X days for serving dashboards. Req…