-
- [ ] CSV input
- [ ] Can handle 10 GB files
-
- [x] Update config.yaml
- [x] Update secrets.yaml [Optional]
- [x] Update params.yaml
- [x] Update the entity
- [x] Update the configuration manager in src config.
- [x] Update the components
- [x] U…
-
# Background
- I made some changes to the repo in PR #35. You can review the PR to note the changes made to the repo.
- I added logic for programmatically downloading the FFIEC, HMDA, FDIC, and SB…
-
Research approaches to transfer data from Gatsby to Strapi
Possible approaches include:
- CSV import
- API endpoints
- Database import
- Custom importing scripting
- [ ] Determine and docu…
-
Do we want to be able to import data from arbitrary s3 buckets? What about other archives? I feel like we should design the user interface to allow for data ingestion from sources other than DANDI. Ma…
-
Setup OpenMetadata in order to integrate with our DBT pipelines (as code, contained in the data product). List of integrations to be tested:
https://docs.open-metadata.org/v1.4.x/connectors/ingestion/…
-
- [x] Update config.yaml
- [x] Update params.yaml
- [x] Research or analysis
- [x] Update the entity
- [x] Update the configuration manager in src config.
- [x] Update the components
- [x] Update the…
-
- [x] Update config.YAML
- [x] Research or analysis
- [x] Update the entity
- [x] Update the configuration manager src config.
- [x] Update the components
- [x] Update the pipeline
-
- [x] Update config.YAML
- [x] research or analysis
- [x] Update the entity
- [x] Update the configuration manager in src config.
- [x] Update the components
- [x] Update the pipeline
- [ ] Test case …
-
**Describe the bug**
Keys with "." in them are not able to be processed.
When ingesting logs from FluentBit -> S3 -> SQS -> Data Prepper / OSIS -> OpenSearch any key that has a dot "." in it is t…