Add table creation from files

smart-on-fhir / cumulus-library

https://docs.smarthealthit.org/cumulus/library/

Apache License 2.0

2 stars 0 forks source link

Add table creation from files #230

Closed dogversioning closed 2 months ago

dogversioning commented 2 months ago

This PR makes the following changes:

Adds a function to databases to upload parquets to S3
Adds templates for creating tables from external files
converts vocab study to use database upload

Checklist

[X] Consider if documentation (like in docs/) needs to be updated
[X] Consider if tests should be added
[X] Update template repo if there are changes to study configuration

dogversioning commented 2 months ago

also noting: file upload means that we're going to need to allow file deletion, in cases where a table needs to be recreated with a new dataset. I don't think it's critical right this second - I'm currently deferring that pending a discussion on the mrconso scope, where i might need to do this. but i'll create a issue regardless if we're ok with ignoring for now.