search

databricks-industry-solutions / pixels

Facilitates simple large scale processing of HLS Medical images, documents, zip files. Previously at https://github.com/dmoore247/pixels
https://databricks-industry-solutions.github.io/pixels/
Other
25 stars 15 forks source link

Add Incremental Ingest (AutoLoader) capabilities #39

Closed dmoore247 closed 8 months ago

dmoore247 commented 9 months ago

Is your feature request related to a problem? Please describe. I would like to incrementally ingest data off of cloud storage and run the DicomMetadata extract, continue to append to the target delta table

Describe the solution you'd like Looks like AutoLoader is a good fit to monitory the source folder. TriggerAvailableNow is appropriate, need to run this once an hour or once a day.

Describe alternatives you've considered spark.readStream()...