search

databricks-industry-solutions / pixels

Facilitates simple large scale processing of HLS Medical images, documents, zip files. Previously at https://github.com/dmoore247/pixels
https://databricks-industry-solutions.github.io/pixels/
Other
25 stars 15 forks source link

Improved unzip logic #60

Closed erinaldidb closed 2 months ago

erinaldidb commented 3 months ago

The new unzip logic will create a separate flow that unzips the files, calculate the necessary partitions based on the last 10 commits in the delta table and rebalances the unzipped files.