initial implementation for pixels with streaming

databricks-industry-solutions / pixels

Facilitates simple large scale processing of HLS Medical images, documents, zip files. Previously at https://github.com/dmoore247/pixels

Other

25 stars 15 forks source link

Closed erinaldidb closed 8 months ago

erinaldidb commented 8 months ago

To enable the streaming capability a parameter is required in catalog reading step.

from dbx.pixels import Catalog

catalog = Catalog(spark, table)
catalog_df = catalog.catalog(path, streaming=True)

dmoore247 commented 8 months ago

Note that test and lint failed. @erinaldidb Please run make style and update PR.