databricks-industry-solutions / pixels

Facilitates simple large scale processing of HLS Medical images, documents, zip files. Previously at https://github.com/dmoore247/pixels
https://databricks-industry-solutions.github.io/pixels/
Other
25 stars 15 forks source link

ENH: Add signed URL transformer #46

Open dmoore247 opened 9 months ago

dmoore247 commented 9 months ago

Is your feature request related to a problem? Please describe. Need to make a cohort of images available to a specific consumer.

Describe the solution you'd like Based on the cloud storage path, create a short lived signed URL for compatibility with other tools

Based on the cloud platform and integrated IAM role ... create a signed URL. Return the URL as a column in the output_col. The Transformer parameters should include duration, default 3600 seconds.

Describe alternatives you've considered Direct paths work find when running on an integrated VM but don't work well with off-cloud consumers.

Return a URL based on a Driver Proxy and DBTunnel is a possible alternative

Additional context