search

areaDetector / ADCore

The home of the core components of the EPICS areaDetector software. It includes base classes for drivers and code for all of the standard plugins.
https://areadetector.github.io/master/index.html
Other
20 stars 65 forks source link

Create AWS S3 Interface for TIFF Files #458

Open stuwilkins opened 3 years ago

stuwilkins commented 3 years ago

@MarkRivers .. This is more of an exploratory "DEMO" PR, I wanted to test using the AWS S3 storage directly from AD that we talked about over e-mail.

I have verified that this works at the moment writing to a MinIO object store at SDCC here at BNL but I have no reason it would not work directly to Amazon S3 object storage.

MinIO is here:

https://min.io/

This basically gets rid of the requirement to have a file system to write images to.

This is "super hacky" and is so brute force and ugly it might even be elegant.

With some changes to the base NDFileTIFF it would only be a small shim over the files.

Comments from all welcome...

P.S. I realize that there is copied code in there. I didn't want to play too much with the NDFileTIFF at this stage. However, one could separate this out a little bit and make the NSFileTIFFS3 really simple.

coveralls commented 3 years ago

Coverage Status

Coverage remained the same at 38.916% when pulling 1a6cc516705e762e10e262990aadfc3b2f3cd2af on stuwilkins:add_tiff_s3 into 1324d63350638f5106201fda288b6ece9af4974b on areaDetector:master.

tacaswell commented 3 years ago

https://github.com/constantinpape/z5 may also be of interest to target more cloud-native data formats.

stuwilkins commented 3 years ago

Thanks @tacaswell This is just a test, not sure I am married to TIFF (as you know!). But, by using the AWS S3 async upload I can get line rate out of my office computer.........