Open oxinabox opened 4 years ago
We could support that, but it won't be useful for our use case on AWS Batch. On batch, we don't want large temp files in our containers as this will complicate our configuration and result in blocking on IO more often.
While none of these formats support directly streaming, and need to file we could always work around that by creating tempfiles, then reading them back in.
And that will be fine since most linux puts
TMPDIR
on a ramdrive right?this is how I used to save data in the JLD, on SwiftObjectStorage https://github.com/oxinabox/SwiftObjectStores.jl/blob/45acf6e5b9446672674964122763528a4d452322/src/upload.jl#L42-L51