invenia / JLSO.jl

Julia Serialized Object (JLSO) file format for storing checkpoint data.
MIT License
90 stars 5 forks source link

Support HDF5/JLD/JLD2 format #23

Open oxinabox opened 4 years ago

oxinabox commented 4 years ago

While none of these formats support directly streaming, and need to file we could always work around that by creating tempfiles, then reading them back in.

And that will be fine since most linux puts TMPDIR on a ramdrive right?

this is how I used to save data in the JLD, on SwiftObjectStorage https://github.com/oxinabox/SwiftObjectStores.jl/blob/45acf6e5b9446672674964122763528a4d452322/src/upload.jl#L42-L51

rofinn commented 4 years ago

We could support that, but it won't be useful for our use case on AWS Batch. On batch, we don't want large temp files in our containers as this will complicate our configuration and result in blocking on IO more often.