search

JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
553 stars 88 forks source link

BufferedReader/writer for general IO? #233

Closed grero closed 1 month ago

grero commented 4 years ago

I was wondering whether it would be feasible to have BufferedReader/Writer work for general io, rather than just for IOStream? The reason I'm asking is that I'm experimenting with reading parts of large jld2 files (several 10s of G) over sftp. I implemented an IO compliant interface for reading/writing and seeking, and now I would like to have BufferedReader be able to read from this IO. Looking at the source, it seems that if the io field of BufferedReader were of type IOrather than IOStream, I could use it without having to re-implement the BufferedReader interface. I'd be interesting in any thoughts on this approach. Thanks!

JonasIsensee commented 4 years ago

Hi @grero ,

this sounds like an interesting idea that could be useful in some applications. I must admit I have not looked at that part of the code before and at this point cannot help with that but your suggestion sounds plausible.

I would recommend that you fork JLD2 and just try it out and you are very welcome to report your findings. If it works and doesn't break anything else in JLD2, we could then likely adapt the signatures to your needs.

grero commented 4 years ago

I'm close to having something that sort of works here: https://github.com/grero/RemoteFiles.jl/pull/4 Once I have something a bit more polished perhaps we can discuss whether to include the functionality in JLD2 itself. Thanks for your reply!

JonasIsensee commented 3 years ago

Hi @grero,

I just wanted to ask whether you've made any progress on this

grero commented 3 years ago

Hey, I'm so sorry, I sort of got distracted with other projects. Regarding the tests, I was trying to figure out how best to structure them. I gather there aren't currently any tests that explicitly tests the BufferedReader? Would the best approach be to add those, or to basically run the current tests again, using a mocked up bytesavailble?

JonasIsensee commented 3 years ago

Hi, no worries. This isn't urgent but I just didn't want the idea to be forgotten.

I think you're right. I haven't seen any explicit tests for that either. If you've had a look at the tests you'll realize that they are in a bad state in terms of organization. Anything is appreciated but I think it'll be best to just add a few small tests for your mockup bytesavailable. (Really just some tiny tests that reproduce the bug you found)

JonasIsensee commented 1 month ago

Closing as duplicate of #304 and #346