search

JuliaData / Feather.jl

Read and write feather files in pure Julia
https://juliadata.github.io/Feather.jl/stable
Other
109 stars 27 forks source link

No warning when harddrive is full #125

Closed felixholub closed 4 years ago

felixholub commented 5 years ago

When saving a feather on a (very) full hard-drive, I noticed the following behavior: Once the hard-drive is at 100%, writing stops without throwing any error or warning. One only notices that the resulting feather file is broken when one tries to open it.

ExpandingMan commented 5 years ago

Interesting. My first question would be whether this is really the responsibility of the program writing the file, it seems like the responsiblity of the OS. What are you running this on?

I believe handling this would require talking to the OS's API. In the linux case, in the worst case scenario we could probably just use df via bash, but this raises all sorts of interesting questions about whether it's even possible to infer which device the user is trying to write the file to. I don't believe one can check any of this in Julia's built-in POSIX bindings.

Also, this package nominally supports Windows, so in those cases presumably we'd need to talk to the Windows API, for which I do not believe any Julia wrapper exists, and I don't want to experience any of the macabre horrors of attempting to call it, powershell or any other Windows-related monstrosity.

So, I'm sort of doubtful that anything can or should be done here, but I'd be interested to hear from anyone knowledgable about how to handle this sort of thing. I suppose if it turns out that it is appropriate to handle this case, PR's would be called for in every Julia program that writes files.

felixholub commented 5 years ago

Thank you for the interesting explanation! I'm using Debian 9. So this could also affect writing txt files in Julia core?

nalimilan commented 4 years ago

Don't you just need to check that write returns a value greater than 0? If it doesn't then you know something went wrong and an error should be thrown.

ExpandingMan commented 4 years ago

I suppose we could check the return value of write, but it seems to me one would have to check the exact value, not just whether it is > 0. More to the point: are we really supposed to be doing this? If so, it would seem every package everywhere should constantly be checking how many bytes it was able to write. Last I checked, most packages just don't do that. It really sounds like an edge case to me that should be handled by the OS, but isn't in this case.

Any opinions on this? If there is some concensus we should do this, I'm not opposed to it, but it doesn't seem right.

davidanthoff commented 4 years ago

I agree with @ExpandingMan, I think covering these edge cases is pretty hopeless, to be honest. Once your hard drive is really literally full, lots of things on your system probably won't work properly.

nalimilan commented 4 years ago

I guess ideally we would get an exception in that case. Actually that sounds more like something Julia could handle, as the OS provides the needed information: it's just too annoying to handle properly in all possible places. Maybe worth asking on Discourse.

I agree that all sorts of bad things will happen if the disk is full, but it would be nice to cancel the write and remove the file if we can detect that situation. One reason for doing it here is that data files are generally large, so they can easily fill your disk. If we remove the partial write and throw an error, the disk is likely to have a lot of free space, making it a harmless error; if we don't, other things may go bad.

ExpandingMan commented 4 years ago

Is this perhaps a libuv issue? This smells to me a bit like something that is handled by some operating systems, but which linux tends to be lazy about because it is designed to also be able to run on a light swtich.

davidanthoff commented 4 years ago

Trust me, if your disc is full on Windows, everything stops working, certainly no graceful exit :)

nalimilan commented 4 years ago

AFAICT write(::IOStream, ::Array) ends up calling unsafe_write, which calls ios_write. So libuv doesn't seem to be involved. I haven't checked very thoroughly, but ios_write doesn't seem to check very consistently that the return code is positive: sometimes it adds it to the number of written bytes, which apparently assumes that it's positive. Maybe that's because it knows no error can happen there, but that sounds suspicious and maybe worth filing an issue in Julia.

ExpandingMan commented 4 years ago

This sounds like it is a Julia issue, if anything. I'm going to close this issue.