search

eval-so / cruncher

A re-implementation of minibcs in Haskell.
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Buffer stdout and stderr to disk #1

Open duckinator opened 9 years ago

duckinator commented 9 years ago

The Shelly library buffers stdout (and presumably stderr) to memory by default. If the buffer gets too large the OOMKiller will kill the entire thing.

Can be reproduced with the following Ruby code: loop { puts 1 } Untested, but can presumably also be reproduced with: loop { warn 1 }

The easiest way to fix this would be to buffer to disk.

It would probably also be a good idea to set a fairly high buffer limit (a few megabytes, possibly?) to avoid excessive disk usage, but that can be pulled out into a separate issue if necessary.

The runHandles function may be useful in resolving this: https://hackage.haskell.org/package/shelly-1.6.1.2/docs/Shelly.html#g:3

Raynes commented 9 years ago

Random thought: it'd be nice (and I don't know that anyone has done this) if we had a websocket (i.e. socket.io) in there so an additional API with streaming could be provided. No buffering to worry about if you're streaming output. ^.^

relrod commented 9 years ago

That would be really cool, actually, but probably a longer term goal. I think it would be possible since Shelly can take any Handle. So if you can get a Handle that is sent automatically to some websocket connection (does websockets do this? Not sure. Another option is to see about maybe adding Pipes or Conduit support to shelly (or a fork or something). I feel like those kinds of libraries are made for solving this kind of thing), then you could do it pretty easily I think. The issue is figuring out where the user should connect for streaming output. For example, if they are using the API, do they make a call to get a stream URL first, then connect to that and hit the API at the same time? I think working out that process would require some thinking. But I think it's definitely worth looking into. I've had thoughts of looking into streaming for user input but have always abandoned those because I'm pretty set on not wanting to give up the timeout for that benefit. But streaming output is a much easier problem and we could just stream until the timeout was hit.

Raynes commented 9 years ago

I'm imagining an actual full on socket.io server. There seem to be Haskell implementations available, thankfully.

With socket.io, all the hard work is done. It can be interactive. You can listen to events from clients and clients can listen to events from you. Super straight forward. Check out my micdrop project for a small example of building this kind of API.

It's actually because of that project that I'm mentioning it. Fell in love with the concept :P