search

bcpeinhardt / simplifile

Simple file operations for Gleam that work on all targets (Erlang/Node/Deno)
71 stars 10 forks source link

File streams #22

Closed richard-viney closed 6 months ago

richard-viney commented 6 months ago

It would be great if this library offered support for file streams when reading and writing data. Right now I believe users are required to read all data into memory, but file streams would help avoid that.

One could use the existing append() to "stream" data out to a file, but that probably has a lot of overhead of constantly opening and closing the file etc.

I'm not 100% aware of the state of IO streams in Gleam at present, maybe someone is already looking at it? If so then I'd be happy to help out as I'm working on a project that needs them and I'm currently writing my own interfaces to the relevant Erlang functions.

Thanks.

bcpeinhardt commented 6 months ago

Hello!

I'm willing to look into this a little further, but at first glance it might be too complicated for simplifile haha. Simplifile's goal is to provide a unified interface for common file operations when targeting Erlang/Node/Deno. That means on the JavaScript side, we're limited to using nodes sync file APIs, as the concurrency systems of Erlang and JavaScript are fundamentally different. If you come up with a decent setup for streaming files in either target, I would encourage you to publish it and I'll happily link to it in Simplifile's docs as more people will want this down the line.

richard-viney commented 6 months ago

I think you may be right that the APIs are a bit too different and Node.sj would require async callbacks and/or promises, which I think are just not possible to wrap?

I have some code I'm using for file streaming in Gleam on the Erlang target which works fine, if I end up publishing it I'll let you know, or I might just attach it here for the next person.

Thanks!