search

elanthia-online / illthorn

Electron-based FE
18 stars 10 forks source link

Storage of Streams #103

Closed chriscoyier closed 4 years ago

chriscoyier commented 4 years ago

This seems pretty straightforward, assuming Storage is the right place for this?

If so, I just need to make sure to trim the storage to a reasonable level and then deal with storage on a per-account basis.

ondreian commented 4 years ago

I think this is fine for a PoC to start on the UI, but this will not work long-term.

Each character should have their own $XDG_HOME/data/<character>.thoughts file or something that is a Duplex stream with an append-only format (one item per line or a binary fixed size) so we can efficiently subset a feed without needing to read the whole file into memory and page to the appropriate file subset.

Most importantly all of these operations need to be async or we are going to run into jittering issues with the UI

chriscoyier commented 4 years ago

Each character should have their own $XDG_HOME/data/.thoughts file or something that is a Duplex stream with an append-only format (one item per line or a binary fixed size) so we can efficiently subset a feed without needing to read the whole file into memory and page to the appropriate file subset.

That sounds awesome but I don't know how to do that. Happy to follow your lead there, or feel free to point me to some resources to bone up.

Most importantly all of these operations need to be async or we are going to run into jittering issues with the UI

I tried making them async but not sure if I was successful or not.

I am now limiting the number of messages it will store and retrieve, so hopefully, that's an improvement.

The thing that will "finish" making this a working prototype is storing the streams at a per-character storage location, but I just can't figure out how to access the Session properly in the streams.js file and spinning my wheels on it, so I left some TODOs in those spots.

chriscoyier commented 4 years ago

OK this is in better shape.

It's not using flat files yet. I understand that's the correct direction, but this should be functional. It doesn't store oodles of data, just the last 100. And the functions should be async (I don't notice any jitter). It also stores streams on a per-character basis.

I'll probably merge unless there is objection to this implementation being on master. I just really like being able to refresh and not losing my streams.