search

kreeben / resin

Vector space index based search engine that's available as a HTTP service or as an embedded library.
MIT License
568 stars 40 forks source link

Create abstraction layer for System.IO #23

Closed kreeben closed 4 years ago

kreeben commented 7 years ago

Today we want to be able to promise that a commit is acctually commited. We can do this by making the same promise to our clients as the file system make to us. In other words, we can write a file to disk, verify it's really there, commited and in a readable state and then we can tell our client that the scope of our promise has been fulfilled.

Later, we want to make other types of promises that the file system cannot subscribe to, e.g. we want to promise that data has been persisted not only one machine but on two and that both machines have the data in a readable state.

Thus the need for a file system abstraction layer.

ReubenBond commented 7 years ago

@kreeben have you taken a look at xoofx/zio? Maybe it would be suitable for this purpose (file system watchers are a TODO - but it looks like a good base).

As far as replication is concerned, I think that's best left to an implementation of something like Raft. I started prototyping one on top of dotnet/orleans, but it's certainly not production ready yet. It would be a fun project to host Resin on Orleans and productionize the Raft impl.

kreeben commented 7 years ago

@ReubenBond yes I read about it just the other day. It's on my watch list.

Hadn't heard of Orleans. Very interesting. I was thinking gRPC + some sort of mediation logic + some sort of load balancing but maybe this is more suitable. Tnx for the links.

jhashemi commented 7 years ago

@ReubenBond @kreeben

See: https://www.microsoft.com/en-us/research/publication/indexing-in-an-actor-oriented-database/

They did it on top of Orleans. I'm very interested in having a full text search based process on Orleans

kreeben commented 7 years ago

I shall have a look. Thx!

kreeben commented 4 years ago

This is not needed.