search

fergiemcdowall / norch

A search server that can be installed with npm
655 stars 45 forks source link

Switch from web-server to DAT-server? #160

Closed eklem closed 2 years ago

eklem commented 5 years ago

Since search-index may be possible with hyperdb (DAT-protocol), should Norch move to be a DAT-server instead of a web-server?

It seems that applications can run DAT-stuff without the browser needing to support the DAT-protocol through dat://[url]. If not, it's a bit further into the future before this happens.

Anyway, not something that needs to happen in the near future.

fergiemcdowall commented 5 years ago

It would certainly be really interesting if we could get that working

eklem commented 5 years ago

Here is an app that is served over HTTP but is DAT at the core: https://olafchat.netlify.com/

eklem commented 5 years ago

I'll make something working here, but it will be simpler. So index still stored the same way as before. It's the data to index that will actually be distributed. And will do it for search-index in the browser first.

eklem commented 4 years ago

I think the norch-usecase for this is to have the full index searchable instantly and then use either replication to swap to a browser-based, or use DAT for the replication. DAT comes in handy if the application will get a lot of users. So far the easiest way of getting the DAT-part to work would be to store the data you want indexed, not the actual index itself, since there is no leveldown compatible DAT-implementation.

fergiemcdowall commented 2 years ago

I'm going to close this issue since it essentially depends on upstream development in the level ecosystem. But yes- a decentralized index is a tantalizing idea.

eklem commented 2 years ago

I've just started to look at how Webtorrents can work as storage and sharing. So, decentralizing the data before it's indexed. DAT/Hyper has moved to mostly just node, which means you'll need a server. Normal people don't have servers, so I want to move away from that and try to do most parts in a browser.

With Webtorrent, either do incremental webtorrents as in an incremental backup scheme, or check if it's possible to stream the data more fluently. Then each client can index the data them selves.

fergiemcdowall commented 2 years ago

I feel like this could be the web3 thing that actually means something :)