search

fergiemcdowall / norch

A search server that can be installed with npm
655 stars 45 forks source link

Proposal: index arrays instead of newline delimited "streams" #158

Closed fergiemcdowall closed 2 years ago

fergiemcdowall commented 6 years ago

Personally although I like the promise of streams,I think that they are becoming an increasingly esoteric way of reading large blobs of data, and are creating a barrier to people who just want to get going with Norch.

Whould anybody be unhappy if we changed the /add endpoint to accept arrays instead of streams?

eklem commented 6 years ago

I love the indexing part of the stream API on search-index, but since we're talking about the server-side here, I don't think it's a problem.

But maybe be able to add streams as well, even if through some tools. Today search-index has add and concurrentAdd (for arrays). You're thinking of swapping to the concurrentAdd? The memory issues of indexing went away with streams, so for large data-sets we should maybe keep a way to do this for search-index, while the norch-endpoints could be the user friendlier version?

fergiemcdowall commented 6 years ago

@eklem I feel like it would be better if Norch's add accepted standard JSON.parse-able objects as default with the option to specify more advanced inputs such as streams as an add-on, or preprocessing stage

eklem commented 6 years ago

Yes, sounds good to have add as the standard choice with easy-to-use arrays as input. Also have some dirty tricks to parse streams in the browser when querying a norch-instance. Could be nice to get rid of those too =)

fergiemcdowall commented 2 years ago

Upcoming version is de-streamified