custom document processing pipe without touching norch?

fergiemcdowall / norch

A search server that can be installed with npm

655 stars 45 forks source link

custom document processing pipe without touching norch? #153

Open eklem opened 7 years ago

eklem commented 7 years ago

I was thinking that it could be good to choose different custom document processors when adding documents. This could be done by (almost) not touching norch code.

Could be done by adding the document processors as dependencies in package.json and then calling them from command line:

Here a twitter document processor is inserted before it's added to the index.

curl -X POST -p twitterPipeline -d @justTen.str http://localhost:3030/add

fergiemcdowall commented 7 years ago

It should be possible to use unix pipes:

cat data.json | twitterPipeline | curl -H "Content-Type: application/json" -X POST -d @- http://localhost:3030/add

eklem commented 7 years ago

Ah, nice! A little unclear issue. I was also thinking of when the post request comes from i.e. Zapier and I can only rely on the regular /add functionality. A way to get Norch to choose one of a set of custom pipelines is what I'm looking for.

fergiemcdowall commented 7 years ago

Yup- I see what you mean- what should the API look like?

eklem commented 7 years ago

Hmm, maybe something could be sent as header key/value pair: customPipeline | [source]

eklem commented 7 years ago

I need to fix a bug in a demo, but I can see what this looks like on the norch server side and get back to you.

eklem commented 7 years ago

Could be done through the JSON, but that's a bit ugly.