search

lior-k / fast-elasticsearch-vector-scoring

Score documents using embedding-vectors dot-product or cosine-similarity with ES Lucene engine
Apache License 2.0
395 stars 112 forks source link

Elastic 6.1.1 #7

Closed sctrueew closed 5 years ago

sctrueew commented 6 years ago

Hello! How can I build it for Elastic version 6.1.1?

lior-k commented 6 years ago

Hi there, I suppose changing 5.6.0 in all files to 6.1.1 + mvn clean install should do the trick, that is if there are no changes in elastic-search plugin API.

Please try it & let me know how it goes

sctrueew commented 6 years ago

OK,I'll try it thank you

sctrueew commented 6 years ago

Hi! I tried it and I've installed without any problems and then I've restarted my Elasticsearch, but when I've run query I got this error:

1. PUT /test { "mappings": { "t": { "properties": { "embedding_vector": { "type": "binary", "doc_values": true } } } } }

2. PUT /test/t/1 { "id": 1, "embedding_vector": "v7l48eAAAAA/s4VHwAAAAD+R7I5AAAAAv8MBMAAAAAA/yEI3AAAAAL/IWkeAAAAAv7s480AAAAC/v6DUgAAAAL+wJi0gAAAAP76VqUAAAAC/sL1ZYAAAAL/dyq/gAAAAP62FVcAAAAC/tQRvYAAAAL+j6ycAAAAAP6v1KcAAAAC/bN5hQAAAAL+u9ItAAAAAP4ckTsAAAAC/pmkjYAAAAD+cYpwAAAAAP5renEAAAAC/qY0HQAAAAD+wyYGgAAAAP5WrCcAAAAA/qzjTQAAAAD++LBzAAAAAP49wNKAAAAC/vu/aIAAAAD+hqXfAAAAAP4FfNCAAAAA/pjC64AAAAL+qwT2gAAAAv6S3OGAAAAC/gfMtgAAAAD/If5ZAAAAAP5mcXOAAAAC/xYAU4AAAAL+2nlfAAAAAP7sCXOAAAAA/petBIAAAAD9soYnAAAAAv5R7X+AAAAC/pgM/IAAAAL+ojI/gAAAAP2gPz2AAAAA/3FonoAAAAL/IHg1AAAAAv6p1SmAAAAA/tvKlQAAAAD/I2OMAAAAAP3FBiCAAAAA/wEd8IAAAAL94wI9AAAAAP2Y1IIAAAAA/rnS4wAAAAL9vriVgAAAAv1QxoCAAAAC/1/qu4AAAAL+inZFAAAAAv7aGA+AAAAA/lqYVYAAAAD+kNP0AAAAAP730BiAAAAA=" }

3.

POST test/_search { "query": { "function_score": { "boost_mode": "replace", "script_score": { "script": { "inline": "binary_vector_score", "lang": "knn", "params": { "cosine": false, "field": "embedding_vector", "vector": [ -0.09217305481433868, 0.010635560378432274, -0.02878434956073761, 0.06988169997930527, 0.1273992955684662, -0.023723633959889412, 0.05490724742412567, -0.12124507874250412, -0.023694118484854698, 0.014595639891922474, 0.1471538096666336, 0.044936809688806534, -0.02795785665512085, -0.05665992572903633, -0.2441125512123108, 0.2755320072174072, 0.11451690644025803, 0.20242854952812195, -0.1387604922056198, 0.05219579488039017, 0.1145530641078949, 0.09967200458049774, 0.2161576747894287, 0.06157230958342552, 0.10350126028060913, 0.20387393236160278, 0.1367097795009613, 0.02070528082549572, 0.19238869845867157, 0.059613026678562164, 0.014012521132826805, 0.16701748967170715, 0.04985826835036278, -0.10990987718105316, -0.12032567709684372, -0.1450948715209961, 0.13585780560970306, 0.037511035799980164, 0.04251480475068092, 0.10693439096212387, -0.08861573040485382, -0.07457160204648972, 0.0549330934882164, 0.19136285781860352, 0.03346432000398636, -0.03652812913060188, -0.1902569830417633, 0.03250952064990997, -0.3061246871948242, 0.05219300463795662, -0.07879918068647385, 0.1403723508119583, -0.08893408626317978, -0.24330253899097443, -0.07105310261249542, -0.18161986768245697, 0.15501035749912262, -0.216160386800766, -0.06377710402011871, -0.07671763002872467, 0.05360138416290283, -0.052845533937215805, -0.02905619889497757, 0.08279753476381302 ] } } } } }, "size": 100 }

===> error

{ "error": { "root_cause": [ { "type": "query_shard_exception", "reason": "script_score: the script could not be loaded", "index_uuid": "U-tuPNjRTNe5ahOmIa99vw", "index": "test" } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "test", "node": "RBUDFDK0Sv2YJX5n86pacg", "reason": { "type": "query_shard_exception", "reason": "script_score: the script could not be loaded", "index_uuid": "U-tuPNjRTNe5ahOmIa99vw", "index": "test", "caused_by": { "type": "illegal_argument_exception", "reason": "script_lang not supported [knn]" } } } ] }, "status": 400 }

lior-k commented 6 years ago

Please paste the Elasticsearch log here

On Mon, May 14, 2018, 3:28 PM mehrdad mazhari notifications@github.com wrote:

Hi! i tried it and I've installed without any problems but when I've run query I got this error:

1.

PUT /test { "mappings": { "t": { "properties": { "embedding_vector": { "type": "binary", "doc_values": true } } } } }

1.

PUT /test/t/1 { "id": 1, "embedding_vector": "v7l48eAAAAA/s4VHwAAAAD+R7I5AAAAAv8MBMAAAAAA/yEI3AAAAAL/IWkeAAAAAv7s480AAAAC/v6DUgAAAAL+wJi0gAAAAP76VqUAAAAC/sL1ZYAAAAL/dyq/gAAAAP62FVcAAAAC/tQRvYAAAAL+j6ycAAAAAP6v1KcAAAAC/bN5hQAAAAL+u9ItAAAAAP4ckTsAAAAC/pmkjYAAAAD+cYpwAAAAAP5renEAAAAC/qY0HQAAAAD+wyYGgAAAAP5WrCcAAAAA/qzjTQAAAAD++LBzAAAAAP49wNKAAAAC/vu/aIAAAAD+hqXfAAAAAP4FfNCAAAAA/pjC64AAAAL+qwT2gAAAAv6S3OGAAAAC/gfMtgAAAAD/If5ZAAAAAP5mcXOAAAAC/xYAU4AAAAL+2nlfAAAAAP7sCXOAAAAA/petBIAAAAD9soYnAAAAAv5R7X+AAAAC/pgM/IAAAAL+ojI/gAAAAP2gPz2AAAAA/3FonoAAAAL/IHg1AAAAAv6p1SmAAAAA/tvKlQAAAAD/I2OMAAAAAP3FBiCAAAAA/wEd8IAAAAL94wI9AAAAAP2Y1IIAAAAA/rnS4wAAAAL9vriVgAAAAv1QxoCAAAAC/1/qu4AAAAL+inZFAAAAAv7aGA+AAAAA/lqYVYAAAAD+kNP0AAAAAP730BiAAAAA=" }

1.

POST test/_search { "query": { "function_score": { "boost_mode": "replace", "script_score": { "script": { "inline": "binary_vector_score", "lang": "knn", "params": { "cosine": false, "field": "embedding_vector", "vector": [ -0.09217305481433868, 0.010635560378432274, -0.02878434956073761, 0.06988169997930527, 0.1273992955684662, -0.023723633959889412, 0.05490724742412567, -0.12124507874250412, -0.023694118484854698, 0.014595639891922474, 0.1471538096666336, 0.044936809688806534, -0.02795785665512085, -0.05665992572903633, -0.2441125512123108, 0.2755320072174072, 0.11451690644025803, 0.20242854952812195, -0.1387604922056198, 0.05219579488039017, 0.1145530641078949, 0.09967200458049774, 0.2161576747894287, 0.06157230958342552, 0.10350126028060913, 0.20387393236160278, 0.1367097795009613, 0.02070528082549572, 0.19238869845867157, 0.059613026678562164, 0.014012521132826805, 0.16701748967170715, 0.04985826835036278, -0.10990987718105316, -0.12032567709684372, -0.1450948715209961, 0.13585780560970306, 0.037511035799980164, 0.04251480475068092, 0.10693439096212387, -0.08861573040485382, -0.07457160204648972, 0.0549330934882164, 0.19136285781860352, 0.03346432000398636, -0.03652812913060188, -0.1902569830417633, 0.03250952064990997, -0.3061246871948242, 0.05219300463795662, -0.07879918068647385, 0.1403723508119583, -0.08893408626317978, -0.24330253899097443, -0.07105310261249542, -0.18161986768245697, 0.15501035749912262, -0.216160386800766, -0.06377710402011871, -0.07671763002872467, 0.05360138416290283, -0.052845533937215805, -0.02905619889497757, 0.08279753476381302 ] } } } } }, "size": 100 }

===> error

{ "error": { "root_cause": [ { "type": "query_shard_exception", "reason": "script_score: the script could not be loaded", "index_uuid": "U-tuPNjRTNe5ahOmIa99vw", "index": "test" } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "test", "node": "RBUDFDK0Sv2YJX5n86pacg", "reason": { "type": "query_shard_exception", "reason": "script_score: the script could not be loaded", "index_uuid": "U-tuPNjRTNe5ahOmIa99vw", "index": "test", "caused_by": { "type": "illegal_argument_exception", "reason": "script_lang not supported [knn]" } } } ] }, "status": 400 }

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/lior-k/fast-elasticsearch-vector-scoring/issues/7#issuecomment-388799324, or mute the thread https://github.com/notifications/unsubscribe-auth/AExkSOR_wiQVWkeRzUxBNg2L2H81O9NAks5tyXiEgaJpZM4Tzk_E .

sctrueew commented 6 years ago

[2018-05-15T08:18:51,488][DEBUG][o.e.a.s.TransportSearchAction] [.....] All shards failed for phase: [query] org.elasticsearch.index.query.QueryShardException: script_score: the script could not be loaded at org.elasticsearch.index.query.functionscore.ScriptScoreFunctionBuilder.doToFunction(ScriptScoreFunctionBuilder.java:99) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.index.query.functionscore.ScoreFunctionBuilder.toFunction(ScoreFunctionBuilder.java:138) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.index.query.functionscore.FunctionScoreQueryBuilder.doToQuery(FunctionScoreQueryBuilder.java:298) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$2(QueryShardContext.java:304) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:316) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:303) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.search.SearchService.parseSource(SearchService.java:701) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.search.SearchService.createContext(SearchService.java:552) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:528) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:324) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:310) [elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:306) [elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:996) [elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) [elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.1.jar:6.1.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.lang.IllegalArgumentException: script_lang not supported [knn] at org.elasticsearch.script.ScriptService.getEngine(ScriptService.java:240) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.script.ScriptService.compile(ScriptService.java:294) ~[elasticsearch-6.1.1.jar:6.1.1] at org.elasticsearch.index.query.functionscore.ScriptScoreFunctionBuilder.doToFunction(ScriptScoreFunctionBuilder.java:95) ~[elasticsearch-6.1.1.jar:6.1.1]

sctrueew commented 6 years ago

What do I do?

lior-k commented 6 years ago

For some reason I fail to compile it for 6.1.1 can you issue a merge request with your 6.1.1 branch?

will3216 commented 6 years ago

I just spent a few hours looking at this, and 6.x.x has breaking changes in it which will require the plugin to be rewritten. I tried and failed to rewrite it myself (no experience with java or elasticsearch), so for now I'm just going to downgrade my elasticsearch cluster since this work I am using this with is only in the testing phases. If I get a chance, I will try again to rewrite the plugin to support 6.x.x but no promises!

tknobi commented 6 years ago

Here is a updated version of the plugin, using elasticsearch v6.1. Update: Build failed..., did anyone found a working version?

wmelton commented 6 years ago

We ported lior-k's original update to 6.4.x+. This includes changes for breaking changes and utilizes the new ScoreScript (released in 6.4.0) in accordance with Elasticsearch's example/recommendation. Plugin can be found here - https://github.com/StaySense/fast-cosine-similarity

lior-k commented 5 years ago

Added support for ES 6 & 7. see branches es-7.1.0 & es-6.8.1 respectably