search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.6k stars 1.01k forks source link

Add a demo search server [LUCENE-5376] #6440

Closed asfimport closed 8 years ago

asfimport commented 10 years ago

I think it'd be useful to have a "demo" search server for Lucene.

Rather than being fully featured, like Solr, it would be minimal, just wrapping the existing Lucene modules to show how you can make use of these features in a server setting.

The purpose is to demonstrate how one can build a minimal search server on top of APIs like SearchManager, SearcherLifetimeManager, etc.

This is also useful for finding rough edges / issues in Lucene's APIs that make building a server unnecessarily hard.

I don't think it should have back compatibility promises (except Lucene's index back compatibility), so it's free to improve as Lucene's APIs change.

As a starting point, I'll post what I built for the "eating your own dog food" search app for Lucene's & Solr's jira issues http://jirasearch.mikemccandless.com (blog: http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html ). It uses Netty to expose basic indexing & searching APIs via JSON, but it's very rough (lots nocommits).

Migrated from LUCENE-5376 by Michael McCandless (@mikemccand), 1 vote, resolved Jul 29 2016 Attachments: lucene-demo-server.tgz

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1557694 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1557694

LUCENE-5376: fix nocommits, add tests, get RAMDir indices working more correctly; rename 'index' param to 'search' when registering a field

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1557855 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1557855

LUCENE-5376: all custom Directory implementations accessible in CP; always return suggest count from buildSuggest; fix highlighting in AnalyzingInfixSuggster to render strait to JSONObject; add HTTP compression nocommits

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1557868 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1557868

LUCENE-5376: add merge max MB/sec rate limiting

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1559196 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1559196

LUCENE-5376, #6462: add missingLast support to lucene server

asfimport commented 10 years ago

Areek Zillur (@areek) (migrated from JIRA)

6467 adds support for counting # of entries a lookup was built with and returns it on the build command, Dictionary uses InputIterator instead of BytesRefIterator among other things. I think it will get rid of some nocommits here?

asfimport commented 10 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

6467 adds support for counting # of entries a lookup was built with and returns it on the build command, Dictionary uses InputIterator instead of BytesRefIterator among other things. I think it will get rid of some nocommits here?

Wow, super, this will fix a few nocommits in the server. Thanks Areek, I'll have a look!

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1559524 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1559524

LUCENE-5376: cutover to analysis factories to create tokenizer and token filters; still need char filters & whole analyzers

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1559542 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1559542

LUCENE-5376: always throw unchecked cause of the InvocationTargetException

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1559670 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1559670

LUCENE-5376: get 'file-like resources' working with char filters, fix nocommits, tests, clean up the analysis factory code

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1559805 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1559805

LUCENE-5376: show how to do distance drill down with lucene server; clean up some tests

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1559973 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1559973

LUCENE-5376: add search timeout (TimeLimitingCollector)

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1559984 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1559984

LUCENE-5376: finish the simple distance range facets example; fix pre-existing bug when range facet filters are used with drill sideways

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1560946 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1560946

LUCENE-5376: improve comments / nocommits

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1561524 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1561524

LUCENE-5376: add some nocommits; cutover another pretty-print from jackson to json-smart; small simplification to replication

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1561529 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1561529

LUCENE-5376: relax type DocsEnum -> DocIdSetIterator; fix typos

asfimport commented 10 years ago

Arcadius Ahouansou (@arcadius) (migrated from JIRA)

Hello.

I have checked out this branch and did in the lucene directory an ant clean package-zip The build was successful and many artefacts were created including:

I dropped the war into a fresh jetty 9 install and jetty was not happy (see stacktrace below).

My questions is:

Thanks.

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:297)
        at org.eclipse.jetty.start.Main.start(Main.java:724)
        at org.eclipse.jetty.start.Main.main(Main.java:103)
        2014-01-27 22:21:36.288:WARN:lucene-xml-query-demo:main: unavailable
        javax.servlet.UnavailableException: org.apache.lucene.xmlparser.webdemo.FormBasedXmlQueryDemo
        at org.eclipse.jetty.servlet.BaseHolder.doStart(BaseHolder.java:102)
        at org.eclipse.jetty.servlet.ServletHolder.doStart(ServletHolder.java:294)
asfimport commented 10 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Hi Arcadius, the server is currently not a webapp; you don't drop it into any web container to run it.

Instead, its a standalone app, using Netty for the HTTP interface; run it like this:

    java -cp ... org.apache.lucene.server.Server -port NNNN -stateDir /path/to/dir

You'll need to put a bunch of Lucene's JARs on the classpath since the server exposes most of Lucene's modules (grouping, joins, analysis, highlighter, facets, queries, queryparser, suggest, expressions), as well as the jars under lucene/server/lib after you've run "ant resolve".

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1562403 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1562403

LUCENE-5376: sync trunk

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1563180 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1563180

6478,LUCENE-5376: get MultiTermQuery highlighting working; fix compilation errors

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1563248 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1563248

LUCENE-5376: merge trunk

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1564769 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1564769

LUCENE-5376: minor cleanups in replication

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1565184 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1565184

LUCENE-5376: merge trunk

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1565325 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1565325

LUCENE-5376: add some more test cases for dynamic expressions; clean up test code

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1566454 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1566454

LUCENE-5376: sync trunk

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1572063 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1572063

LUCENE-5376: sync trunk

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1572637 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1572637

LUCENE-5376: add factory for SuggestStopFilter; get PostingsHighlighter MTQ highlighting working with block join queries; fix 0.0 score from block join group parent; add explicit label faceting; fix analyzing infix suggester highlighting; allow drill-downs on range facets

asfimport commented 10 years ago

Chris Male (migrated from JIRA)

Hey Mike,

What's the endzone here? Any thoughts on it coming back into trunk?

asfimport commented 10 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

What's the endzone here? Any thoughts on it coming back into trunk?

Yes, I'd like to land it in trunk / 4.x, but it's probably going to be a while still ... lots of nocommits, though probably most could be downgraded to TODOs.

The server works well (it's running at http://jirasearch.mikemccandless.com), and is a nice showcase of many Lucene modules (faceting, expressions, joins, grouping, highlighting, etc.). It's also helped uncover a number of "rough edges" in Lucene...

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1573003 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1573003

LUCENE-5376: merge trunk

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1573080 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1573080

LUCENE-5376, #6540: add near-real-time suggest updates when using AnalyzingInfixSuggester to Lucene demo server

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1573227 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1573227

LUCENE-5376: expose set/getCommitUserData

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1578132 from @mikemccand in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1578132

LUCENE-5376: make new branch

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1578133 from @mikemccand in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1578133

LUCENE-5376: carry over last branch

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1578144 from @mikemccand in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1578144

LUCENE-5376: merge trunk

asfimport commented 10 years ago

Chris Male (migrated from JIRA)

What's motivated the new branch?

asfimport commented 10 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

What's motivated the new branch?

Oh, sorry, something went wrong with the merge props on the old branch, such that when I tried to merge as I always do ("svn merge ../trunk") it hit strange conflicts in files never changed on the branch and then stopped merging and asked me to resolve the conflicts and re-run the merge, which caused further conflicts in files that shouldn't have conflicted...

I figured it was easier to just rebranch.

asfimport commented 10 years ago

Chris Male (migrated from JIRA)

Sweet!

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1579299 from @mikemccand in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1579299

LUCENE-5376: add missing cause for some exceptions

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1591283 from jdyer@apache.org in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1591283

LUCENE-5376: add "lib" directory to svn:ignore

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1591292 from jdyer@apache.org in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1591292

LUCENE-5376: Use a default 'stateDir' instead of throwing NPE

asfimport commented 10 years ago

James Dyer (@jdyer1) (migrated from JIRA)

Just playing around with this a little bit. I'm running into some barriers for a quick start, wonder if we could...

I'd be happy to try and add this but wasn't sure if the intent was to make this strictly a server-to-server app, or also something users could play around with interactively with a browser.

asfimport commented 10 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Thanks James!

If user doesn't specify a handler, or an invalid one, return a list of valid handlers, possibly with a little param documentation instead of the current IllegalArgumentException

+1: it's really important that all errors that come back from the server are transparent/clear as possible. Maybe point to the live docs handler?

Allow commands like "createIndex" be execute as GET and not require POST. Maybe let users pass parameters on the URL and not always expect an incoming JSON document? And/or have a parameter "json" for such a document?

I think this makes sense as long as the GET API is just a mirror of what you could do via JSON? E.g., maybe take all CGI args, turn into the corresponding JSON struct, and pretend that JSON had arrived via POST? Something like that ...

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1591762 from jdyer@apache.org in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1591762

LUCENE-5376: add HelpHandler to aid new users and provide quick reference

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1591764 from jdyer@apache.org in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1591764

LUCENE-5376: HelpHandler fixes

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1591984 from jdyer@apache.org in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1591984

LUCENE-5376: HelpHandler fix for incoming parameter

asfimport commented 10 years ago

ASF subversion and git services (migrated from JIRA)

Commit 1591986 from jdyer@apache.org in branch 'dev/branches/lucene5376_2' https://svn.apache.org/r1591986

LUCENE-5376: convert GET parameters to JSON

asfimport commented 9 years ago

Vincent Murphy (migrated from JIRA)

Just looking for the code? https://github.com/apache/lucene-solr/tree/lucene5376_2

asfimport commented 8 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

I've moved this "thin wrapper Lucene server to its own github project: https://github.com/mikemccand/luceneserver