Closed asfimport closed 8 years ago
Michael McCandless (@mikemccand) (migrated from JIRA)
I'm attaching the current sources (tgz archive)... they are standalone now but to add it into Lucene I think we should put it under lucene/demo or lucene/server or something.
It uses "custom" (Python) build scripts, because I became frustrated with ant; after extracting, python3 build.py test
should run the tests.
These are just the sources for the server side of the http://jirasearch.mikemccandless.com app.
There are many issues to fix, e.g. cut back to ant (there are some old ant build scripts there), use only one JSON parser (it uses two now), but it does support a number of basic indexing/search APIs: add/update document/s, bulk add/update documents, suggest, search/After, block joins, highlighting, live field values, snapshots, basic index statistics (for diagnostics).
It has limited support for "plugins", but I'm tempted to remove that before committing. The only plugin it has now is Tika, to crack binary documents into text for indexing.
Han Jiang (@sleepsort) (migrated from JIRA)
+1, it will be great to have an 'active' demo to show the features :)
I think we should remove those hardcoded classpaths, e.g. in post.py:30?
And will this demo be expected to be the same as jirasearch? Will we need further configuration to get the demo webside working? For example I cannot find search.py in the sourcecodes.
Michael McCandless (@mikemccand) (migrated from JIRA)
Thanks Han.
I think we should remove those hardcoded classpaths, e.g. in post.py:30?
Good catch, I'll fix that ... that's a minimal example of how to issue commands to the server to create an index and register a few fields, from a Python client.
And will this demo be expected to be the same as jirasearch? Will we need further configuration to get the demo webside working? For example I cannot find search.py in the sourcecodes.
These sources are just for the server side; I didn't include the jirasearch UI/indexing sources. But I agree it would be useful to have that too, i.e. an example search app/UI that runs against this server. I'll think about how to fold it in ...
Yonik Seeley (@yonik) (migrated from JIRA)
I think there are plenty of lucene-based search servers already in existence... We shouldn't bloat lucene/solr even further by adding yet another. Something like this belongs as a separate project (collaborate on github with whoever else wants to build/maintain this).
ASF subversion and git services (migrated from JIRA)
Commit 1553272 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553272
LUCENE-5376: mike's tgz as-is to branch
ASF subversion and git services (migrated from JIRA)
Commit 1553275 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553275
LUCENE-5376: fixup the ant/ivy a bit
ASF subversion and git services (migrated from JIRA)
Commit 1553282 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553282
LUCENE-5376: fix compile
ASF subversion and git services (migrated from JIRA)
Commit 1553283 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553283
LUCENE-5376: add svn:ignore
ASF subversion and git services (migrated from JIRA)
Commit 1553287 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553287
LUCENE-5376: get tests passing
ASF subversion and git services (migrated from JIRA)
Commit 1553288 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553288
LUCENE-5376: use consistent dependencies versions, dont download the internet
ASF subversion and git services (migrated from JIRA)
Commit 1553289 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553289
LUCENE-5376: clean up some forbidden apis
ASF subversion and git services (migrated from JIRA)
Commit 1553291 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553291
LUCENE-5376: get ant javados passing
ASF subversion and git services (migrated from JIRA)
Commit 1553294 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553294
LUCENE-5376: fix documentation-lint, rat-sources, remove python build, remove plugin (it can be a separate module?)
ASF subversion and git services (migrated from JIRA)
Commit 1553297 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553297
LUCENE-5376: minimize jdocs deps, nuke unused imports
ASF subversion and git services (migrated from JIRA)
Commit 1553300 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553300
LUCENE-5376: clean up forbidden system outs / logging
ASF subversion and git services (migrated from JIRA)
Commit 1553301 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553301
LUCENE-5376: remove code duplication
Shai Erera (@shaie) (migrated from JIRA)
I think we should put it under lucene/demo or lucene/server or something.
I think we should put it under lucene/server. We can have the JIRA search example (or a simpler one) under demo if we really want to, but I think a LuceneServer is a good component by itself. First, we never know what it will turn out into, what APIs will be developed etc. Also, it would be nice to see clean examples of distributed search (simple, facet, spatial) as well as suggest (maybe even distributed suggest). So +1 for adding it as a new module.
Areek Zillur (@areek) (migrated from JIRA)
This is a good idea. I was thinking of having a suggest demo in the lucene demo package! But this would be a good place to put it. I will look into the suggest stuff after a week or so.
ASF subversion and git services (migrated from JIRA)
Commit 1553379 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553379
LUCENE-5376: try to address nocommit...
ASF subversion and git services (migrated from JIRA)
Commit 1553384 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553384
LUCENE-5376: remove crazy use of jackson for pretty-printing. jackson only used in the indexing api now
Robert Muir (@rmuir) (migrated from JIRA)
I created 'lucene5376' branch (https://svn.apache.org/repos/asf/lucene/dev/branches/lucene5376) with the patch and cleaned up the build and so on.
notes/questions:
Michael McCandless (@mikemccand) (migrated from JIRA)
i removed the tika plugin (for now). is this ok? we can make it a separate module?
That's fine ... let's leave it for later?
which json parser to keep? jackson is only used rarely, i assume this is the one to remove? We just have to change the indexing code now to remove those jars.
I had to add jackson for streaming / incremental parsing for the bulk APIs ... I think (not sure!) that we should just move to jackson for everything, but I haven't started that yet. I'll look into it.
forbidden-apis was angry about the SOPs. i tried to figure out which ones were intentional and which ones were not. you can see the list of files exempted in build.xml.
Thanks for fixing!
javadocs-lint is working but angry because mostly all javadocs (package.html's, class docs) are missing. this seems to need some help.
I'll work on this.
TestPlugins is disabled, but i think we can fix it if we build a .zip on-the-fly, pulling in MockPlugin.class and other files with .getResourceAsStream? Uwe will love this.
That sounds great!
Thanks Rob.
Robert Muir (@rmuir) (migrated from JIRA)
I had to add jackson for streaming / incremental parsing for the bulk APIs ... I think (not sure!) that we should just move to jackson for everything, but I haven't started that yet. I'll look into it.
OK, I may have sent this in the wrong direction with my latest commit, so just back it out if you want to move to jackson. I assumed the opposite, since jackson was only used in a few places. I will say I do prefer the simpler underengineered api of the json-smart vs the ... no comment... jackson, even if it has less features. Reminds me of HttpURLConnection vs httpclient.
ASF subversion and git services (migrated from JIRA)
Commit 1553403 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553403
LUCENE-5376: fix and re-enable test
ASF subversion and git services (migrated from JIRA)
Commit 1553477 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553477
LUCENE-5376: javadocs
ASF subversion and git services (migrated from JIRA)
Commit 1553486 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553486
LUCENE-5376: javadocs fixes
ASF subversion and git services (migrated from JIRA)
Commit 1553499 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553499
LUCENE-5376: move the hack wtf into one place
ASF subversion and git services (migrated from JIRA)
Commit 1553500 from @rmuir in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553500
LUCENE-5376: don't create and destroy dirs in the CWD
ASF subversion and git services (migrated from JIRA)
Commit 1553670 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553670
LUCENE-5376: javadocs
ASF subversion and git services (migrated from JIRA)
Commit 1553821 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1553821
LUCENE-5376: javadocs
ASF subversion and git services (migrated from JIRA)
Commit 1554012 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1554012
LUCENE-5376: documentation-lint finally passes (method level)
ASF subversion and git services (migrated from JIRA)
Commit 1554013 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1554013
LUCENE-5376: simplify sync'd code
ASF subversion and git services (migrated from JIRA)
Commit 1554020 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1554020
LUCENE-5376: clean up / document addDocument code
ASF subversion and git services (migrated from JIRA)
Commit 1554022 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1554022
LUCENE-5376: rename singleValued -> multiValued
ASF subversion and git services (migrated from JIRA)
Commit 1554207 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1554207
LUCENE-5376, #6315: expose DocumentDictionary (to build suggestor from stored documents) in demo server
ASF subversion and git services (migrated from JIRA)
Commit 1554409 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1554409
ASF subversion and git services (migrated from JIRA)
Commit 1555341 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1555341
LUCENE-5376: test sugar: insert the current index name to outgoing requests to make writing tests easier
ASF subversion and git services (migrated from JIRA)
Commit 1555629 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1555629
LUCENE-5376: sync trunk changes, cutover to new facets APIs, simplify search handler
ASF subversion and git services (migrated from JIRA)
Commit 1555726 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1555726
LUCENE-5376: consolidate logic to pull the searcher from indexGen, version, snapshot or current
ASF subversion and git services (migrated from JIRA)
Commit 1556045 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556045
ASF subversion and git services (migrated from JIRA)
Commit 1556508 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556508
ASF subversion and git services (migrated from JIRA)
Commit 1556546 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556546
LUCENE-5376: turn on scoring when sorting by field if any of the sort fields or retrieved fields require scores, e.g. when they are an expression field that uses _score
ASF subversion and git services (migrated from JIRA)
Commit 1556555 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556555
LUCENE-5376: don't need to make ScoreValueSource public
ASF subversion and git services (migrated from JIRA)
Commit 1556564 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556564
LUCENE-5376: add another expression test case; add nocommit for bcp47 cutover
ASF subversion and git services (migrated from JIRA)
Commit 1556620 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556620
LUCENE-5376: also allow dynamic expression per-request
ASF subversion and git services (migrated from JIRA)
Commit 1556627 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556627
LUCENE-5376: remove recency blending hack: just use expressions instead
ASF subversion and git services (migrated from JIRA)
Commit 1556775 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556775
LUCENE-5376: allow setting norms format, including compressed norms
ASF subversion and git services (migrated from JIRA)
Commit 1556786 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1556786
ASF subversion and git services (migrated from JIRA)
Commit 1557073 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1557073
ASF subversion and git services (migrated from JIRA)
Commit 1557517 from @mikemccand in branch 'dev/branches/lucene5376' https://svn.apache.org/r1557517
LUCENE-5376: expose control over underlying facet index field, fix some nocommits
I think it'd be useful to have a "demo" search server for Lucene.
Rather than being fully featured, like Solr, it would be minimal, just wrapping the existing Lucene modules to show how you can make use of these features in a server setting.
The purpose is to demonstrate how one can build a minimal search server on top of APIs like SearchManager, SearcherLifetimeManager, etc.
This is also useful for finding rough edges / issues in Lucene's APIs that make building a server unnecessarily hard.
I don't think it should have back compatibility promises (except Lucene's index back compatibility), so it's free to improve as Lucene's APIs change.
As a starting point, I'll post what I built for the "eating your own dog food" search app for Lucene's & Solr's jira issues http://jirasearch.mikemccandless.com (blog: http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html ). It uses Netty to expose basic indexing & searching APIs via JSON, but it's very rough (lots nocommits).
Migrated from LUCENE-5376 by Michael McCandless (@mikemccand), 1 vote, resolved Jul 29 2016 Attachments: lucene-demo-server.tgz