Make Luke a Lucene/Solr Module [LUCENE-2562]

[asfimport]

see "RE: Luke - in need of maintainer": http://markmail.org/message/m4gsto7giltvrpuf "Web-based Luke": http://markmail.org/message/4xwps7p7ifltme5q

I think it would be great if there was a version of Luke that always worked with trunk - and it would also be great if it was easier to match Luke jars with Lucene versions.

While I'd like to get GWT Luke into the mix as well, I think the easiest starting point is to straight port Luke to another UI toolkit before abstracting out DTO objects that both GWT Luke and Pivot Luke could share.

I've started slowly converting Luke's use of thinlet to Apache Pivot. I haven't/don't have a lot of time for this at the moment, but I've plugged away here and there over the past work or two. There is still a lot to do.

---

Migrated from LUCENE-2562 by Mark Miller (@markrmiller), 11 votes, resolved Apr 24 2019 Attachments: LUCENE-2562.patch (versions: 3), LUCENE-2562-ivy.patch, LUCENE-2562-Ivy.patch (versions: 3), luke1.jpg, luke2.jpg, luke3.jpg, Luke-ALE-1.png, Luke-ALE-2.png, Luke-ALE-3.png, Luke-ALE-4.png, Luke-ALE-5.png, lukeALE-documents.png, luke-javafx1.png, luke-javafx2.png, luke-javafx3.png, screenshot-1.png, スクリーンショット 2018-11-05 9.19.47.png Linked issues:

• LEGAL-396

Pull requests: https://github.com/apache/lucene-solr/pull/420, https://github.com/apache/lucene-solr/pull/490, https://github.com/apache/lucene-solr/pull/512

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Re: Apache Pivot

So far its been a nice experience - the main draw back to it from my perspective is that it heavily favors fixed sized layouts. I really don't like this - I like fluid, resizing layouts. So far I haven't been able to make that happen, but its early yet - though it may be more difficult than worth it short term. For now I am just targeting a decent fixed size while the bulk of the grunt work is being done.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I haven't had any time to really work on this in a while, but I did bite the bullet and join the pivot mailing list and figured out my issues with making a fluid resizing layout - which is sweet and will hopefully motivate me to make some progress here soon.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

So, there is really a lot to do here - but I have gotten a fair amount done.

I'm near the point that I would like to share something - mostly so that I can perhaps get some help doing more.

I think we can start the module well before this version of Luke can do everything the current version of Luke can - but I'd like a spot to share something that's not even really ready for a module spot. I'm looking for a shared scratch pad to use to get to a rough module point.

Anyone mind if I make a spot in svn? lucene/luke, lucene/sandbox/luke or something?

I'll pull any of the thinlet stuff out first. We may need to consider other issues like code grants, but I think we can at least have a contained starting point in svn before we get those details resolved? Thoughts? Otherwise I'll need to share on an svn outside apache, which may cause the need for further code grants issues, etc (developing outside of apache) that I'd rather just avoid. But either way.

Also, if anyone is into translating, I'm working on localization from the ground up, as well as modularizing and cleaning up a lot of the code. Luke in German anyone?

[asfimport]

Ryan McKinley (@ryantxu) (migrated from JIRA)

Anyone mind if I make a spot in svn? lucene/luke, lucene/sandbox/luke or something?

+1 for lucene/sandbox/luke

We can always move it later

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

Anyone mind if I make a spot in svn? lucene/luke, lucene/sandbox/luke or something?

I just wonder why you don't branch and put it already in the right location. I think we should use branches more often though.

Also, if anyone is into translating, I'm working on localization from the ground up, as well as modularizing and cleaning up a lot of the code. Luke in German anyone?

Heh, sure I will do the German translation. - robert what about Thai :D

[asfimport]

Andrzej Bialecki (@sigram) (migrated from JIRA)

+1. I'm happy to do a code grant if people think it's necessary - after all, the non-thinlet parts are already licensed under ASL.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I just wonder why you don't branch and put it already in the right location. I think we should use branches more often though

Because I really have no use/want for a branch of lucene/solr - luke is currently completely external to that, so what's the point really? When it's in module form, perhaps that would make more sense, but then I'm prob inclined to just put it in trunk and iterate.

I'm happy to do a code grant if people think it's necessary - after all, the non-thinlet parts are already licensed under ASL.

I think the rule is that if it was developed outside of Apache, they want the grant, regardless of the license - especially for a codebase so large. I've heard different things when the dev is a committer - but they tend to err on the side of caution. We will see what Grant says.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I've checked in the current state at: https://svn.apache.org/repos/asf/lucene/sandbox/luke

The main class is currently: org.apache.lucene.luke.ui.LukeApplication

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

If you just want to check out the current state of things, you can launch a jnlp version at http://www.lucene-solr.com/luke.jnlp

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I've taken Mark Harwood's cool Analyzers Tool plugin and ported it over as a first class citizen with it's own tab.

Great tool for checking out analysis - I'd hadn't really paid much attention to it till Robert pointed it out.

Check it out - I've got it in the latest jnlp at http://www.lucene-solr.com/luke.jnlp

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Would like to get going on pushing this in as a module - anyone want to comment on the legal aspect? Do we need a code grant or not? The original authors (AB and a couple plugin authors?) are committers, but the code was original developed outside Apache...

Anyone want to weigh in so we can move forward on module integration? Robert says he might lend a hand there.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

(P.S. Not to say this is that close to being done - would just like to get the legal stuff out of the way before spreading around too much in svn)

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

Would like to get going on pushing this in as a module - anyone want to comment on the legal aspect? Do we need a code grant or not? The original authors (AB and a couple plugin authors?) are committers, but the code was original developed outside Apache...

I am not a legal expert but shooting an email to legal-discuss would bring clarification and we are on the safe side. Even if we would need one since all are around that should not be a big deal.

simon

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

How many contributors have there been to Luke outside of the ASF? I know Andrzej and Mark. I know I have put up a patch, I think. Was explicit permission given to use? I guess it comes down to ownership. I suppose you could split out where those patches were applied to, as well. Given that we are dropping the Thinlet stuff, anything that was changed there doesn't matter.

[asfimport]

Mark Harwood (@markharwood) (migrated from JIRA)

For the record, I'm obviously happy to grant any of my Luke contributions to Apache.

BTW - If anyone is looking at a revision of the GWT front end then this recent Google acquisition looks like it might be interesting: http://code.google.com/webtoolkit/tools/gwtdesigner/index.html

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

shooting an email to legal-discuss

I'll go to legal discuss if I have to - but we have dealt with this type of thing before - so if the right people respond, I'd rather just go that route.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I'll go to legal discuss if I have to - but we have dealt with this type of thing before - so if the right people respond, I'd rather just go that route.

What's happening with this one? It would be great to get it tied into the build in trunk.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Still going - just kind of a weekend, nights project, of which I've been short of recently. Last weekend, 3 day hiking camping trip, previous 2 weekends before that (and all days in between) in Boston. Looking a little wavy going forward too, but I'm still heavily invested.

If Grant is not worried about getting a code grant, than neither am I. IMO, since the contributors are committers with CLA's, there is not much to worry about. Since unless someone says differently, I'm inclined to just move this (it's already in svn, so moving it under trunk is not much different).

If someone has a concern, we can hit legal-discuss though.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Still going - just kind of a weekend, nights project, of which I've been short of recently. Last weekend, 3 day hiking camping trip, previous 2 weekends before that (and all days in between) in Boston. Looking a little wavy going forward too, but I'm still heavily invested.

Right, i meant really any legal issue/moving to trunk. Though maybe it doesnt have the complete feature set yet, in trunk it would stay current and what you have so far is useful.

Since unless someone says differently, I'm inclined to just move this (it's already in svn, so moving it under trunk is not much different).

+1

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

mark, I marked this as gsoc2011 so if somebody is interested in working on this during GSoC can apply for it. Hope that is ok for you though.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Yeah, that would be great!

Either way, I have def not abandoned this - just don't know when it will get more love...

[asfimport]

Ryan McKinley (@ryantxu) (migrated from JIRA)

Any update on where this is?

I saw something working reasonably well before: http://www.lucene-solr.com/luke.jnlp

What about putting the almost working code in /trunk/modules/luke? And adding proper warnings for it being in progress

We can iterate from there, but it would be awesome to have a version of luke that works with trunk!

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

http://www.lucene-solr.com/luke.jnlp

whoops - was not paying attention, and looks like I let that domain expire. 100 bucks to redeem it - I think I'll let her go for now :)

This version is in svn somewhere under sandbox or something. Robert has threatened to move it to a module while we where at revolution :)

[asfimport]

Simon Willnauer (@s1monw) (migrated from JIRA)

This version is in svn somewhere under sandbox or something. Robert has threatened to move it to a module while we where at revolution

+1

[asfimport]

Shai Erera (@shaie) (migrated from JIRA)

Where does this issue stand? I noticed a thread on the user list today, where people were practically begging for a Luke version that works with recent releases. Is there consensus to have Luke as a Lucene module? If so, is it just a legal thing, or a "forgotten issue"?

[asfimport]

Shai Erera (@shaie) (migrated from JIRA)

If people don't have time to work on this, perhaps still someone should put up a patch that doesn't compile of all Luke's code? I will be happy to take a look, and I'm sure others will too.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I did all the initial work on it. I haven't had even an ounce of time I could put towards it for a long time now. I had a fair amount of functionality working nicely. It was a fun little project, there are no legal things, and hopefully some day it's completed by someone :)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I think Mark's latest state is committed to a branch at https://svn.apache.org/repos/asf/lucene/sandbox/luke/ ?

Would love to see this revived as an actual module of lucene.

[asfimport]

Shai Erera (@shaie) (migrated from JIRA)

Ah I missed that comment! indeed there's a version under sandbox/luke.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

The work was really not that difficult. More just marching through each piece and updating to some more recent apis. I've got so many other things on my plate, even though I enjoyed this work, it's just super low priority. But I could probably help point someone in the right direction if they wanted to get started - perhaps some day I'd even help out with some more work on it when I don't feel so buried.

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1502987 from @markrmiller https://svn.apache.org/r1502987

LUCENE-2562: hack and slash update to Lucene/Solr 4.3.1

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

I've done a quick hack and slash update to Lucene/Solr 4.3 and uploaded some screen shots of the latest state of this.

To run it, simply check out https://svn.apache.org/repos/asf/lucene/sandbox/luke and run the main class org.apache.lucene.luke.ui.LukeApplication

Currently, it's a full eclipse project so there is no build step or ant script or anything.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

It's been a long time, but the last time I talked to @rmuir about this, I think we came to the conclusion that the best way forward would be to tie up these basic features and perhaps stop worrying about porting over anything else from GetOpt Luke - like the plugins for instance. This already has the majority of Luke functionality implemented - it might make sense to simply polish it off and button it up, get it in as a module and then lets new features and development happen organically.

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1502989 from @markrmiller https://svn.apache.org/r1502989

LUCENE-2562: Update build file

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1503004 from @markrmiller https://svn.apache.org/r1503004

LUCENE-2562: Update to Apache Pivot 2.0.2

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1516057 from @markrmiller https://svn.apache.org/r1516057

LUCENE-2562: update to Lucene/Solr 4.4 and Pivot 2.0.2

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

At Ajay Bhat's request, I have updated to Lucene/Solr 4.4 and Pivot 2.0.2.

Ajay Bhat, feel free to comment on this JIRA issue regarding your current progress.

Also, it would be great to commit what you have so far when you are ready.

[asfimport]

Ajay Bhat (migrated from JIRA)

Re:[\~markrmiller@gmail.com] and everyone else,

I've done a very small patch, but I'm still facing some problems.

I've checked the Analyzers in the tool manually, and these Analyzers give the java.lang.InstantiationException when used.

org.apache.lucene.analysis.miscellaneous.LimitTokenCountAnalyzer org.apache.lucene.analysis.miscellaneous.PatternAnalyzer org.apache.lucene.analysis.query.QueryAutoStopWordAnalyzer org.apache.lucene.analysis.snowball.SnowballAnalyzer org.apache.lucene.collation.CollationKeyAnalyzer org.apache.lucene.solr.analysis.TokenizerChain

I'm a bit confused because as of Lucene 4.0alpha, the PatternAnalyzer is deprecated from misc package, but it's still present here in miscellaneous package in Lucene 4.4 release.

[asfimport]

Erick Erickson (@ErickErickson) (migrated from JIRA)

bq: I'm a bit confused because as of Lucene 4.0alpha, the PatternAnalyzer is deprecated from misc package, but it's still present here in miscellaneous package in Lucene 4.4 release.

Deprecated code is maintained through one major version. So it'll stay marked as deprecate through all of 4.x and be eligible to be removed in 5.0. That way people have time to adjust...

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Ajay Bhat, can you explain the patch? It seems to only add a tokenstream reset call? Is that the only change so far?

I'll take a look at the analyzer init issue as soon as I get a chance.

[asfimport]

Ajay Bhat (migrated from JIRA)

The TokenStream reset call was needed to display the tokens generated by the Analyzer. I think that's the only change that was required. The main problem for me is the analyzers above are not giving the result, which I've been looking into. I had figured that since PatternAnalyzer was deprecated it would not give the result and so it might be a good idea to remove it from the list of analyzers. But there are also some analyzers that aren't deprecated, like the Snowball Analyzer and QueryAutoStopWordAnalyzer.

Also, as per the schedule of my proposal I've done some work on the themes of the Application. I'll contribute another patch for that soon.

[asfimport]

Ajay Bhat (migrated from JIRA)

LUCENE-2562.patch Date : 13th Sept, 2013 Added: Support for 5 themes, through a recursive style change function Exit option in File menu Status bar Analyzer tokenstream reset call Documentation for above features

Pending Work to be done: Status Bar implementation not complete Themes cannot be applied to one tab

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Nice, thanks Ajay!

Unfortunately, I am having trouble with the patch - it won't cleanly apply to AnalyzersTab.java. I left that file as is and checked out the color scheming though - looks good!

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

One comment - I think I like the gray theme for the default best.

Once I can apply a clean patch, I'll commit your current progress.

[asfimport]

Ajay Bhat (migrated from JIRA)

Thanks for the comments, @markrmiller. The only change for AnalyzersTab.java is the tokenstream reset call.

I used the default theme as per the one used in original Thinlet Luke. If you'd like, I'll do a slight modification so Gray is the default theme.

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1523019 from @markrmiller https://svn.apache.org/r1523019

LUCENE-2562: Ajay Bhat

Support for 5 themes, through a recursive style change function Exit option in File menu Status bar Analyzer tokenstream reset call Documentation for above features

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

FYI, I see a lot of the following type thing in the console:

"borderColor" is not a valid style for org.apache.pivot.wtk.TextArea "activeBackgroundColor" is not a valid style for org.apache.pivot.wtk.TextArea

Duplicate listener org.apache.lucene.luke.ui.LukeWindow$2@4a9a1ac added to org.apache.pivot.wtk.Component$ComponentMouseListenerList [org.apache.pivot.wtk.skin.terra.TerraPushButtonSkin@451dfada, org.apache.lucene.luke.ui.LukeWindow$2@4a9a1ac]

Perhaps that is currently expected, but FYI.

[asfimport]

Ajay Bhat (migrated from JIRA)

Re: ["borderColor" is not a valid style for org.apache.pivot.wtk.TextArea "activeBackgroundColor" is not a valid style for org.apache.pivot.wtk.TextArea]

Duly noted. I'll make sure to look into it and take care of it in the next patch.

[asfimport]

Tomoko Uchida (@mocobeta) (migrated from JIRA)

Hi, is this issue still active?

I've attached a patch (LUCENE-2562-ivy.patch) to follow latest Lucene updates. This patche does

• add ant task "ivy-resolve" and "ivy-clean" to resolve dependencies using Ivy "ivy-resolve" task creates lib-ivy directory and download necessary libraries from maven repository.
• modify "compile", "dist", and other related tasks use libraries downloaded by "ivy-resolve"
• make small source code changes to go with Lucene 4.10.3

lib/*.jar, lib/solr/*.jar, lib/hadopp/\*.jar are no longer used for compile and distribute, so they can be removed from the repository later, except for lib/js.jar (I could not identify this.) Note: Eclipse's .classpath file is not changed.

Compiled o.a.l.luke.ui.LukeApplication can browse indices generated by Lucene 4.10.3.

If you're interested, I hope to take more time for the project.

[asfimport]

Mark Miller (@markrmiller) (migrated from JIRA)

Hey @Tomoko Uchida - I still think this is an important issue, I just don't have the time to drive it currently.

If you have the desire to do some work on it, I'm sure I can ramp up enough to apply some patches for you.

I'll try and take a look at the attached patch before long.