need to document/fix/remove build.xml dependencies on javascript [LUCENE-4415]

[asfimport]

Radim Kolar pointed out on the solr-user mailing list that he was unable to build Solr on his FreeBSD machine due to the use of <script language="javascript"... /> tags in the ant build file, and the lack of javascript by default in his Java install. There was no clear indication that he should need anything else.

It seems like prior to releasing 4.0, we should do one of the following:

1) find a way to remove the dependency on using javascript in the build.xml files (ie: our own custom ant target written in java?) 2) use ivy to ensure we have the rhino jar in a classpath we pass to <script> tag (it appears we already do something similar for pegdown) 3) update lucene/BUILD.txt and solr/README.txt to make it clear that the rhino js.jar is a build requirement (and link to a wiki somewhere that provides tips on how to install it in ant if your JVM doesn't include it)

(Assigning to Uwe in hopes that he has some inspiration, i know he's familiar with this scripting deficiency on FreeBSD)

https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201209.mbox/%3C505CF59E.4000600%40filez.com%3E

---

Migrated from LUCENE-4415 by Chris M. Hostetter (@hossman), resolved Sep 22 2012 Attachments: LUCENE-4415.patch, LUCENE-4415-option2.patch (versions: 2)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I don't agree with blocker. The important part of this email is:

ant create-package fails with:

This is different from 'unable to build Solr'. This is creating your own 'full binary release'.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

regardless of what target was mentioned in the original email, unless i'm missing something it means you can't build javadocs from local source - that seems worthy of a blocker to me.

I'm fine with a quick fix of adding a line to the build requirements, but not being able to build javadocs seems like a really big deal.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Building javadocs is different than building 'create-package': 'create-package' also runs the pegdown generation for documentation (which has had javascript for some time, at least 4.0-ALPHA/BETA).

I am fine with either:

• adding line to requirements
• pulling whatever this jar is via ivy

I think it should stay implemented as javascript though, because its too frustrating to deal with the possible Windows issues involved with this URL stuff.

In all cases we should disable the 'create-package' target as its untested by itself, it may not even work regardless, as it is never tested by jenkins. (i opened #5482 for this)

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

attaching my really amateurish attempt at option#2 - using ivy to fetch js.jar.

I'm not even sure how to test that it's working, since im not sure how to realisticly reproduce the original problem (removing lib/rhino.jar from my openjdk install didn't cause an "Unable to create ..." error from any, it just caused a hard and ugly NoClassDefFoundError from some com.sun packages)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I'm not even sure how to test that it's working

we can probably test from freebsd, but in my opinion the only way to go here is to remove the js.jar from our jenkins' servers JDK installation, so we KNOW it works.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

There is also some solr test that depends on this javascript dependency, too.

Its dependencies should be fixed in the same way.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

as far as i know, all of hte solr tests that depending on the javascript ScriptEngine use "assume()" to verify that it's available – if there are some missing that assume() it's easy to add.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

but in my opinion the only way to go here is to

remove the js.jar from our jenkins' servers JDK installation, so we KNOW it works.

this is why i was hoping to get uwe's sign off – i know he's looked into this in depth before, and decided it made sense to add it to the FreeBSD jenkins machine instead of changing the build – IIRC he said it was considered a "bug" in the FreeBSD java packaging and so us ading hte jar as "ok" because that's what all users should do too – but i can't remember for certain. if that is really the case: then we should just resolve with documentation, if not: we should fix out build.xml files once and for all not to assume it will always exist.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

right, but we would actually prefer those tests execute, correct?

and what is the test doing? is it just testing an arbitrary scripting language to show some scripting integration works? or is there actual solr code that depends upon this functionality?

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

IIRC he said it was considered a "bug" in the FreeBSD java packaging and so us ading hte jar as "ok" because that's what all users should do too

I'm not sure about this. What about other Java implementations other than Oracle/Sun? Are they guaranteed to ship with a javascript engine? I really like the ivy idea from this perspective.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

and what is the test doing? is it just testing an arbitrary scripting language to show some scripting integration works? or is there actual solr code that depends upon this functionality?

Both? ... i'm confused by the two options in your question – Solr has functionality that lets users define scripts to control. we have tests for this functionality that use javascript scripts to prove that the fnctionality works (ie: that the script affects the behavior)

Yes we want the tests to run, but if Bob doesn't have support for that scriptengine in his JVM, there is no other way to re-write the test so that Bob could test it, and Bob couldn't use that functionality in a real Solr instance anyway.

I'm not sure about this. What about other Java implementations other than Oracle/Sun? Are they guaranteed to ship with a javascript engine? I really like the ivy idea from this perspective.

I have all the same questions. I think Uwe said somewhere/sometime that "the spec" required javascript be provided, hence FreeBSD was broken so he added the jar to jenkins – but he may have retracted that comment at a later date (i certainly never found anything to back it up)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Both? ... i'm confused by the two options in your question – Solr has functionality that lets users define scripts to control. we have tests for this functionality that use javascript scripts to prove that the fnctionality works (ie: that the script affects the behavior)

I mean does Solr have functionality that specifically depends on the javascript scripting engine, or does it just support scripting engines in general and we are testing it with javascript?

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

It supports scripting engines in general and we are testing that support with javascript.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

OK I guess the ideal situation here would be to add js.jar to this solr module's test-classpath? Is that enough for the scripting integration to work so the test will run?

This way we could remove our "installation" of js.jar from our freebsd JDK, and but also keep test coverage for this scripting stuff.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

Updated patch: nows uses ivy to fetch bsf as well, and forces ant to use the BSF engine instead of the built n javax.script engine.

I'm not sure if this works "better" then my last patch, but i can say that when i butchered the rhino.jar in my openjdk setup, the build still worked find with this patch.

One thing i forgot to mention before: no idea what the licenses for BSF/rhino are, or how using them in this way affects us.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

its MPL2.0 ... maybe we should go the documentation route for 4.0 and take our time on this?

(not saying there is any problem, just it makes it a bigger scarier change IMO)

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

OK I guess the ideal situation here would be to add js.jar to this solr module's test-classpath? Is that enough for the scripting integration to work so the test will run?

On a freebsd machine? i have no idea - but either way that really seems like it should be a separate issue from this one (and shoulnd't be a blocker for 4.0)...

1) it's not related to the ant build, and i doubt you mean you would then have the ant macros use a classpath based on jars downloaded for the the solr modules test-classpath 2) it doesn't currently prevent the build from working, or tests from passing, on any known machine. as i mentioned: those tests already use assume() to skip themselves if you don't have javascript support: just like other test we already have that skip themselves if your JVM doesn't support a pre-req. 3) it's not clear to me that it's even a good idea. doing that could give people who run the tests the impression that javascript engine support will work for them in solr on their machine (because the tests will say "PASS" instead of "SKIPPED") but then in a real solr instance it won't work unless they customize the jvm or the classpath.

...but like i say: seems like we should worry about that in a distinct issue

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I brought it up on this issue because i don't want us committing anything to the ant build for this issue if we don't regularly test it in jenkins.

So that means removing javascript from our jenkins ant/lib or whatever it is, which would have the side effect of reducing test coverage for that solr test.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

On a freebsd machine? i have no idea - but either way that really seems like it should be a separate issue from this one (and shoulnd't be a blocker for 4.0)...

Will not work, as the FreeBSD VM also violates the spec, because the whole SPI framework for rhino/JavaScript is missing, too.

I am -1 to the above patch, as it introduces a highly outdated dependency to BSF, which is wrong to use with Java 6 (BSF is not javax.scripting, its something different and outdated for Java 4 or 5!!!). So adding BSF would neither help ANT nor JavaScript to use the official javax.scripting.

Oracle JDK does not ship whith Rhino, it ships with its own scripting engine (based on Rhino, but definitely not Rhino!) as an SPI. The bug with FreeBSD and using Rhino as replacement "hack" (without BSF) is only, that the stock Rhino JARs on their download page do not ship with the javax.scripting SPI. Unfortunately as far as I know, this SPI JAR (script-js.jar) is not in Maven (I don't know, never found it).

FreeBSD has a bug and we cannot work around all bugs! Every JDK out there has scripting (JavaScript), only FreeBSD JDK not. This is a violation of the spec, as the spec mandates that at least one scripting engine is shipped with JDK, shipping with no scripting engine is a bug in the JDK.

We should simply note in BUILD.txt that FreeBSD users should install a correct JDK or add the missing JARs to their ext directory.

Finally: -1 to the patch +1 to adding a note about FreeBSD to BUILD.txt (I can manage that)

[asfimport]

Radim Kolar (migrated from JIRA)

OpenJDK do not contains rhino, but its not freebsd specific bug

from http://openjdk.java.net/groups/core-libs/

JDK 6 includes source code for the Mozilla Rhino javascript engine in src/share/classes/sun/org/mozilla/javascript. The .class files for this engine are in rt.jar. The source code has been removed from OpenJDK. In order to use scripting, a JSR 223 compatible scripting engine must be made available on the CLASSPATH.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

It is FreeBSD specific: Linux distributions contain the minimal Rhino, as this is required by the Java 6 spec. They mandate the existence of a scripting engine. If you ship a binary Java distribution, you have to add a SPI like you have to ship with an XML parser (which is also not part of the OpenJDK source code, as it is Apache Licensed). The FreeBSD makefile downloads the XML parser but misses to download rhino. That's the bug, only on FreeBSD. According to the spec, FreeBSDs distribution is not Java compliant. Linux packages are gernerally correct, as they depend on IcedTea's build scripts, that download rhino, patch them with jarjar to create internal package names and add the missing SPI.

[asfimport]

Radim Kolar (migrated from JIRA)

To get it work in openjdk/Ant you need to place js.jar from rhino + script-js.jar from http://download.oracle.com/otndocs/jcp/sjp-1.0-fr-oth-JPR/ into classpath or ant/lib.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Exactly, ideally into the extra lib folder of OpenJDK directly. We have that as a fix on Apache's build machines.

We can place your comment into BUILD.txt and make ant fail with a nice message - that's all I intend to do.

Patch comes later when back at home.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Every JDK out there has scripting (JavaScript), only FreeBSD JDK not. This is a violation of the spec, as the spec mandates that at least one scripting engine is shipped with JDK, shipping with no scripting engine is a bug in the JDK.

I can make rmuir-JDK that has a scripting (Python) but not freebsd, and we will still have the problem.

The spec doesnt require javascript.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Stated anotehr way: show me in the spec where javascript is required, or retract your -1 (its invalid)

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

My -1 was related to the patch! The patch is not good, as it replaces the official Java 6 javax.scripting framework by the "outdated" BSF framework, which is a completely different implementation made for Java 4+. This patch would only work with ANT, but not Lucene/Solr internal depending on JavaScript with javax.scripting framework. I would agree to add the download of rhino.jar and the above script-js.jar from the Oracle website (http://download.oracle.com/otndocs/jcp/sjp-1.0-fr-oth-JPR/) into the bootstrap path, nothing more. Just to repat: I AM AGAINST ADDING OUTDATED BSF!

I agree that the Spec does not mandate to add "Javascript", but there must be one scripting implementation. In fact the rmuir private JDK is stupid, as a large number of projects would no longer work and this is also the reason for every-except-FreeBSD JDK out there to ship with JavaScript.

The bug is on the FreeBSD side - I will check their bugtracker and suggest to use partially IcedTea build (they already use patches from IcedTea).

By the way: I can also build a "private" thetaphi-JDK without the XERCES and XALAN XML components. It would also be correct according to the spec, but then 90% of all Java projects would no longer work. The critical mass of Javascript requiring projects is just not as big to see the impact in FreeBSD.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

By the way: I can also build a "private" thetaphi-JDK without the XERCES and XALAN XML components. It would also be correct according to the spec, but then 90% of all Java projects would no longer work. The critical mass of Javascript requiring projects is just not as big to see the impact in FreeBSD.

Not even ANT would work, LOL. But its still a conformant JDK.

[asfimport]

Radim Kolar (migrated from JIRA)

what about to use other scripting language like groovy or jython? It can be downloaded from maven repo easily.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

That is another idea, but I will not rewrite the scripts. There are actually more than these 2 Javascripts in ANT's build. The root extra-target.xml contains a very complex javascript task.

At least the following need to be rewritten: macrodef for pegdown, the javadoc.dir property setter in solr and the extra-target SVN checker one.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

I can look into rewriting the scripts to groovy, but I cannot guarantee to do that until release (as Robert wants to put pressure on me). I am definitely against jython, but groovy is easy to port.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I already stated i am against the scripts being rewritten differently for 4.0

This won't happen: I spent hours and then Uwe spent hours with the windows crap and I refuse to do that again.

I am just asking questions about the spec because I don't like that it seems we assume javascript is there, when its not really specified.

I agree the freebsd install has a bug, but that bug is NOT missing javascript. its bug could be fixed by having the port install python.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I can look into rewriting the scripts to groovy, but I cannot guarantee to do that until release (as Robert wants to put pressure on me).

I never put any pressure on you: hossman assignd this issue to you.

I am -1 to the scripts being rewritten for 4.0: this is too risky.

In fact ive moved this issue to 4.1.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Here the patch with converting those ANT scripts to GROOVY 2.0.4.

I would prefer this solution, because now the build does no longer depend on Javascript (in a Rhino-compatible variant) in the JDK. The prior scripts were not only relying on Javascrip, also rhino, as they depended on the special way, how rhino imports java packages or implement java interfaces (see svn task).

This patch is 100% invariant on the JDK vendor, only 1.6 is needed with javax.scripting support.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

I will commit this now to 4.x and trunk branch so it can bake. I hope the RM puts this into 4.0, too.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Committed to trunk revision: 1388813 Committed to 4.x revision: 1388814

Robert, it's up to you!

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

I would be happier if i knew the smoke-tester was passing on windows and linux with it...

But smoketester is currently not even passing at all.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

But smoketester is currently not even passing at all.

I am looking into the broken pom.xml file. Can you add a println showing the broken source file? (unrelated to this issue)

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

https://builds.apache.org/job/Lucene-Solr-SmokeRelease-trunk/16/console passed, heavy committing now

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Committed also to 4.0 branch (r1388875)

[asfimport]

Commit Tag Bot (migrated from JIRA)

[branch_4x commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1388814

Merged revision(s) 1388813 from lucene/dev/trunk: LUCENE-4415: Port javascript in build files to Groovy (JSR-223)

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Closed after release.