search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.69k stars 1.04k forks source link

Bump minimum Java version requirement to 11 [LUCENE-8738] #9783

Closed asfimport closed 5 years ago

asfimport commented 5 years ago

See vote thread for reference: https://markmail.org/message/q6ubdycqscpl43aq.

Migrated from LUCENE-8738 by Adrien Grand (@jpountz), resolved Apr 16 2019 Attachments: LUCENE-8738.patch, LUCENE-8738-javadoc-locale-en-US.patch, LUCENE-8738-solr-CoreCloseListener.patch Linked issues:

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Hi, next to updating the -release flag in the JDK (and removal for the legacy -source/target switch in the build), there are some thing to take care of:

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit ab391a34875153e9761f27c7f984650d67415b48 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ab391a3

LUCENE-8738: Require Java 11.

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

Hi Uwe, thanks for this checklist. I have started looking into what needs fixing to bump the minimum version requirement to Java 11 on the jira/LUCENE-8738 branch if you are interested to have a look or help. I haven't been through all items that you have listed yet.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Ah I was about to ask if you are already working on it :-)

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

I am sitting in a train so I can work abit on it. The first code changes look fine. I will now remove the remaining parts and fine-tune the build files (by partially reverting some of my older commits). I will just heavy commit to the branch.

I have no idea about the Python Linter, maybe the fixes are easy and we need to just change some regexes.

asfimport commented 5 years ago

Kevin Risden (@risdenk) (migrated from JIRA)

Saw that maven template was updated. Might need to take into account SOLR-11579 as well which looks at Maven plugins.

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

@uschindler Feel free to heavy commit, I don't plan to touch it again until tomorrow.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

More TODO points:

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit c1d0889d1c997cd502c1b3bbba3d7256241e9d6e in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Uwe Schindler https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c1d0889

LUCENE-8738: Update ECJ to latest version (it's no a separate component, no longer bundled with JDT, so it looks like version goes back)

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Hi, I bumped the ECJ version. Actually it goes back, but this is caused by the fact that ECJ is now a separate JAR file which is no longer included in JDT-Core. ECJ 3.17.0 is the latest released this year in January. I quickly tested it with a hack in build file and it works on Lucene-Core. Javadocs linting still does not works as it is disabled because the python stuff needs fixing. I'll take care of this later.

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit c12ceb6c651fb35ff06cc9d608ad6eaa9c2d4f74 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Uwe Schindler https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c12ceb6

LUCENE-8738: Enable HTML5 for Javadocs. Index is still disabled, but it might now good to enable it again, as there are no frames anymore!

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

I decided not to enable the search box in the javadocs, as it adds a lot of Javascript crap to the javadocs folder. It might be good to do this in the future, as javadocs have no frames anymore, so navigation is harder. If someone wants to create the index (and also the search box, which is enabled when the standard javadocs index is built), he/she can pass -Djavadoc.noindex=false

I also switched to HTML5 when generating javadocs.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

My question: Should we remove the MR-JAR code completely? It is also in tools/src/groovy. @jpountz only disabled the ANT tasks by commenting them out, but we can also completely remove it. It can always be restored from GIT history. Thoughts?

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

You argued to keep this code around in the vote thread, which is why I haven't removed it. I have a slight preference for removing it, no strong feelings though.

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit 2172613c9422c08473fb406e5ce0cd8a41a1ea97 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2172613

LUCENE-8738: Remove calls to deprecated APIs.

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

@uschindler I'm curious if you have an idea how to make tidy work with Java 11?

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit d108edce57e6bdea4286721dd6ef1b8eef68ffd6 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=d108edc

LUCENE-8738: Make nightly-smoke pass.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

You made nightly smoke pass, so I think you also fixed the javadocs checker? So I think we can enable javadocs linting also in build.xml. what's the issue with tidy?

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

Currently the build prevents tidy from running if the Java version is not 1.8. I tried to remove this constraint to see how it goes and it fails silently. I can dig but I was wondering if you knew anything about it.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Hi, I figured out the same:

-documentation-lint:
     [echo] checking for broken html...
[ivy:cachepath] downloading https://repo1.maven.org/maven2/net/sf/jtidy/jtidy/r938/jtidy-r938.jar ...
[ivy:cachepath] ........................... (244kB)
[ivy:cachepath] .. (0kB)
[ivy:cachepath]         [SUCCESSFUL ] net.sf.jtidy#jtidy;r938!jtidy.jar (646ms)
    [jtidy] Checking for broken html (such as invalid tags)...

BUILD FAILED
C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\lucene\build.xml:142: The following error occurred while executing this line:
C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\lucene\build.xml:148: The following error occurred while executing this line:
C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\lucene\common-build.xml:2120: Tidy was unable to process file C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\lucene\build\docs\analyzers-common\allclasses.html, 1 returned.

I was expecting it to fail because of some "internal API" shit, but actually it's not able to parse the file.

I have the feeling and the documentation/bugreports on jTIDY shows: It's not able to handle HTML5. Java 11 is able to produce HTML5, so for future-proveness we should really keep this enabled.

To find out what's going on, I will quickly change to HTML4 and let it run again. If this is the case and it works, we have 2 possibilities:

Uwe

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Hi, it's indeed like said before: jTIDY can't process HTML5 - plain simple. It passes the check on Java 11 when I add the -html4 parameter to Javadocs call (currently it uses -html5, but that param is obsolete as this is the default).

I just had to disable the javadocs warning check to make build pass, otherwise it failed:

  [javadoc] javadoc: warning - You have specified the HTML version as HTML 4.01 by using the -html4 option.
  [javadoc] The default is currently HTML5 and the support for HTML 4.01 will be removed
  [javadoc] in a future release. To suppress this warning, please ensure that any HTML constructs
  [javadoc] in your comments are valid in HTML5, and remove the -html4 option.

I see currently only one possibility: disable jTIDY until we find a replacement. I will do this now and add a TODO. I will also remove the obsolete -html5 parameter.

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

+1 to disable jtidy for now

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit 65fc3c62582442af939a12fefd774227af2375c9 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Uwe Schindler https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=65fc3c6

LUCENE-8738: Disable jTIDY, as it can't handle HTML5. Enable Javadocs Linting in Java 11 only

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

OK, done. We may have to think about, if we now want to enable Javadocs liting for all types of JDKs. Currently the "documentation-lint.supported" code looks crazy: It checks for exact "Java 11" (the reason for this is that on later versions they often change HTNML format and the the python linter breaks) and also for "hotspot". Maybe we can clean that up.

I will now also remove the commented out MR-JAR stuff and the groovy patcher files. Those can easily be restored using GIT, so why keep it in source tree.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

I kept the jtidy-macro, so I did not have to edit the Lucene/Source main build files where the macro was called. If we decide to abandon the check, we can remove that completly (just grep for "jtidy").

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit ba01df2c22840b699a758df930e0212e1682457b in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Uwe Schindler https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ba01df2

LUCENE-8738, #9015: Remove MR-JAR classfile patching from Lucene 9 completely (minimum Java 11)

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

I removed the MR-JAR parts. To easily restore it in case we would like to use it for Java 12+, this commit should survive in GIT, so we should not squash the branch of this issue.

I kept the documentation unmodified in the README.maven file, as the artifacts built by Maven should never be used in production, which is unrelated to MR-JAR.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

I have some comment about the deprecation removal:

The Class#newInstance() stuff was replaced by clazz.getDeclaredConstructor().newInstance(). This is perfectly fine and also suggested in the deprecation notice. But I'd like to use getConstructor() instead, as we are only interested in "public" constructors.

We should also think about the fact that the code may now throw different exceptions. About the changes @jpountz did: Most code ways already wrapped with a general catch-all code, but maybe we can improve and unwrap runtime excetions from the InvocationTargetException. I will have a closer look again.

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

+1 The other part that needs reviews is the replacement of Observable/Observer with utility classes from java.beans.

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit 9d8d69c1fd6725747379214abfd6e4554a49ec28 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=9d8d69c

LUCENE-8738: Remove Java 8 unmapping.

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

There seems to be issues with links to the standard API. I wonder that it might be related to the move from package-list to element-list.

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit e410a35e8cd49415ee4460266afa3d1eba65371e in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e410a35

Merge branch 'master' into jira/LUCENE-8738

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

There seems to be issues with links to the standard API. I wonder that it might be related to the move from package-list to element-list.

I have seen this, too, javadocs builds break with a warning message and then it fails build. Is this what you are seeing? The problem only occurs if you have mixed javadoc builds. It helps to "ant clean" from root folder and start from scratch.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

(this issue always existed also in Java 8 vs. Java 11 on current master branch)

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

Apparently the issue can be worked around by calling the file package-list locally, even though it is supposed to be called element-list with the move to modules. I'll push a fix shortly.

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit 541b77afe78b08190fbb8bfa2e091862d9ff08c8 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=541b77a

LUCENE-8738: Keep the file called package-list locally to make links work.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Ok, maybe we should do the same on 8.x, so switching branches and jvms don't bring you trouble all the time.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Ok, maybe we should do the same on 8.x, so switching branches and jvms don't bring you trouble all the time.

Sorry this does not affcet 8.x as it's about the downloaded files from the Oracle web page. The issue I am talking about is the same for cross module references inside Lucene. If you compile in Lucene 8.x it produces files with a different name in Java 8 or Java 11. So if you switch JDKs while working, you get errors. Maybe we can also tell javadoc command to produce the output files with a consistent name.

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit 744b375f2739112ca8326e036839f0b283065aba in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=744b375

LUCENE-8738: Make documentation-lint pass.

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

Sorry Uwe, I don't understand what you are suggesting.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Sorry, forget about it. I will try to fix it in 8.x branch in a separate issue. The problem is:

So my wish would be to make javadoc tool in Lucene 8.x produce the same package-list files on all JVMs. If you compile and build javadocs on Lucene 8.x with Java 11 it produces element-list and fails. But that's more an issue of 8.x only.

This issue was getting on my nerves yesterday. But sorry for the hassle here, it just looked like the same problem...

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit 62294cfd7130c14d0abb641059e92eedf17876ea in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Uwe Schindler https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=62294cf

Merge branch 'master' of https://gitbox.apache.org/repos/asf/lucene-solr into jira/LUCENE-8738

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit ebd235c765aa70d2a7ef5b380f7a8fc35d9803a8 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Uwe Schindler https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ebd235c

LUCENE-8738: Remove TODOs about Java 11 (charsets in Luke)

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

After Luke was included into Lucene I just pushed some Java 11 specific cleanup that I had on my todo list already while reviewing and improving @Tomoko Uchida's code.

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit d13874a0da37732c90cb764b8f9165bfeb8713da in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Uwe Schindler https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=d13874a

Merge branch 'master' of https://gitbox.apache.org/repos/asf/lucene-solr into jira/LUCENE-8738

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit cfe959b81458bac06f1ad5d8161e59adb85b87b2 in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=cfe959b

LUCENE-8738: Update Eclipse config.

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit a11c3974b9e6b80b7b566bc1cd8b08c5d1db8a6b in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a11c397

LUCENE-8738: Update Idea configuration (untested).

asfimport commented 5 years ago

ASF subversion and git services (migrated from JIRA)

Commit 65ddd311288411e9dd56fd05e69a1a68c64a84ea in lucene-solr's branch refs/heads/jira/LUCENE-8738 from Adrien Grand https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=65ddd31

LUCENE-8738: Use getConstructor instead of getDeclaredConstructor.

I had to make FileExchangeRateProvider public.

asfimport commented 5 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

@uschindler Do you know what still needs to be done before merging back to master? When we are done, ore close to being done, I plan to send an email to the list to ask for some more eyes on changes that I did before merging, especially the Observable/Observer removal.

asfimport commented 5 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

I don't see anything that needs to be done before. I had not yet tested Eclipse config, did you do this? For Java 11 work I have to first update my main Eclipse installation (AAAAARGH). You did not write "untested" so I think you did it.

When we merge to master, we should do this in some coordinated way, as Apache Jenkins and Policeman Jenkins need update of configs. Not sure about other Jenkins servers around.

The FileExchangeRateProvider thing is strange, it worked before as the package names were somehow correct, but that's a bug anyways if it appears in some solr config but it's not a public class! So that's a nice find, we should at least backport the "public" part to 8.x - as it's a bug!