solrconfig.xml provided in solr/conf compatibility with Solr 9

sgurnick commented 3 weeks ago

I provisioned a fresh Solr 9.7 server in conjunction with an upgrade of Avalon from 7.6 to 7.7.2. It is my understanding that Avalon 7.7.2 does support the use of Solr 9.

After I confirmed my standalone Solr 9 instance was up and running, I created a new core and used the files provided in solr/conf of the Avalon repo for the 7.7.2 tag.

After restarting Solr, the core could not load. These warnings and errors appeared in the solr.log file:

2024-10-30 17:09:32.929 WARN  (coreLoadExecutor-13-thread-1) [c: s: r: x:avalon t:] o.a.s.c.SolrConfig Couldn't add files from /opt/solr-9.7.0/contrib/analysis-extras/lib filtered by null to classpath: java.nio.file.NoSuchFileException: /opt/solr-9.7.0/contrib/analysis-extras/lib
2024-10-30 17:09:32.929 WARN  (coreLoadExecutor-13-thread-1) [c: s: r: x:avalon t:] o.a.s.c.SolrConfig Couldn't add files from /opt/solr-9.7.0/contrib/analysis-extras/lucene-libs filtered by null to classpath: java.nio.file.NoSuchFileException: /opt/solr-9.7.0/contrib/analysis-extras/lucene-libs
2024-10-30 17:09:32.931 WARN  (coreLoadExecutor-13-thread-1) [c: s: r: x:avalon t:] o.a.s.c.SolrConfig Couldn't add files from /opt/solr-9.7.0/contrib/extraction/lib filtered by .*\.jar to classpath: java.nio.file.NoSuchFileException: /opt/solr-9.7.0/contrib/extraction/lib
2024-10-30 17:09:32.931 WARN  (coreLoadExecutor-13-thread-1) [c: s: r: x:avalon t:] o.a.s.c.SolrConfig Couldn't add files from /opt/solr-9.7.0/dist filtered by solr-cell-\d.*\.jar to classpath: java.nio.file.NoSuchFileException: /opt/solr-9.7.0/dist

...

2024-10-30 20:04:20.752 WARN  (coreLoadExecutor-13-thread-1) [c: s: r: x:avalon t:] o.a.s.s.FieldTypePluginLoader TokenizerFactory is using deprecated 5.0.0 emulation. You should at some point declare and reindex to at least 8.0, because 7.x emulation is deprecated and will be removed in 9.0
...

2024-10-30 17:09:34.538 ERROR (coreLoadExecutor-13-thread-1) [c: s: r: x: t:] o.a.s.c.CoreContainer SolrCore failed to load on startup => org.apache.solr.common.SolrException: Unable to create core [avalon]
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1787)
org.apache.solr.common.SolrException: Unable to create core [avalon]
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1787) ~[?:?]
        at org.apache.solr.core.CoreContainer.lambda$loadInternal$13(CoreContainer.java:1083) ~[?:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212) ~[metrics-core-4.2.26.jar:4.2.26]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.solr.common.SolrException:  Error loading class 'org.apache.solr.handler.extraction.ExtractingRequestHandler'

It appears the problem is caused by lines 31-37 in the solrconfig.xml file: https://github.com/avalonmediasystem/avalon/blob/c1a19e0a89fdea66ba40f7a1a93fd66c6dbe21aa/solr/conf/solrconfig.xml#L31-L37

In a fresh install of Solr 9:

contrib and dist sub-directories within the solr install directory are no longer present
solr-cell-\d.*\.jar file no longer exists

These are reflected in the Other section of the Major Changes in Solr 9 page:

Contrib modules are now just "modules".
The "dist" folder in the release has been removed.
The solr-extraction module has been cleaned up to produce solr-extraction- jar instead of solr-cell- jars.

To get the avalon core loaded in my Solr 9 instance, I updated the lines 31-37 to just the following lines:

<luceneMatchVersion>9.11</luceneMatchVersion>

<lib dir="${solr.install.dir:../../../..}/modules/analysis-extras/lib" />
<lib dir="${solr.install.dir:../../../..}/modules/extraction/lib" regex=".*\.jar" />

After restarting Solr, the core loaded successfully without the above mentioned warnings/errors.

Questions:

are the files in solr/conf in the avalon repo what I should be using when creating a new Solr 9 core for use with Avalon 7.7 or higher?
- if yes: are these files meant to simply be dropped into the solr 9 avalon core as-is without changes?
- if yes: I'm not sure what I might have missed because I couldn't get the avalon core to load without making changes mentioned above
- if no: is there is documentation somewhere indicating what avalon-specific changes to the configs are necessary to support Solr 9?
- if no: what is the location of the appropriate Solr 9 conf files that should be used?

Thank you for your help and guidance.

cjcolvar commented 3 weeks ago

@sgurnick I thought the config in solr/conf worked out of the box with the version of solr 9 (I think 9.6 locally) that we have. Your changes seem to make sense so maybe they're safe to make no matter the solr 9 version. I'll take a look at this more when I'm back in the office on Monday and get back to you.

cjcolvar commented 3 weeks ago

I started a fresh solr 9.7 locally and it loaded the the solr core for me with the config in solr/conf. FWIW it looks like the only change between Avalon 7.7.2 and 7.8 is commenting out a couple copyField configs (https://github.com/avalonmediasystem/avalon/compare/v7.7.2...main#diff-76699b2586a37ec4cfcb45a02364fc31ea75c15298528af372eb6e6c4fe82ed1) Also the way we're setting up the core is running solr-precreate avalon /path/to/avalon/config in a solr docker container.

avalonmediasystem / avalon

solrconfig.xml provided in solr/conf compatibility with Solr 9 #6094