JarHell check prevents having Derby on the classpath

[hakanai]

We are trying to use Elasticsearch in an application which also happens to have Derby on the classpath.

Setting aside the huge bugs with running under a security manager which the Elasticsearch developers appear to be unwilling to fix but which can be worked around for now by granting additional security privileges to the wildcard code source, JarHell then proceeds to crash, complaining about other issues. Some of its complaints are legit, others not so much.

For instance, take Apache Derby. The Derby build contains three jars (ignoring tools), all of which have some overlapping classes with the others but none of which are a strict superset of any other. (Like a Venn diagram, I guess.) We use two of these in our desktop application and three of them in the server application.

The JarHell check chokes on these, making it impossible to use both Derby and Elasticsearch in the same application.

I found a ticket over at Derby about extracting a "core" jar with common code, but even though that ticket was created a decade ago, it doesn't look like any work has begun. For completeness, I will file it here as well, as I notice that there are exceptions for some other libraries (e.g. log4j), so I guess adding exceptions for individual libraries does not appear to be out of the question.

[s1monw]

Putting aside your passive aggressive voice on this ticket, I think you are totally missing to communicate your usecase. Are you embedding Elasticsearch, if so there is no jarhell checks. Are you using Elasticsaerch as a client application then there is not jarhell check. Are you building a plugin then you have to provide a clean classpath that's just what we agreed on which prevents bugs rather than being one. If you are using our test-framework and running into jarhell then you can disable this now with 2.2.

A general advice or a personal note, if you feel like you need help on an open source project no matter if backed by a company or a foundation it's wise to be friendly and to explain your problem on a technical level. Statements like this:

Setting aside the huge bugs with running under a security manager which the Elasticsearch developers appear to be unwilling to fix but which can be worked around for now by granting additional security privileges to the wildcard code source, JarHell then proceeds to crash, complaining about other issues. Some of its complaints are legit, others not so much.

are not helpful and don't make anybody want to help you. Even further is a reference of your attitude on the internet which I think needs to be put out there with care. I don't expect you to respond to this but I felt like I am putting it out there.

[hakanai]

We're embedding it, but the JarHell checks are still being done.

[s1monw]

are you installing plugins?

[hakanai]

Yeah, we have the analysis-icu and analysis-kuromoji plugins present.

[hakanai]

Sample test:

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

import org.elasticsearch.client.Client;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.node.Node;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;

import static org.elasticsearch.node.NodeBuilder.*;

public class TestEmbeddedElasticSearch
{
    @Rule
    public final TemporaryFolder temporaryFolder = new TemporaryFolder();

    @Test
    public void test() throws Exception
    {
        try (Node node = createNode();
             Client client = node.client())
        {

        }
    }

    private Node createNode() throws Exception
    {
        File nodeHome = temporaryFolder.newFolder();
        Path pluginsPath = Paths.get("storage-core/build/elasticsearch-plugins");
        if (!Files.exists(pluginsPath))
        {
            throw new IllegalStateException("Could not find ES plugins path. Tests will fail. Plugin path: " + pluginsPath);
        }

        return nodeBuilder()
            .settings(Settings.settingsBuilder()
                              .put("path.home", nodeHome.getAbsolutePath())
                              .put("path.plugins", pluginsPath.toString())
                              .put("index.number_of_shards", 1)
                              .put("index.number_of_replicas", 0)
                              .put("node.http.enabled", "false"))
            .local(true)
            .node();
    }
}

Fails with:

java.lang.IllegalStateException: failed to load bundle [] due to jar hell

    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:421)
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:129)
    at org.elasticsearch.node.Node.<init>(Node.java:146)
    at org.elasticsearch.node.Node.<init>(Node.java:128)
    at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
    at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:152)
    at TestEmbeddedElasticSearch.createNode(TestEmbeddedElasticSearch.java:49)
    at TestEmbeddedElasticSearch.test(TestEmbeddedElasticSearch.java:25)
    ...junit launcher stuff...
Caused by: java.lang.IllegalStateException: jar hell!
class: org.apache.commons.logging.impl.NoOpLog
jar1: /Users/daniel/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.1.3/f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f/commons-logging-1.1.3.jar
jar2: /Users/daniel/.gradle/caches/modules-2/files-2.1/org.slf4j/jcl-over-slf4j/1.7.12/adef7a9e1263298255fdb5cb107ff171d07c82f3/jcl-over-slf4j-1.7.12.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:280)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:186)
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:419)
    ... 30 more

[rjernst]

@trejkaz I see your test is trying to run an elasticsearch node in the same process as your tests, along with having plugins already installed in a "special directory". This won't work because the plugin service is meant to be used by elasticsearch running as a server, which is what starting a node in the way you are does. In the master branch of ES, we have removed NodeBuilder and replaced it with simply constructing a Node, which takes "classpath plugins". These are what tests use (like ESIntegTestCase), as they expect the plugins and ES to be in the same test process. I would recommend using that test case as a base, or creating a subclass of Node to use the constructor that is now exposed in master.

However, a better solution to test the real integration here would be to start ES in a separate process. This is what our integration tests do in 2.x and master, and it has proven to be very useful for testing how things interact with ES in a realistic way. If developing a plugin, and using the shared POM in 2.x (or the gradle plugin in master), you can utilize the existing build infrastructure to start the ES node and install plugins before integ tests start.

[hakanai]

This workaround works, using

    Environment environment = InternalSettingsPreparer.prepareEnvironment(settings, null);
    Collection<Class<? extends Plugin>> plugins = ImmutableList.of(AnalysisICUPlugin.class, AnalysisKuromojiPlugin.class);
    return new Node(environment, Version.CURRENT, plugins){}.start();

instead of the builder. The settings are identical. And in many ways, having the plugins loaded from the classpath is much nicer anyway.

[s1monw]

oh great you found a solution.