Possible missing dependency: Apache HttpCore 4.3.x

[spencerwi]

When upgrading our dependencies, we encountered this error in our stack trace:

Exception Thrown
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.NoClassDefFoundError: org/apache/http/util/Args
  at org.elasticsearch.discovery.ec2.Ec2Discovery.<init>(Unknown Source)
  while locating org.elasticsearch.discovery.ec2.Ec2Discovery
  while locating org.elasticsearch.discovery.Discovery
    for parameter 3 at org.elasticsearch.node.service.NodeService.<init>(Unknown Source)
  while locating org.elasticsearch.node.service.NodeService
Caused by: java.lang.NoClassDefFoundError: org/apache/http/util/Args

After checking through the pomfile for elasticsearch-cloud-aws, we didn't find any references to a specific version of Apache HTTPCore (so we weren't able to directly identify which version elasticsearch-cloud-aws depends on).

Should this dependency be added to the pomfile?

[dadoonet]

Did you update cloud-aws project as well?

What are your elasticsearch version and aws plugin version in your pom.xml file?

[spencerwi]

We are using elasticsearch 1.5.0 and cloud-aws 2.5.0 (per the helpful README in cloud-aws!); we were using these in a previous release as well, but had another dependency that happened to provide HTTPCore 4.3.x as a transitive dependency. When that lib was upgraded, it specified HTTPCore 4.2.5, which appears not to contain the org.apache.http.util.Args class.

So when we were no longer lucky enough to have another dependency transitively satisfying cloud-aws's dependency on HTTPCore 4.3.x, since cloud-aws doesn't specify HTTPCore 4.3.x in its pomfile, we got a NoClassDefFound error.

It was something like this before:

                     OUR_APP
          /                         \
         /                           \
        /                             \
    DepA                     ElasticSearch
       |                               /  (dep not in pomfile)
 HTTPCore <---------------------------/        
 (no version specified, so grabs 4.3.x)

And now we have this:

                     OUR_APP
          /                         \
         /                           \
        /                             \
    DepA                     ElasticSearch
       |                               /  (dep not in pomfile)
 HTTPCore <---------------------------/        
   (4.2.5, breaks ElasticSearch)

[dadoonet]

I think I got it. Thanks!

What would happen if you add in your pom.xml version 4.3.2 of HTTPCore? Would this break your code?

[jerry4]

We downgraded the library that specified an old version and now we are back to pulling in HTTPCore 4.3.3. I suspect 4.3.2 would work just fine since 4.3.3 does.

[spencerwi]

As @jerry4 mentioned, downgrading only the lib that "destablized" things worked for us (resulted in Maven fetching HTTPCore 4.3.3). That works for us for now, since the version bump on that lib is noncritical.

Other folks in a similar situation may need to explicitly depend on HTTPCore 4.3.3 (or 4.3.2, if it works just as well) in their pomfiles to avoid the issue. It's not a great solution -- manually specifying transitive dependencies can be painful -- but it should work.

[dadoonet]

Is the lib you mentioned an internal project or a public one?

If the later, could you give a link to that lib?

I think you can close the issue, right?

[spencerwi]

The lib we mentioned is a public lib: twilio-java

I suppose we could close the issue since we found a workaround for the problem in cloud-aws's pomfile, but I would prefer that the issue be resolved. If you'd like, I can submit a PR.

[spencerwi]

Looking at the stack trace, it's possible that this is a missing dep for ElasticSearch rather than cloud-aws.

On that basis, if you guys deem it not the right move, I can close this PR and take the issue over there.

[dadoonet]

Was that the full stack trace?

[spencerwi]

Minus plenty of direct-passthrough catch-and-rethrow from servlet filters, etc (our app is a webapp), yes.

[dadoonet]

My theory here is that NodeService tries to create the AWS class instance which fails because AWS SDK can not create one of its classes and throws java.lang.NoClassDefFoundError: org/apache/http/util/Args.

[tlrx]

I don't think this is a problem with a missing dependency in elasticsearch or elasticsearch-cloud-aws projects.

elasticsearch 1.5.0 depends on httpcore as a transitive dependency for its tests:

$ mvn dependency:tree | grep http
[INFO] ------------------------------------------------------------------------
[INFO] Building elasticsearch 1.5.0
[INFO] ------------------------------------------------------------------------
...
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.3.5:test
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.3.2:test

elasticsearch-cloud-aws 2.5.0 depends on aws-java-sdk-ec2 which depends on httpcore as a transitive runtime dependency:

$ mvn dependency:tree | grep http
[INFO] ------------------------------------------------------------------------
[INFO] Building Elasticsearch AWS cloud plugin 2.5.0
[INFO] ------------------------------------------------------------------------
...
[INFO] +- com.amazonaws:aws-java-sdk-ec2:jar:1.9.23:compile
[INFO] |  \- com.amazonaws:aws-java-sdk-core:jar:1.9.23:compile
[INFO] |     +- org.apache.httpcomponents:httpclient:jar:4.3.5:compile
[INFO] |     |  \- org.apache.httpcomponents:httpcore:jar:4.3.2:compile

Those declarations looks OK to me. The problem here is that you are using different versions of the same library in the same application.

[spencerwi]

Well, we've workaround the missing (explicitly-versioned, at least) dependency for our use-case, so since there appears to be a great resistance to actually resolving the problem, I'll just close this. I really ought to have foreseen this kind of pushback from the ES team.

[clintongormley]

@spencerwi We can't list every transitive dependency of every dependency that we use. It would be a huge responsibility maintenance and testing-wise etc and we have to delegate that for the most part to the projects responsible. If the aws-java-sdk-ec2 pom doesn't list the dependency, it should be fixed there, not worked around in Elasticsearch.

[spencerwi]

@clintongormley And that's a fair point. But if that's the case, all that was necessary was to say "we don't directly depend on this either, the problem is upstream", not

The problem here is that you are using different versions of the same library in the same application.

(i.e. "you're just doing things wrong, fix your app.")

[kimchy]

FYI, historically, the aws sdk was pretty bad at listing the dependencies version in its Pom. For example, if you download the zip version from the site for a specific version, and just list the same version in the Pom, you would end up with 2 different http lib versions.

I always found this concerning, so in the Pom I listed the explicit versions that came with the fully packaged zip one. Should we get back to do that?

[tlrx]

@spencerwi I'm sorry if my sentence hurt you, it was of course not my goal. I should have explained a bit more and thanksfully @clintongormley did it.

We might list all dependencies but the POM file of the AWS SDK looks quite complete to me in its latest versions. I think this issue is a "Jar Hell" one, and for what I've usually seen the easiest way to solve that is to declare the transitive dependency in the application POM file (and fix the right version to use) and to exclude it when declaring the dependencies (with the maven tag).

[spencerwi]

@tlrx Nah, no offense taken, it's cool. The explanation from @clintongormley was helpful because it informed me that the issue lay upstream rather than simple being a "won't-fix" for non-technical reasons. That's all I was pushin' for.

[gundubandagar]

compile('com.omertron:themoviedbapi:4.2') { exclude group: 'org.apache.httpcomponents', module: 'httpclient' exclude module: 'junit' }