search

spotify / async-google-pubsub-client

[SUNSET] Async Google Pubsub Client
Apache License 2.0
158 stars 40 forks source link

configure relocation/shading to support usage with Hadoop #2

Closed pettermahlen closed 9 years ago

pettermahlen commented 9 years ago

Hadoop uses com.ning:async-http-client and io.netty:netty versions that are not compatible with the ones used by this client. That means that it's not possible to use this client when writing to HDFS.

pettermahlen commented 9 years ago

@pehrs @danielnorberg does this look OK?

danielnorberg commented 9 years ago

What version of netty would that be?

Seems awkward and fragile to shade in this project to work around a dependency issue elsewhere. Could the shading/relocation happen in the project that uses this library instead?

pettermahlen commented 9 years ago

The reason for the PR is that when writing to HDFS, you need Hadoop, and Hadoop uses io.netty:netty:jar:3.6.2.Final - in many of its various jars. @pehrs may know more about whether there might be some combination of Hadoop jars/excludes that allows you to write to HDFS without Netty, but it doesn't look like it to me.

I would have said it's less fragile and awkward to shade in the library than in every application that wants to use the library and some other version of Netty? I guess it's arguable who creates the dependency issue, but to me, since Netty is an implementation detail in the asynch-google-pubsub-client, it makes sense to encapsulate that via shading rather than pollute the namespaces of all users. Shading is a sucky mechanism for that, but it's the best one I know of.

pettermahlen commented 9 years ago

That's Hadoop 2.2.0, btw.