search

foursquare / fsqio

A monorepo that holds all of Foursquare's opensource projects
Apache License 2.0
252 stars 54 forks source link

hadoop-lzo jar resolution #62

Open jglesner opened 4 years ago

jglesner commented 4 years ago

Hello,

I'm running into an issue using the latest fsqio master where ivy is consistently unable to resolve hadoop-lzo and fails to run serve.py. 

[NOT FOUND ] com.hadoop.gplcompression#hadoop-lzo;0.4.19!hadoop-lzo.jar (0ms)
==== maven.twttr.com-maven: tried
https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/hadoop-lzo-0.4.19.jar

I'm not all too familiar with the dependency management aspects of twofishes. I'm running this from an ubuntu 14.04 docker container, and everything has worked historically ... until now. Any recommendations on how to figure out what's going on / solve for this? I would greatly appreciate it.

Strangely, when I git clone the fsqio repo, and run serve.py from the command line on an Amazon Linux 1 EC2 instance it works perfectly. No dependency issues. I need to run it from a docker container, though preferably.

jglesner commented 4 years ago

Not sure if this is the correct fix, but I was able to get past this issue by adding a new global exclusion to Ivy by editing the pants.ini file under [resolve.ivy]: {"org": "com.hadoop.gplcompression", "name": "hadoop-lzo"},

mateor commented 4 years ago

I think that your fix is likely the right answer. We are getting ready to update fsq.io, would you be interested in sending a PR?

jglesner commented 4 years ago

Sure, I can submit it tonight (if that fits with your timeline).

mateor commented 4 years ago

That sounds great! The update is addressing unrelated bug in our dependencies, so we may need to land that first in order to get the build green anyway