Instructions for setting up pom-resolve on a new repo?

[toddgardner]

Hi @mateor,

Following up from slack, I was wondering if you could give any guidance on setting up pom-resolve and validation stuff on a new repo; I don't see it on packages on pypi for anything but buildgen. It looks like neat stuff! Solves a lot of the problems I was having with pants deps.

(I was having problems getting buildgen to work, but I think I've root caused it to changes in the setup script from https://github.com/pantsbuild/setup/pull/15 which causes version conflicts on setuptools by what the script installs and what pants requires when fsqio requires it; I'm not clear what the right fix is)

[mateor]

Hi Todd :)

So I think I see 4 distinct issues mentioned:

1. trouble with python dependencies using buildgen (setuptools was mentioned)
2. help trying pom-resolve
3. A Slack posted error: Package SourcePackage(u'file:///fsqio.pants.contrib.buildgen.python-1.1.0.tar.gz') is not translateable by ChainedTranslator(EggTranslator, SourceTranslator)
4. validation

I will try and give a little color on consuming these plugins in general, but if you don't mind, I would appreciate if you could file separate issues for each specific issue. It isn't so much that I can't help here, but a hope that resolving it in a single place would be useful to future users :)

Questions I have for you:

• What language are you looking to support? I am guessing Scala because you are interested in both buildgen and pom-resolve. Are you also trying to add Python support?
• What version of Pants are you using?
• Are you converting a repo to Pants or just adding plugins to something that already uses Pants?

Okay - so let's see. I will post a follow-up comment with some more concrete info.

[mateor]

The central benefit of pom-resolve is that it uses a single, global set of dependencies. It has deep cache integration, so users only run resolution when they update 3rdparty entries or change pom-resolve options.

This has given Foursquare tremendous benefit - but there are a few caveats you should know about.

Caveats

1. It is a custom resolver, with all the good and bad that implies.

   • It may not always play nice with the latest Ivy or Maven feature.
   • In practice, this is not a big issue, but it is worth knowing in advance.

2. The resolve step can be pretty slow - it was never optimized much because it is deeply integrated with the artifact cache.

   • We have around 300 jars in 3rdparty, and a resolve after changing a jar version can take 3-4 mins if that resolve has not been cache anywhere.
   • Generally our engineers just download the resolve from the cache and rarely have to resolve on their machine.
   • If you have an artifact cache set up, this is a non-issue, and perhaps a non-issue without it.

3. An internal Nexus or another jar repo is a very useful escape hatch for problem deps.

   • If we find a pom that gives pom-resolve trouble, we sometimes just republish it internally.
     • This is pretty rare, maybe 3-4 times I am aware of.
   • But a Nexus makes using pom-resolve a lot easier, since there is always at least one answer to a resolution problem - republish.

Adoption I would be interested in how many global dependencies you have (we have about 300 direct deps and maybe 800 transitive deps).

In order to POC, I think I will ask you to add this as a loose-source plugin, just while doing your POC. This means:

• copy and paste the pom directory into a checkout of your repo.
• Add that location to the pythonpath entry in your pants.ini
• Add the plugin to backend_packages
• Run a ./pants pom-resolve.
  • The first resolve will take a long time because it has to resolve AND download every jar.
  • To get a feel for what development is like, toggle some versions in 3rdparty and run it again.

I know it sounds gross, but it is the quickest way to POC - if you decide you want to adopt, I will happily create a package for you. Fsq.io will be a pretty good resource for you - this repo is split directly from our internal monorepo - the configuration you see here is the exact code we run internally.

[mateor]

That buildgen issue is something I have seen before in another context - but stopped poking at because it turned out to not be the actual problem at hand.

The details are a bit hazy - but I think the TLDR of it is that actually what is happening is that Pants is trying to install that package in the bootstrap process, at a point in the pipeline where is is assuming a certain packaging format: https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/python/interpreter_cache.py#L197

I would do the following before digging too deeply there:

I have a commit that shows how to consume buildgen. I just tried it in fsq.io and it still works. The commit is here - the quirks of the namespace is because we published that plugin out of a Pants checkout.

The hard thing about consuming Scala buildgen is the third_party_map. That is a json file that maps your target names to import namespace. It can be tedious to get setup the first time - but once you do buildgen is by far the best value you can add.

You can add/override the map in your ini file, and I have considered expanding the ability to override from a file.

[toddgardner]

Thanks @mateor, will break this out; 1 and 3 are the same I believe but I think I can get a reproducer.

What language are you looking to support? I am guessing Scala because you are interested in both buildgen and pom-resolve. Are you also trying to add Python support?

Scala (mainly), python and node/react.

What version of Pants are you using?

I started with 1.2.0 but it can be whichever.

• Are you converting a repo to Pants or just adding plugins to something that already uses Pants?

New repo, new company; not a lot of code or any unified build system. I used pants at the last company, using managed_jar_dependencies, so I'm familiar with the problem space. We ran into some problems with caching cutouts (compile some/path:: followed by compile some/path/under::) and problems with exclusions (sl4j), so I thought I'd give pom-resolve a spin this time around.

I figured the answer would be copying but thought I'd ask.

[mateor]

It is copying for now...but if you think you are move forward with pom-resolve I will figure out some sort of publishing for it. The code is already to be packaged, there is just some trial and error around publishing from fsq.io.

But as much as possible I think it would behoove the buildgen/pom-resolve consumers to standardize to one source tree if possible. Foursquare is willing to compromise on the details of such a collaboration in just about every detail.