search

foursquare / fsqio

A monorepo that holds all of Foursquare's opensource projects
Apache License 2.0
252 stars 54 forks source link

Error installing buildgen plugin in new repository. #28

Closed toddgardner closed 7 years ago

toddgardner commented 7 years ago

Adding the buildgen from pypi to the plugins list causes some build failures (full stack trace included below).

I have a reproducer here: https://github.com/toddgardner/fsqio-plugin-test

(I think this is due to unpinning the setuptools version in the pants script, from the PRs around: https://github.com/pantsbuild/setup/issues/14, specifically https://github.com/pantsbuild/setup/commit/4eccc24a1108bacad84f9dd8882de6c0132a610b) Not sure what the fix would be here.

(this corresponds to 1 and 3 from https://github.com/foursquare/fsqio/issues/27)

$ ./pants compile
New python executable in /Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.xyWThV/install/bin/python
Installing setuptools, pip, wheel...done.
Requirement already satisfied (use --upgrade to upgrade): setuptools<31.0,>=5.4.1 in /Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.xyWThV/install/lib/python2.7/site-packages
Collecting pantsbuild.pants==1.2.0
  Downloading pantsbuild.pants-1.2.0.tar.gz (892kB)
    100% |████████████████████████████████| 901kB 845kB/s 
Collecting twitter.common.collections<0.4,>=0.3.1 (from pantsbuild.pants==1.2.0)
Collecting ansicolors==1.0.2 (from pantsbuild.pants==1.2.0)
Collecting setproctitle==1.1.10 (from pantsbuild.pants==1.2.0)
Collecting six<2,>=1.9.0 (from pantsbuild.pants==1.2.0)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting packaging==16.7 (from pantsbuild.pants==1.2.0)
  Using cached packaging-16.7-py2.py3-none-any.whl
Collecting pathspec==0.3.4 (from pantsbuild.pants==1.2.0)
Collecting scandir==1.2 (from pantsbuild.pants==1.2.0)
Collecting twitter.common.dirutil<0.4,>=0.3.1 (from pantsbuild.pants==1.2.0)
Collecting requests<2.6,>=2.5.0 (from pantsbuild.pants==1.2.0)
  Using cached requests-2.5.3-py2.py3-none-any.whl
Collecting pystache==0.5.3 (from pantsbuild.pants==1.2.0)
Collecting psutil==4.3.0 (from pantsbuild.pants==1.2.0)
Collecting pex==1.1.13 (from pantsbuild.pants==1.2.0)
  Downloading pex-1.1.13-py2.py3-none-any.whl (105kB)
    100% |████████████████████████████████| 112kB 2.0MB/s 
Collecting docutils<0.13,>=0.12 (from pantsbuild.pants==1.2.0)
Collecting Markdown==2.1.1 (from pantsbuild.pants==1.2.0)
Collecting Pygments==1.4 (from pantsbuild.pants==1.2.0)
Collecting twitter.common.confluence<0.4,>=0.3.1 (from pantsbuild.pants==1.2.0)
Collecting fasteners==0.14.1 (from pantsbuild.pants==1.2.0)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting coverage<3.8,>=3.7 (from pantsbuild.pants==1.2.0)
Collecting pytest<2.7,>=2.6 (from pantsbuild.pants==1.2.0)
  Downloading pytest-2.6.4.tar.gz (512kB)
    100% |████████████████████████████████| 522kB 1.2MB/s 
Collecting pytest-cov<1.9,>=1.8 (from pantsbuild.pants==1.2.0)
  Downloading pytest-cov-1.8.1.tar.gz
Collecting futures==3.0.5 (from pantsbuild.pants==1.2.0)
  Using cached futures-3.0.5-py2-none-any.whl
Collecting cffi==1.7.0 (from pantsbuild.pants==1.2.0)
  Using cached cffi-1.7.0-cp27-cp27m-macosx_10_10_intel.whl
Collecting lmdb==0.89 (from pantsbuild.pants==1.2.0)
Collecting pywatchman==1.3.0 (from pantsbuild.pants==1.2.0)
Collecting twitter.common.lang==0.3.9 (from twitter.common.collections<0.4,>=0.3.1->pantsbuild.pants==1.2.0)
Collecting pyparsing (from packaging==16.7->pantsbuild.pants==1.2.0)
  Using cached pyparsing-2.1.10-py2.py3-none-any.whl
Collecting twitter.common.log==0.3.9 (from twitter.common.confluence<0.4,>=0.3.1->pantsbuild.pants==1.2.0)
Collecting monotonic>=0.1 (from fasteners==0.14.1->pantsbuild.pants==1.2.0)
  Using cached monotonic-1.2-py2.py3-none-any.whl
Collecting py>=1.4.25 (from pytest<2.7,>=2.6->pantsbuild.pants==1.2.0)
  Downloading py-1.4.32-py2.py3-none-any.whl (82kB)
    100% |████████████████████████████████| 92kB 2.0MB/s 
Collecting cov-core>=1.14.0 (from pytest-cov<1.9,>=1.8->pantsbuild.pants==1.2.0)
  Downloading cov-core-1.15.0.tar.gz
Collecting pycparser (from cffi==1.7.0->pantsbuild.pants==1.2.0)
Collecting twitter.common.options==0.3.9 (from twitter.common.log==0.3.9->twitter.common.confluence<0.4,>=0.3.1->pantsbuild.pants==1.2.0)
Building wheels for collected packages: pantsbuild.pants, pytest, pytest-cov, cov-core
  Running setup.py bdist_wheel for pantsbuild.pants ... done
  Stored in directory: /Users/todd/Library/Caches/pip/wheels/ce/ff/3a/2d8598b56bdfb9468ea67ee963a03c3b35b22dd597084a7cb1
  Running setup.py bdist_wheel for pytest ... done
  Stored in directory: /Users/todd/Library/Caches/pip/wheels/ca/1c/fe/8b76e537572f91c810910e822cccb178ba3156e432e644ac89
  Running setup.py bdist_wheel for pytest-cov ... done
  Stored in directory: /Users/todd/Library/Caches/pip/wheels/e5/94/08/eab43cda4e17e6ae729ed2cd832d7ca4e3ddca7fa6886ec2b8
  Running setup.py bdist_wheel for cov-core ... done
  Stored in directory: /Users/todd/Library/Caches/pip/wheels/86/e1/c2/9ff8cfe9773ce07003f2c2be096e169af4614c2f634671d49b
Successfully built pantsbuild.pants pytest pytest-cov cov-core
Installing collected packages: twitter.common.lang, twitter.common.collections, ansicolors, setproctitle, six, pyparsing, packaging, pathspec, scandir, twitter.common.dirutil, requests, pystache, psutil, pex, docutils, Markdown, Pygments, twitter.common.options, twitter.common.log, twitter.common.confluence, monotonic, fasteners, coverage, py, pytest, cov-core, pytest-cov, futures, pycparser, cffi, lmdb, pywatchman, pantsbuild.pants
Successfully installed Markdown-2.1.1 Pygments-1.4 ansicolors-1.0.2 cffi-1.7.0 cov-core-1.15.0 coverage-3.7.1 docutils-0.12 fasteners-0.14.1 futures-3.0.5 lmdb-0.89 monotonic-1.2 packaging-16.7 pantsbuild.pants-1.2.0 pathspec-0.3.4 pex-1.1.13 psutil-4.3.0 py-1.4.32 pycparser-2.17 pyparsing-2.1.10 pystache-0.5.3 pytest-2.6.4 pytest-cov-1.8.1 pywatchman-1.3.0 requests-2.5.3 scandir-1.2 setproctitle-1.1.10 six-1.10.0 twitter.common.collections-0.3.9 twitter.common.confluence-0.3.9 twitter.common.dirutil-0.3.9 twitter.common.lang-0.3.9 twitter.common.log-0.3.9 twitter.common.options-0.3.9
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Exception caught: (<class 'pkg_resources.ContextualVersionConflict'>)
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/bin/pants", line 11, in <module>
    sys.exit(main())
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 44, in main
    PantsRunner(exiter).run()
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 57, in run
    options_bootstrapper=options_bootstrapper)
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 46, in _run
    return LocalPantsRunner(exiter, args, env, options_bootstrapper=options_bootstrapper).run()
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 53, in run
    self._maybe_profiled(self._run)
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 50, in _maybe_profiled
    runner()
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 59, in _run
    options, build_config = OptionsInitializer(options_bootstrapper, exiter=self._exiter).setup()
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/options_initializer.py", line 174, in setup
    backends)
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/options_initializer.py", line 83, in _load_plugins
    return load_backends_and_plugins(plugins, working_set, backend_packages)
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/extension_loader.py", line 37, in load_backends_and_plugins
    load_plugins(build_configuration, plugins or [], working_set)
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pants/bin/extension_loader.py", line 83, in load_plugins
    aliases = entries['build_file_aliases'].load()()
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.0/lib/python2.7/site-packages/pkg_resources/__init__.py", line 834, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)

Exception message: (setuptools 21.2.1 (/Users/todd/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.xyWThV/install/lib/python2.7/site-packages), Requirement.parse('setuptools==5.4.1'), set(['pantsbuild.pants']))
mateor commented 7 years ago

Taking a look today - thanks for breaking out such great issues!

mateor commented 7 years ago

I sent a pull request that installs buildgen - although it fails to run due to some assumptions of buildgen - namely that there would be code in the repo ;)

The upstream pants bootstrapping code had changed some...the process is still not great. Some of that is because upstream Pants has changed and the buildgen publish is old.

But this should get you started, I believe. I will probably use your repro as the new reference repo, thanks for creating that.

mateor commented 7 years ago

Whoops - here is the link, future reader: https://github.com/mateor/fsqio-plugin-test/commits/master