Packaging bug, include rfc3987 explicitly?

[david4096]

Just installed from master using pip3 and got

david@david-t480 ~/g/identifier-interoperability> ga4gh_dos_server
Traceback (most recent call last):
  File "/home/david/.local/bin/ga4gh_dos_server", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/david/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3095, in <module>
    @_call_aside
  File "/home/david/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3079, in _call_aside
    f(*args, **kwargs)
  File "/home/david/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3108, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/david/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 570, in _build_master
    ws.require(__requires__)
  File "/home/david/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 888, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/david/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 774, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'rfc3987; extra == "format"' distribution was not found and is required by jsonschema

Looks like we need to explicitly include this distribution. This should be floated up to bravado, which I believe is responsible for this bug.

[natanlao]

I cannot reproduce this on 5364b06 with a Python 3 environment or a Python 2 environment.

[DailyDreaming]

I just successfully reproduced this bug with a wes-service installation (and bravado dependency).

[natanlao]

@DailyDreaming Can you provide commands and a traceback? Is it similar to pradyunsg/zazo#4?

[natanlao]

$ virtualenv 137-test -p python3
$ . 137-test/bin/activate.fish
$ pip3 install ,
$ ga4gh_dos_server
 * Serving Flask app "ga4gh.dos.server" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
...

$ pip show bravado
Name: bravado
Version: 9.2.2
Summary: Library for accessing Swagger-enabled API's
Home-page: https://github.com/Yelp/bravado
Author: Digium, Inc. and Yelp, Inc.
Author-email: opensource+bravado@yelp.com
License: BSD 3-Clause License
Location: /home/natan/cgl/data-object-service-schemas/137-test/lib/python3.6/site-packages
Requires: bravado-core, python-dateutil, msgpack-python, pyyaml, six, requests
Required-by: ga4gh-dos-schemas

$ python --version
Python 3.6.3

$ python -c "import ga4gh.dos; print(ga4gh.dos.__version__)"
0.3.1

[natanlao]

Nice. I can now reproduce this.

$ python3 -m venv 137-test
$ . 137-test/bin/activate.fish
$ pip3 install ga4gh-dos-schemas
$ ga4gh_dos_server
Traceback (most recent call last):
  File "/home/natan/cgl/data-object-service-schemas/137-test/bin/ga4gh_dos_server", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/natan/cgl/data-object-service-schemas/137-test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3019, in <module>
    @_call_aside
  File "/home/natan/cgl/data-object-service-schemas/137-test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside
    f(*args, **kwargs)
  File "/home/natan/cgl/data-object-service-schemas/137-test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3032, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/natan/cgl/data-object-service-schemas/137-test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 655, in _build_master
    ws.require(__requires__)
  File "/home/natan/cgl/data-object-service-schemas/137-test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 963, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/natan/cgl/data-object-service-schemas/137-test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 849, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'rfc3987; extra == "format"' distribution was not found and is required by jsonschema

[natanlao]

pkg_resources.DistributionNotFound: The 'webcolors; extra == "format"' distribution was not found and is required by jsonschema

[natanlao]

I believe this bug to be a manifestation of pypa/pip#4957. Here is the pipdeptree output for jsonschema:

$ pipdeptree --reverse --packages jsonschema
jsonschema==2.6.0
  - bravado-core==4.13.4 [requires: jsonschema>=2.5.1]
    - bravado==9.2.2 [requires: bravado-core>=4.11.0]
      - ga4gh-dos-schemas==0.4.2 [requires: bravado==9.2.2]
    - ga4gh-dos-schemas==0.4.2 [requires: bravado-core==4.13.4]
  - connexion==1.4.2 [requires: jsonschema>=2.5.1]
    - ga4gh-dos-schemas==0.4.2 [requires: connexion==1.4.2]
  - swagger-spec-validator==2.4.1 [requires: jsonschema]
    - bravado-core==4.13.4 [requires: swagger-spec-validator>=2.0.1]
      - bravado==9.2.2 [requires: bravado-core>=4.11.0]
        - ga4gh-dos-schemas==0.4.2 [requires: bravado==9.2.2]
      - ga4gh-dos-schemas==0.4.2 [requires: bravado-core==4.13.4]
    - connexion==1.4.2 [requires: swagger-spec-validator>=2.0.2]
      - ga4gh-dos-schemas==0.4.2 [requires: connexion==1.4.2]

Yelp/bravado-core requires jsonschema[format]>=2.5.1 and zalando/connexion requires jsonschema>=2.5.1. These dependencies present the necessary conditions for the aforementioned pip bug - essentially, because both jsonschema and jsonschema[format] are specified as dependencies, only the former will be installed.

Doing pip install jsonschema[format] solves the issue. In the short term, this can be solved by manually specifying the dependencies of jsonschema[format] in our setup.py.