dataclasses distribution was not found

[m-stclair]

command-line invocation fails with the error

pkg_resources.DistributionNotFound: The 'dataclasses<1.0.0,>=0.6.0' distribution was not found and is required by nr.parsing.date

this is in python=3.9.6, so dataclasses should be natively available. I've also tried explicitly installing several different versions of dataclasses using conda and pip with no effect.

[NiklasRosenstein]

Hey @m-stclair , thanks for reporting the issue. What is the Pydoc-Markdown version you installed? What did you installt it with? Could you attach the install command and logs? That might help determine what's going on. (Haven't had a chance yet to try and reproduce)

[m-stclair]

the version is 4.1.6. I'm installing it in a conda env via conda and the conda-forge channel, so perhaps that's part of the issue? I get the same result if I install to an env with pip, though. Minimal steps to reproduce (assuming conda is installed in the environment) are:

conda create -n pydoc-markdown-env
conda install -n pydoc-markdown-env pydoc-markdown mkdocs
conda activate pydoc-markdown
pydoc-markdown > api.md

I should note that nr.parsing.date itself seems to be working fine; e.g., from python shell:

>>> from nr.parsing.date import ISO_8601
>>> ISO_8601.parse_datetime('2021-04-21T10:13:00.124+0000')
datetime.datetime(2021, 4, 21, 10, 13, 0, 124000, tzinfo=datetime.timezone.utc)

install output from conda is:

michael@djuna ~/D/l/docs (main)> conda create -n pydoc-markdown pydoc-markdown mkdocs
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/michael/miniconda3/envs/pydoc-markdown

  added / updated specs:
    - mkdocs
    - pydoc-markdown

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
  argh               conda-forge/noarch::argh-0.26.2-pyh9f0ad1d_1002
  brotlipy           conda-forge/linux-64::brotlipy-0.7.0-py39h3811e60_1001
  ca-certificates    conda-forge/linux-64::ca-certificates-2021.5.30-ha878542_0
  certifi            conda-forge/linux-64::certifi-2021.5.30-py39hf3d152e_0
  cffi               conda-forge/linux-64::cffi-1.14.6-py39he32792d_0
  chardet            conda-forge/linux-64::chardet-4.0.0-py39hf3d152e_1
  charset-normalizer conda-forge/noarch::charset-normalizer-2.0.0-pyhd8ed1ab_0
  click              conda-forge/noarch::click-7.1.2-pyh9f0ad1d_0
  cryptography       conda-forge/linux-64::cryptography-3.4.7-py39hbca0aa6_0
  databind.core      conda-forge/noarch::databind.core-1.1.4-pyhd8ed1ab_0
  databind.json      conda-forge/noarch::databind.json-1.1.4-pyhd8ed1ab_0
  dataclasses        conda-forge/noarch::dataclasses-0.8-pyhc8e2a94_1
  deprecated         conda-forge/noarch::deprecated-1.2.12-pyh44b312d_0
  docspec            conda-forge/noarch::docspec-1.0.1-pyhd8ed1ab_0
  docspec-python     conda-forge/noarch::docspec-python-1.0.1-pyhd8ed1ab_0
  ghp-import         conda-forge/noarch::ghp-import-2.0.1-pyhd8ed1ab_0
  idna               conda-forge/noarch::idna-3.1-pyhd3deb0d_0
  importlib-metadata conda-forge/linux-64::importlib-metadata-4.6.3-py39hf3d152e_0
  jinja2             conda-forge/noarch::jinja2-3.0.1-pyhd8ed1ab_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_2
  libffi             conda-forge/linux-64::libffi-3.3-h58526e2_2
  libgcc-ng          conda-forge/linux-64::libgcc-ng-11.1.0-hc902ee8_8
  libgomp            conda-forge/linux-64::libgomp-11.1.0-hc902ee8_8
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-11.1.0-h56837e0_8
  markdown           conda-forge/noarch::markdown-3.3.4-pyhd8ed1ab_0
  markupsafe         conda-forge/linux-64::markupsafe-2.0.1-py39h3811e60_0
  mergedeep          conda-forge/noarch::mergedeep-1.3.4-pyhd8ed1ab_0
  mkdocs             conda-forge/noarch::mkdocs-1.2.2-pyhd8ed1ab_0
  ncurses            conda-forge/linux-64::ncurses-6.2-h58526e2_4
  nr.fs              conda-forge/noarch::nr.fs-1.6.3-pyhd8ed1ab_0
  nr.optional        conda-forge/noarch::nr.optional-0.2.0-pyhd8ed1ab_0
  nr.parsing.date    conda-forge/noarch::nr.parsing.date-1.0.3-pyhd8ed1ab_0
  nr.preconditions   conda-forge/noarch::nr.preconditions-0.0.4-pyhd8ed1ab_0
  nr.pylang.utils    conda-forge/noarch::nr.pylang.utils-0.1.3-pyhd8ed1ab_0
  nr.stream          conda-forge/noarch::nr.stream-0.2.3-pyhd8ed1ab_0
  nr.utils.re        conda-forge/noarch::nr.utils.re-0.3.1-pyhd8ed1ab_0
  openssl            conda-forge/linux-64::openssl-1.1.1k-h7f98852_0
  packaging          conda-forge/noarch::packaging-21.0-pyhd8ed1ab_0
  pip                conda-forge/noarch::pip-21.2.4-pyhd8ed1ab_0
  pycparser          conda-forge/noarch::pycparser-2.20-pyh9f0ad1d_2
  pydoc-markdown     conda-forge/noarch::pydoc-markdown-4.1.6-pyhd8ed1ab_0
  pyopenssl          conda-forge/noarch::pyopenssl-20.0.1-pyhd8ed1ab_0
  pyparsing          conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
  pysocks            conda-forge/linux-64::pysocks-1.7.1-py39hf3d152e_3
  python             conda-forge/linux-64::python-3.9.6-h49503c6_1_cpython
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.2-pyhd8ed1ab_0
  python_abi         conda-forge/linux-64::python_abi-3.9-2_cp39
  pyyaml             conda-forge/linux-64::pyyaml-5.4.1-py39h3811e60_0
  pyyaml-env-tag     conda-forge/noarch::pyyaml-env-tag-0.1-pyhd8ed1ab_0
  readline           conda-forge/linux-64::readline-8.1-h46c0cb4_0
  requests           conda-forge/noarch::requests-2.26.0-pyhd8ed1ab_0
  setuptools         conda-forge/linux-64::setuptools-49.6.0-py39hf3d152e_3
  six                conda-forge/noarch::six-1.16.0-pyh6c4a22f_0
  sqlite             conda-forge/linux-64::sqlite-3.36.0-h9cd32fc_0
  tk                 conda-forge/linux-64::tk-8.6.10-h21135ba_1
  toml               conda-forge/noarch::toml-0.10.2-pyhd8ed1ab_0
  typing-extensions  conda-forge/noarch::typing-extensions-3.10.0.0-hd8ed1ab_0
  typing_extensions  conda-forge/noarch::typing_extensions-3.10.0.0-pyha770c72_0
  tzdata             conda-forge/noarch::tzdata-2021a-he74cb21_1
  urllib3            conda-forge/noarch::urllib3-1.26.6-pyhd8ed1ab_0
  watchdog           conda-forge/linux-64::watchdog-2.1.3-py39hf3d152e_0
  wheel              conda-forge/noarch::wheel-0.37.0-pyhd8ed1ab_0
  wrapt              conda-forge/linux-64::wrapt-1.12.1-py39h3811e60_3
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  yaml               conda-forge/linux-64::yaml-0.2.5-h516909a_0
  zipp               conda-forge/noarch::zipp-3.5.0-pyhd8ed1ab_0
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1010

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

in case it matters, the contents of pydoc-markdown.yml are:

loaders:
  - type: python
    search_path: ['../lhorizon']
    ignore_when_discovered: ['examples', 'shiny_scratch', 'constants', 'tests', '_type_aliases', '__init__']
processors:
  - type: filter
  - type: smart
  - type: crossref
renderer:
  type: markdown
  output_directory: .
  pages:
    - title: API Documentation
      name: index
  mkdocs_config:
    site_name: lhorizon
    theme: readthedocs
  descriptive_module_title: True
  add_method_class_prefix: True
  add_member_class_prefix: True
  signature_with_def: True
  render_toc: True
  toc_maxdepth: 4

It was working without complaint in an older version of the environment, and I get the same error if I run pydoc-markdown with a freshly-bootstrapped .yml file.

[NiklasRosenstein]

Thanks for the details @m-stclair ! I can reproduce the issue in Conda, not in a Virtualenv with Pip though.

I think I've found the root cause, but unsure about the path forward. Posted this in Gitter to ask for help:

Hi, I'm getting a CI check failure on https://github.com/conda-forge/nr.parsing.date-feedstock/pull/2 because pip check complains about dataclasses not being installed. However dataclasses is listed as a a run dependency and you can even see in the CI logs that it is being installed by Conda. Do you have an idea what is going wrong here?

It turns out that Conda does not actually install a dataclasses distribution for you in Python >= 3.7, which makes sense, but that seems to break packages that depend on it. In the above I removed the pip check (which actually raised a valid issue) and so released nr.parsing.date with a requirement on dataclasses, but that now breaks pydoc-markdown which depends on nr.parsing.date where we at some point end up invoking pkg_resources validating that the required distributions are satisfied:

pkg_resources.DistributionNotFound: The 'dataclasses<1.0.0,>=0.6.0' distribution was not found and is required by nr.parsing.date

You can easily reproduce this with

$ conda create -p test python==3.9.* dataclasses
$ test/bin/python -c 'import pkg_resources as p; print(p.get_distribution("dataclasses"))'

which shows a "DistributionNotFound" error. It works when using python==3.6.* or when not using Conda and just installing dataclasses via Pip.

Is this something we can fix? (I.e. even if dataclasses technically doesn't need to be installed, it still needs to be to satisfy requirements being checked in pkg_resources).

[NiklasRosenstein]

@m-stclair The temporary workaround is to downgrade nr.parsing.date to 1.0.2 which does not declare a dependency on dataclasses (even thought technically it does need it if you were on Python 3.6, which you aren't because Pydoc-Markdown requires 3.7+ 😄 )

[m-stclair]

@NiklasRosenstein thanks for looking into this -- what an annoying upstream bug!

so, explicitly downgrading nr.parsing.date to 1.0.2 does bypass the DistributionNotFound exception, but gives me another exception:

TypeError: dataclass_to_schema() missing 1 required positional argument: 'type_hint_adapter'

do I have a version mismatch somewhere wrt your recent migration to databind.core? should I be explicitly installing an older version of pydoc-markdown as well?

[NiklasRosenstein]

Sounds like you have an older version of databind.core installed. This was a bug in in databind.core-1.1.1. Does conda have an option to upgrade all dependencies? Otherwise try conda install databind.core==1.1.4

[m-stclair]

unfortunately, this is occurring with databind.core 1.1.4 in the environment.

[m-stclair]

never mind -- there was somehow a stray older version of dataclasses.py in the env's site-packages directory, probably having to do with install and subsequent downgrade, that was getting imported instead of the version in databind-core. clearing this out removed the problem and pydoc-markdown now works as expected.

[NiklasRosenstein]

Great! Going to close this out then!