search

AdaCore / RecordFlux

Formal specification and generation of verifiable binary parsers, message generators and protocol state machines
Apache License 2.0
104 stars 6 forks source link

Source distribution broken #1107

Closed senier closed 2 years ago

senier commented 2 years ago

After the introduction of #1003 our documented way of creating a source distribution does not work anymore.

@treiher Am I missing anything?

$ python3 setup.py sdist
running sdist
running egg_info
writing RecordFlux.egg-info/PKG-INFO
writing dependency_links to RecordFlux.egg-info/dependency_links.txt
writing requirements to RecordFlux.egg-info/requires.txt
writing top-level names to RecordFlux.egg-info/top_level.txt
reading manifest file 'RecordFlux.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'RecordFlux.egg-info/SOURCES.txt'
running check
creating RecordFlux-0.0.0
creating RecordFlux-0.0.0/RecordFlux.egg-info
creating RecordFlux-0.0.0/bin
creating RecordFlux-0.0.0/doc
creating RecordFlux-0.0.0/doc/img
creating RecordFlux-0.0.0/rflx
creating RecordFlux-0.0.0/rflx/generator
creating RecordFlux-0.0.0/rflx/model
creating RecordFlux-0.0.0/rflx/pyrflx
creating RecordFlux-0.0.0/rflx/specification
creating RecordFlux-0.0.0/rflx/templates
copying files to RecordFlux-0.0.0...
copying LICENSE -> RecordFlux-0.0.0
copying MANIFEST.in -> RecordFlux-0.0.0
copying README.md -> RecordFlux-0.0.0
copying SECURITY.md -> RecordFlux-0.0.0
copying pyproject.toml -> RecordFlux-0.0.0
copying setup.py -> RecordFlux-0.0.0
copying RecordFlux.egg-info/PKG-INFO -> RecordFlux-0.0.0/RecordFlux.egg-info
copying RecordFlux.egg-info/SOURCES.txt -> RecordFlux-0.0.0/RecordFlux.egg-info
copying RecordFlux.egg-info/dependency_links.txt -> RecordFlux-0.0.0/RecordFlux.egg-info
copying RecordFlux.egg-info/requires.txt -> RecordFlux-0.0.0/RecordFlux.egg-info
copying RecordFlux.egg-info/top_level.txt -> RecordFlux-0.0.0/RecordFlux.egg-info
copying bin/rflx -> RecordFlux-0.0.0/bin
copying doc/Design-Guide.md -> RecordFlux-0.0.0/doc
copying doc/Development.md -> RecordFlux-0.0.0/doc
copying doc/Language-Reference.adoc -> RecordFlux-0.0.0/doc
copying doc/User-Guide.adoc -> RecordFlux-0.0.0/doc
copying doc/img/logo.svg -> RecordFlux-0.0.0/doc/img
copying rflx/__init__.py -> RecordFlux-0.0.0/rflx
copying rflx/ada.py -> RecordFlux-0.0.0/rflx
copying rflx/cli.py -> RecordFlux-0.0.0/rflx
copying rflx/common.py -> RecordFlux-0.0.0/rflx
copying rflx/const.py -> RecordFlux-0.0.0/rflx
copying rflx/contract.py -> RecordFlux-0.0.0/rflx
copying rflx/error.py -> RecordFlux-0.0.0/rflx
copying rflx/expression.py -> RecordFlux-0.0.0/rflx
copying rflx/graph.py -> RecordFlux-0.0.0/rflx
copying rflx/identifier.py -> RecordFlux-0.0.0/rflx
copying rflx/integration.py -> RecordFlux-0.0.0/rflx
copying rflx/py.typed -> RecordFlux-0.0.0/rflx
copying rflx/typing_.py -> RecordFlux-0.0.0/rflx
copying rflx/validator.py -> RecordFlux-0.0.0/rflx
copying rflx/generator/__init__.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/generator/allocator.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/generator/common.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/generator/const.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/generator/generator.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/generator/message.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/generator/parser.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/generator/serializer.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/generator/session.py -> RecordFlux-0.0.0/rflx/generator
copying rflx/model/__init__.py -> RecordFlux-0.0.0/rflx/model
copying rflx/model/basic_declaration.py -> RecordFlux-0.0.0/rflx/model
copying rflx/model/declaration.py -> RecordFlux-0.0.0/rflx/model
copying rflx/model/message.py -> RecordFlux-0.0.0/rflx/model
copying rflx/model/model.py -> RecordFlux-0.0.0/rflx/model
copying rflx/model/package.py -> RecordFlux-0.0.0/rflx/model
copying rflx/model/session.py -> RecordFlux-0.0.0/rflx/model
copying rflx/model/statement.py -> RecordFlux-0.0.0/rflx/model
copying rflx/model/type_.py -> RecordFlux-0.0.0/rflx/model
copying rflx/pyrflx/__init__.py -> RecordFlux-0.0.0/rflx/pyrflx
copying rflx/pyrflx/bitstring.py -> RecordFlux-0.0.0/rflx/pyrflx
copying rflx/pyrflx/error.py -> RecordFlux-0.0.0/rflx/pyrflx
copying rflx/pyrflx/package.py -> RecordFlux-0.0.0/rflx/pyrflx
copying rflx/pyrflx/pyrflx.py -> RecordFlux-0.0.0/rflx/pyrflx
copying rflx/pyrflx/typevalue.py -> RecordFlux-0.0.0/rflx/pyrflx
copying rflx/pyrflx/utils.py -> RecordFlux-0.0.0/rflx/pyrflx
copying rflx/specification/__init__.py -> RecordFlux-0.0.0/rflx/specification
copying rflx/specification/cache.py -> RecordFlux-0.0.0/rflx/specification
copying rflx/specification/const.py -> RecordFlux-0.0.0/rflx/specification
copying rflx/specification/parser.py -> RecordFlux-0.0.0/rflx/specification
copying rflx/specification/style.py -> RecordFlux-0.0.0/rflx/specification
copying rflx/templates/license_header -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_arithmetic.adb -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_arithmetic.ads -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_builtin_types-conversions.ads -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_builtin_types.ads -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_generic_types.adb -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_generic_types.ads -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_message_sequence.adb -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_message_sequence.ads -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_scalar_sequence.adb -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_scalar_sequence.ads -> RecordFlux-0.0.0/rflx/templates
copying rflx/templates/rflx_types.ads -> RecordFlux-0.0.0/rflx/templates
Writing RecordFlux-0.0.0/setup.cfg
creating dist
Creating tar archive
removing 'RecordFlux-0.0.0' (and everything under it)
$ ls dist/
RecordFlux-0.0.0.tar.gz
treiher commented 2 years ago

I cannot reproduce the problem:

% python3 setup.py sdist                                                                                                                                
running sdist                                                                                                                                                                                                       
running egg_info                                                                                                                                                                                                    
writing RecordFlux.egg-info/PKG-INFO                                                                                                                                                                                
writing dependency_links to RecordFlux.egg-info/dependency_links.txt                                                                                                                                                
writing requirements to RecordFlux.egg-info/requires.txt                                                                                                                                                            
writing top-level names to RecordFlux.egg-info/top_level.txt                                                                                                                                                        
reading manifest template 'MANIFEST.in'                                                                                                                                                                             
adding license file 'LICENSE'                                                                                                                                                                                       
writing manifest file 'RecordFlux.egg-info/SOURCES.txt'                                                                                                                                                             
running check                                                                                                                                                                                                       
creating RecordFlux-0.5.1.dev425+g9e979cc70
[...]

Are you sure that setuptools_scm is installed? What is the output of python -m setuptools_scm? Executing make install_devel should fix the problem if setuptools_scm is missing.

senier commented 2 years ago

It seems to be installed:

$ python -m setuptools_scm
Guessed Version 0.5.1.dev422+g37f5bffa.d20220713

make install_devel doesn't help (in fact, it also produces a 0.0.0 version):

[...]
Installing collected packages: RecordFlux
  Attempting uninstall: RecordFlux
    Found existing installation: RecordFlux 0.0.0
    Uninstalling RecordFlux-0.0.0:
      Successfully uninstalled RecordFlux-0.0.0
  Running setup.py develop for RecordFlux
Successfully installed RecordFlux

I tried both, inside and outside a venv.

senier commented 2 years ago

@treiher When rerunning I get a warning. Maybe that's the problem?

python -m setuptools_scm
/home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/.venv/lib/python3.8/site-packages/setuptools_scm/integration.py:27: RuntimeWarning: 
ERROR: setuptools==44.0.0 is used in combination with setuptools_scm>=6.x

Your build configuration is incomplete and previously worked by accident!
setuptools_scm requires setuptools>=45

This happens as setuptools is unable to replace itself when a activated build dependency
requires a more recent setuptools version
(it does not respect "setuptools>X" in setup_requires).

setuptools>=31 is required for setup.cfg metadata support
setuptools>=42 is required for pyproject.toml configuration support

Suggested workarounds if applicable:
 - preinstalling build dependencies like setuptools_scm before running setup.py
 - installing setuptools_scm using the system package manager to ensure consistency
 - migrating from the deprecated setup_requires mechanism to pep517/518
   and using a pyproject.toml to declare build dependencies
   which are reliably pre-installed before running the build tools

  warnings.warn(
Warning: could not use pyproject.toml, using default configuration.
 Reason: /home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/pyproject.toml does not contain a tool.setuptools_scm section.
/home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/.venv/lib/python3.8/site-packages/setuptools_scm/config.py:61: UserWarning: relative_to is expected to be a file, its the directory '.'
assuming the parent directory was passed
  warnings.warn(
0.5.1.dev422+g37f5bffa.d20220713
treiher commented 2 years ago

This is strange. I never saw such a warning. Maybe switching to build could help. Using setup.py directly is deprecated (it is also recommended to use setup.cfg or only pyproject.toml instead of setup.py). To use build, just execute python -m build after installing it.

treiher commented 2 years ago

@senier Maybe just updating setuptools helps. Here are the versions I'm using:

setuptools          58.1.0
setuptools-scm      7.0.4
senier commented 2 years ago

For some reason, setuptools 44.0.0 are installed in virtualenvs on Ubuntu 20.04. I now upgraded to setuptools 63.1.0 (setuptools_scm 7.0.5), still no luck:

$ python -m setuptools_scm 
Warning: could not use pyproject.toml, using default configuration.
 Reason: /home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/pyproject.toml does not contain a tool.setuptools_scm section.
/home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/.venv/lib/python3.8/site-packages/setuptools_scm/config.py:61: UserWarning: relative_to is expected to be a file, its the directory '.'
assuming the parent directory was passed
  warnings.warn(
0.5.1.dev422+g37f5bffa.d20220713
senier commented 2 years ago

build yields the same result:

[...]
adding 'RecordFlux-0.0.0.dist-info/WHEEL'
adding 'RecordFlux-0.0.0.dist-info/top_level.txt'
adding 'RecordFlux-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built RecordFlux-0.0.0.tar.gz and RecordFlux-0.0.0-py3-none-any.whl
treiher commented 2 years ago

Warning: could not use pyproject.toml, using default configuration. Reason: /home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/pyproject.toml does not contain a tool.setuptools_scm section.

Did you update the submodules, in particular .config/python-style? This warning indicates that you are using an old version of pyproject.toml. There should be a tool.setuptools_scm section.

senier commented 2 years ago

You're right, the submodule was outdated. It still does not work (this is inside a venv):

$ make install_devel
make -C .config/python-style install_devel
make[1]: Entering directory '/home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/.config/python-style'
pip3 install ".[devel]"
Processing /home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/.config/python-style
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/.venv/bin/python3 /tmp/tmp2ho3kmae get_requires_for_build_wheel /tmp/tmpxatqkkcd
       cwd: /tmp/pip-req-build-mr9x8sbu
  Complete output (35 lines):
  Traceback (most recent call last):
    File "/tmp/tmp2ho3kmae", line 280, in <module>
      main()
    File "/tmp/tmp2ho3kmae", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/tmp2ho3kmae", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 1, in <module>
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 139, in setup
      _setup_distribution = dist = klass(attrs)    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 476, in __init__
      _Distribution.__init__(
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 275, in __init__
      self.finalize_options()    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 900, in finalize_options
      ep(self)
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools_scm/integration.py", line 123, in infer_version
      _assign_version(dist, config)
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools_scm/integration.py", line 62, in _assign_version
      _version_missing(config)
    File "/tmp/pip-build-env-h3bk_4ce/overlay/lib/python3.8/site-packages/setuptools_scm/__init__.py", line 109, in _version_missing
      raise LookupError(
  LookupError: setuptools-scm was unable to detect version for /tmp/pip-req-build-mr9x8sbu.

  Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

  For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/.venv/bin/python3 /tmp/tmp2ho3kmae get_requires_for_build_wheel /tmp/tmpxatqkkcd Check the logs for full command output.
make[1]: *** [Makefile:40: install_devel] Error 1
make[1]: Leaving directory '/home/senier/Devel/SPDM_issue_134/contrib/RecordFlux/.config/python-style'
make: *** [Makefile:134: install_devel] Error 2
treiher commented 2 years ago

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

Are you sure there is a correct .git directory?

senier commented 2 years ago

Are you sure there is a correct .git directory?

Yes, contrib/RecordFlux and contrib/RecordFlux/.config/python-style both have .git directories and git status can be used there.

treiher commented 2 years ago

The issue seems to be caused by the used version of pip: 21.2. After an upgrade to 22.1 the problem disappears.

As a consequence of the observed problems, I would propose to require setuptools with at least version 45 in the pyproject.toml. There seems not to be a simple way to enforce a certain version of pip, but we should at least document it. An option would be comparing the output of pip --version or comparing pip.__version__ inside a Python script before/in the install_devel target.