search

tlocke / scramp

Python implementation of the SCRAM protocol
MIT No Attribution
28 stars 5 forks source link

with github tag version tar.gz package it is not possible to create scramp versionned rpm (missing PKG-INFO) #13

Closed bruno-at-bareos closed 1 year ago

bruno-at-bareos commented 1 year ago

I've found in issue 10 the following comment, but I didn't success to update the build of scramp on openSUSE obs build. https://build.opensuse.org/package/show/home:bruno_friedmann:branches:devel:languages:python/python-scramp

Scramp finish with 0.0.0 which is not good. Beware the build being done on an isolated vm with no remote access we can't download whatever from internet.

So is there a way to setup correctly the version with the rpm ${version} variable ?

          Hi @robert-schmidtke, I think the problem may lie with the way Scramp is packaged for conda. Scramp now finds its version by doing:
try:
    from importlib.metadata import version
except ImportError:
    from importlib_metadata import version

__version__ = version("scramp")

so it uses Scramp's metadata to find the version. The version metadata is created using versioningit. So I think my first thought is that the problem lies with the Conda packaging, but let me know if you find that we need to change something in Scramp.

Originally posted by @tlocke in https://github.com/tlocke/scramp/issues/10#issuecomment-1298775295

bruno-at-bareos commented 1 year ago

build changelog extract 

[   13s] -----------------------------------------------------------------
[   13s] + exec rpmbuild -ba --define '_srcdefattr (-,root,root)' --nosignature --define '_build_create_debug 1' /home/abuild/rpmbuild/SOURCES/python-scramp.spec
[   13s] setting SOURCE_DATE_EPOCH=1670976000
[   13s] Executing(%prep): /usr/bin/bash -e /var/tmp/rpm-tmp.UNUDSW
[   13s] + umask 022
[   13s] + cd /home/abuild/rpmbuild/BUILD
[   13s] + cd /home/abuild/rpmbuild/BUILD
[   13s] + rm -rf scramp-1.4.4
[   13s] + /usr/lib/rpm/rpmuncompress -x /home/abuild/rpmbuild/SOURCES/python-scramp-1.4.4.tar.gz
[   13s] + STATUS=0
[   13s] + '[' 0 -ne 0 ']'
[   13s] + cd scramp-1.4.4
[   13s] + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
[   13s] + RPM_EC=0
[   13s] ++ jobs -p
[   13s] + exit 0
[   13s] Executing(%build): /usr/bin/bash -e /var/tmp/rpm-tmp.jo7abu
[   13s] + umask 022
[   13s] + cd /home/abuild/rpmbuild/BUILD
[   13s] + /usr/bin/rm -rf /home/abuild/rpmbuild/BUILDROOT/python-scramp-1.4.4-0.x86_64
[   13s] ++ dirname /home/abuild/rpmbuild/BUILDROOT/python-scramp-1.4.4-0.x86_64
[   13s] + /usr/bin/mkdir -p /home/abuild/rpmbuild/BUILDROOT
[   13s] + /usr/bin/mkdir /home/abuild/rpmbuild/BUILDROOT/python-scramp-1.4.4-0.x86_64
[   13s] + cd scramp-1.4.4
[   13s] ++ '[' -f _current_flavor ']'
[   13s] ++ true
[   13s] + last_flavor=
[   13s] + '[' -z '' ']'
[   13s] + last_flavor=tmp
[   13s] + '[' tmp '!=' python38 ']'
[   13s] + '[' -d build ']'
[   13s] + '[' -d _build.python38 ']'
[   13s] + echo python38
[   13s] + python_flavor=python38
[   13s] + myargs=
[   13s] + '[' -f './dist/*-none-any.whl' ']'
[   13s] + /usr/bin/python3.8 -mpip wheel --verbose --progress-bar off --disable-pip-version-check --use-pep517 --no-build-isolation --no-deps --wheel-dir ./build .
[   14s] Processing /home/abuild/rpmbuild/BUILD/scramp-1.4.4
[   14s]   Preparing metadata (pyproject.toml): started
[   14s]   Running command Preparing metadata (pyproject.toml)
[   14s]   Traceback (most recent call last):
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/git.py", line 146, in ensure_is_repo
[   14s]       self.read(
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/git.py", line 178, in read
[   14s]       return readcmd("git", *args, cwd=str(self.path), **kwargs)
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/util.py", line 71, in readcmd
[   14s]       s = runcmd(*args, stdout=subprocess.PIPE, text=True, **kwargs).stdout
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/util.py", line 66, in runcmd
[   14s]       return subprocess.run(arglist, **kwargs)
[   14s]     File "/usr/lib64/python3.8/subprocess.py", line 493, in run
[   14s]       with Popen(*popenargs, **kwargs) as process:
[   14s]     File "/usr/lib64/python3.8/subprocess.py", line 858, in __init__
[   14s]       self._execute_child(args, executable, preexec_fn, close_fds,
[   14s]     File "/usr/lib64/python3.8/subprocess.py", line 1704, in _execute_child
[   14s]       raise child_exception_type(errno_num, err_msg, err_filename)
[   14s]   FileNotFoundError: [Errno 2] No such file or directory: 'git'
[   14s]
[   14s]   During handling of the above exception, another exception occurred:
[   14s]
[   14s]   Traceback (most recent call last):
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/core.py", line 257, in run
[   14s]       description = self.do_vcs()
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/core.py", line 321, in do_vcs
[   14s]       description = self.vcs(project_dir=self.project_dir)
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/methods.py", line 162, in __call__
[   14s]       return self.method(params=self.params, **kwargs)
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/git.py", line 225, in describe_git
[   14s]       repo.ensure_is_repo()
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/git.py", line 154, in ensure_is_repo
[   14s]       raise NotVCSError("Git not installed; assuming this isn't a Git repository")
[   14s]   versioningit.errors.NotVCSError: Git not installed; assuming this isn't a Git repository
[   14s]
[   14s]   During handling of the above exception, another exception occurred:
[   14s]
[   14s]   Traceback (most recent call last):
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/core.py", line 539, in get_version_from_pkg_info
[   14s]       Path(project_dir, "PKG-INFO").read_text(encoding="utf-8")
[   14s]     File "/usr/lib64/python3.8/pathlib.py", line 1236, in read_text
[   14s]       with self.open(mode='r', encoding=encoding, errors=errors) as f:
[   14s]     File "/usr/lib64/python3.8/pathlib.py", line 1222, in open
[   14s]       return io.open(self, mode, buffering, encoding, errors, newline,
[   14s]     File "/usr/lib64/python3.8/pathlib.py", line 1078, in _opener
[   14s]       return self._accessor.open(self, flags, mode)
[   14s]   FileNotFoundError: [Errno 2] No such file or directory: '/home/abuild/rpmbuild/BUILD/scramp-1.4.4/PKG-INFO'
[   14s]
[   14s]   During handling of the above exception, another exception occurred:
[   14s]
[   14s]   Traceback (most recent call last):
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/hook.py", line 23, in setuptools_finalizer
[   14s]       report = vgit.run(write=True, fallback=True)
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/core.py", line 280, in run
[   14s]       version=get_version_from_pkg_info(self.project_dir)
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/core.py", line 542, in get_version_from_pkg_info
[   14s]       raise NotSdistError(f"{project_dir} does not contain a PKG-INFO file")
[   14s]   versioningit.errors.NotSdistError: /home/abuild/rpmbuild/BUILD/scramp-1.4.4 does not contain a PKG-INFO file
[   14s]
[   14s]   During handling of the above exception, another exception occurred:
[   14s]
[   14s]   Traceback (most recent call last):
[   14s]     File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
[   14s]       main()
[   14s]     File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
[   14s]       json_out['return_val'] = hook(**hook_input['kwargs'])
[   14s]     File "/usr/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
[   14s]       return hook(metadata_directory, config_settings)
[   14s]     File "/usr/lib/python3.8/site-packages/setuptools/build_meta.py", line 377, in prepare_metadata_for_build_wheel
[   14s]       self.run_setup()
[   14s]     File "/usr/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
[   14s]       exec(code, locals())
[   14s]     File "<string>", line 1, in <module>
[   14s]     File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
[   14s]       return distutils.core.setup(**attrs)
[   14s]     File "/usr/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 147, in setup
[   14s]       _setup_distribution = dist = klass(attrs)
[   14s]     File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 475, in __init__
[   14s]       _Distribution.__init__(
[   14s]     File "/usr/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
[   14s]       self.finalize_options()
[   14s]     File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 899, in finalize_options
[   14s]       ep(self)
[   14s]     File "/usr/lib/python3.8/site-packages/versioningit/hook.py", line 28, in setuptools_finalizer
[   14s]       raise RuntimeError(
[   14s]   RuntimeError:
[   14s]   versioningit could not find a version for the project in /home/abuild/rpmbuild/BUILD/scramp-1.4.4!
[   14s]
[   14s]   You may be installing from a shallow clone, in which case you need to unshallow it first.
[   14s]
[   14s]   Alternatively, you may be installing from a Git archive, which is not supported by default.  Install from a git+https://... URL instead.
[   14s]
[   14s]
[   14s]   error: subprocess-exited-with-error
[   14s]
[   14s]   × Preparing metadata (pyproject.toml) did not run successfully.
[   14s]   │ exit code: 1
[   14s]   ╰─> See above for output.
[   14s]
[   14s]   note: This error originates from a subprocess, and is likely not a problem with pip.
[   14s]   full command: /usr/bin/python3.8 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpntpjz1nq
[   14s]   cwd: /home/abuild/rpmbuild/BUILD/scramp-1.4.4
[   14s]   Preparing metadata (pyproject.toml): finished with status 'error'
[   14s] error: metadata-generation-failed
tlocke commented 1 year ago

Hi @bruno-at-bareos, I'm not sure what to do about this. The modern method of Python packaging has the version held in the metadata of the wheel. But that means that the version isn't necessarily present in the tar.gz. Do you have any ideas? Also, I wonder what other Python packages do?

tlocke commented 1 year ago

I've looked into this a bit more, and actually versioningit should be able to read the version from the PKG-INFO file which is in the sdist. So I'm puzzled by the error you found:

FileNotFoundError: [Errno 2] No such file or directory: '/home/abuild/rpmbuild/BUILD/scramp-1.4.4/PKG-INFO

How is it that this file is missing, and yet it's in the .tar.gz?

tlocke commented 1 year ago

In fact I've just done a test where I downloaded scramp-1.4.4.tar.gz, unpacked it, and then was able to build it with python -m build.

bruno-at-bareos commented 1 year ago

I have to review what the dedicated macros does, and I hope none of them are removing the PKG-INFO. Thanks a lot for your support, I'll post the result here.

bruno-at-bareos commented 1 year ago

I'm a bit puzzled by your report, as here the content of the tar.gz doesn't contain the PKG-INFO

tar -tvf 1.4.4.tar.gz
drwxrwxr-x root/root         0 2022-11-01 18:15 scramp-1.4.4/
drwxrwxr-x root/root         0 2022-11-01 18:15 scramp-1.4.4/.github/
drwxrwxr-x root/root         0 2022-11-01 18:15 scramp-1.4.4/.github/workflows/
-rw-rw-r-- root/root      1025 2022-11-01 18:15 scramp-1.4.4/.github/workflows/test.yml
-rw-rw-r-- root/root      1222 2022-11-01 18:15 scramp-1.4.4/.gitignore
-rw-rw-r-- root/root       906 2022-11-01 18:15 scramp-1.4.4/LICENSE
-rw-rw-r-- root/root     18790 2022-11-01 18:15 scramp-1.4.4/README.rst
-rw-rw-r-- root/root      1533 2022-11-01 18:15 scramp-1.4.4/pyproject.toml
drwxrwxr-x root/root         0 2022-11-01 18:15 scramp-1.4.4/scramp/
-rw-rw-r-- root/root       335 2022-11-01 18:15 scramp-1.4.4/scramp/__init__.py
-rw-rw-r-- root/root     22503 2022-11-01 18:15 scramp-1.4.4/scramp/core.py
-rw-rw-r-- root/root       655 2022-11-01 18:15 scramp-1.4.4/scramp/utils.py
-rw-rw-r-- root/root       158 2022-11-01 18:15 scramp-1.4.4/setup.cfg
drwxrwxr-x root/root         0 2022-11-01 18:15 scramp-1.4.4/test/
-rw-rw-r-- root/root       222 2022-11-01 18:15 scramp-1.4.4/test/test_readme.py
-rw-rw-r-- root/root     18116 2022-11-01 18:15 scramp-1.4.4/test/test_scramp.py

And when I extract this locally and try to build it I got this error message

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/versioningit/git.py", line 146, in ensure_is_repo
    self.read(
  File "/usr/lib/python3.10/site-packages/versioningit/git.py", line 178, in read
    return readcmd("git", *args, cwd=str(self.path), **kwargs)
  File "/usr/lib/python3.10/site-packages/versioningit/util.py", line 71, in readcmd
    s = runcmd(*args, stdout=subprocess.PIPE, text=True, **kwargs).stdout
  File "/usr/lib/python3.10/site-packages/versioningit/util.py", line 66, in runcmd
    return subprocess.run(arglist, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'rev-parse', '--is-inside-work-tree']' returned non-zero exit status 128.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/versioningit/core.py", line 257, in run
    description = self.do_vcs()
  File "/usr/lib/python3.10/site-packages/versioningit/core.py", line 321, in do_vcs
    description = self.vcs(project_dir=self.project_dir)
  File "/usr/lib/python3.10/site-packages/versioningit/methods.py", line 162, in __call__
    return self.method(params=self.params, **kwargs)
  File "/usr/lib/python3.10/site-packages/versioningit/git.py", line 225, in describe_git
    repo.ensure_is_repo()
  File "/usr/lib/python3.10/site-packages/versioningit/git.py", line 156, in ensure_is_repo
    raise NotVCSError(f"{self.path} is not in a Git repository")
versioningit.errors.NotVCSError: /ioda/home/bruno/openSUSE/obs/home:bruno_friedmann:branches:devel:languages:python/python-scramp/scramp-1.4.4 is not in a Git repository

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/versioningit/core.py", line 539, in get_version_from_pkg_info
    Path(project_dir, "PKG-INFO").read_text(encoding="utf-8")
  File "/usr/lib64/python3.10/pathlib.py", line 1134, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/lib64/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/ioda/home/bruno/openSUSE/obs/home:bruno_friedmann:branches:devel:languages:python/python-scramp/scramp-1.4.4/PKG-INFO'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/versioningit/hook.py", line 23, in setuptools_finalizer
    report = vgit.run(write=True, fallback=True)
  File "/usr/lib/python3.10/site-packages/versioningit/core.py", line 280, in run
    version=get_version_from_pkg_info(self.project_dir)
  File "/usr/lib/python3.10/site-packages/versioningit/core.py", line 542, in get_version_from_pkg_info
    raise NotSdistError(f"{project_dir} does not contain a PKG-INFO file")
versioningit.errors.NotSdistError: /ioda/home/bruno/openSUSE/obs/home:bruno_friedmann:branches:devel:languages:python/python-scramp/scramp-1.4.4 does not contain a PKG-INFO file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.10/site-packages/build/__main__.py", line 22, in <module>
    from build.env import IsolatedEnvBuilder
  File "/usr/lib/python3.10/site-packages/build/env.py", line 23, in <module>
    import virtualenv
  File "/usr/lib/python3.10/site-packages/virtualenv/__init__.py", line 1, in <module>
    from .run import cli_run, session_via_cli
  File "/usr/lib/python3.10/site-packages/virtualenv/run/__init__.py", line 12, in <module>
    from .plugin.creators import CreatorSelector
  File "/usr/lib/python3.10/site-packages/virtualenv/run/plugin/creators.py", line 4, in <module>
    from virtualenv.create.via_global_ref.builtin.builtin_way import VirtualenvBuiltin
  File "/usr/lib/python3.10/site-packages/virtualenv/create/via_global_ref/builtin/builtin_way.py", line 3, in <module>
    from virtualenv.create.creator import Creator
  File "/usr/lib/python3.10/site-packages/virtualenv/create/creator.py", line 11, in <module>
    from virtualenv.discovery.cached_py_info import LogCmd
  File "/usr/lib/python3.10/site-packages/virtualenv/discovery/cached_py_info.py", line 23, in <module>
    _CACHE[Path(sys.executable)] = PythonInfo()
  File "/usr/lib/python3.10/site-packages/virtualenv/discovery/py_info.py", line 103, in __init__
    self.distutils_install = {u(k): u(v) for k, v in self._distutils_install().items()}
  File "/usr/lib/python3.10/site-packages/virtualenv/discovery/py_info.py", line 186, in _distutils_install
    d = dist.Distribution({"script_args": "--no-user-cfg"})  # conf files not parsed so they do not hijack paths
  File "/usr/lib/python3.10/site-packages/setuptools/dist.py", line 475, in __init__
    _Distribution.__init__(
  File "/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
    self.finalize_options()
  File "/usr/lib/python3.10/site-packages/setuptools/dist.py", line 899, in finalize_options
    ep(self)
  File "/usr/lib/python3.10/site-packages/versioningit/hook.py", line 28, in setuptools_finalizer
    raise RuntimeError(
RuntimeError:
versioningit could not find a version for the project in /ioda/home/bruno/openSUSE/obs/home:bruno_friedmann:branches:devel:languages:python/python-scramp/scramp-1.4.4!

You may be installing from a shallow clone, in which case you need to unshallow it first.

Alternatively, you may be installing from a Git archive, which is not supported by default.  Install from a git+https://... URL instead.

So I wonder why there's a such difference compared to your experience.

tlocke commented 1 year ago

A mystery! So I copied the download link from https://pypi.org/project/scramp/#files and then did:

curl --remote-name https://files.pythonhosted.org/packages/de/26/64cb6f2564c79b8e4e8809e844c9fe464a948221c0b2d173cb9586d06e38/scramp-1.4.4.tar.gz

tar -ztvf scramp-1.4.4.tar.gz

drwxrwxr-x tlocke/tlocke     0 2022-11-01 17:17 scramp-1.4.4/
-rw-rw-r-- tlocke/tlocke   906 2022-10-22 14:23 scramp-1.4.4/LICENSE
-rw-rw-r-- tlocke/tlocke 19808 2022-11-01 17:17 scramp-1.4.4/PKG-INFO
-rw-rw-r-- tlocke/tlocke 18790 2022-11-01 17:14 scramp-1.4.4/README.rst
-rw-rw-r-- tlocke/tlocke  1533 2022-10-22 14:44 scramp-1.4.4/pyproject.toml
drwxrwxr-x tlocke/tlocke     0 2022-11-01 17:17 scramp-1.4.4/scramp/
-rw-rw-r-- tlocke/tlocke   335 2022-05-20 11:43 scramp-1.4.4/scramp/__init__.py
-rw-rw-r-- tlocke/tlocke 22503 2022-11-01 17:11 scramp-1.4.4/scramp/core.py
-rw-rw-r-- tlocke/tlocke   655 2021-07-26 10:16 scramp-1.4.4/scramp/utils.py
drwxrwxr-x tlocke/tlocke     0 2022-11-01 17:17 scramp-1.4.4/scramp.egg-info/
-rw-rw-r-- tlocke/tlocke 19808 2022-11-01 17:17 scramp-1.4.4/scramp.egg-info/PKG-INFO
-rw-rw-r-- tlocke/tlocke   282 2022-11-01 17:17 scramp-1.4.4/scramp.egg-info/SOURCES.txt
-rw-rw-r-- tlocke/tlocke     1 2022-11-01 17:17 scramp-1.4.4/scramp.egg-info/dependency_links.txt
-rw-rw-r-- tlocke/tlocke    69 2022-11-01 17:17 scramp-1.4.4/scramp.egg-info/requires.txt
-rw-rw-r-- tlocke/tlocke     7 2022-11-01 17:17 scramp-1.4.4/scramp.egg-info/top_level.txt
-rw-rw-r-- tlocke/tlocke   197 2022-11-01 17:17 scramp-1.4.4/setup.cfg
drwxrwxr-x tlocke/tlocke     0 2022-11-01 17:17 scramp-1.4.4/test/
-rw-rw-r-- tlocke/tlocke   222 2022-05-20 11:43 scramp-1.4.4/test/test_readme.py
-rw-rw-r-- tlocke/tlocke 18116 2022-11-01 17:11 scramp-1.4.4/test/test_scramp.py

so the PKG-INFO file is there in my listing. Does that help solve it?

bruno-at-bareos commented 1 year ago

My bad the title is misleading, the tar.gz I use was from github because several release in the past the pypi was missing the tests part. (I don't know why there's difference between both tar.gz) So I will give pypi a new try.

tlocke commented 1 year ago

So for the releases on PyPI (both the source distribution and the wheel) the version is in the package's metadata. For the Github snapshot of the code there is no metadata, and so build tools can't determine the version. When a release is created, Scramp uses a tool called versioningit to find the version from a git tag, and so versioningit requires a git repository rather than just a snapshot. Hope that explains the results that you're seeing.

bruno-at-bareos commented 1 year ago

Yes Finally make sense, we packager are often caught between a rock and a hard place, I'm sorry for the noise, after you add fixed the missing tests in pypi package I should have changed the source. Maybe you can envisage to create a release tarball also for github in the future too. Thanks a lot for you help.

tlocke commented 1 year ago

Great, glad it's sorted out!