search

freedomofpress / securedrop-builder

Packaging logic for building SecureDrop-related Debian packages
GNU General Public License v3.0
7 stars 11 forks source link

"pip download" command tries to build packages without our bootstrap #457

Closed legoktm closed 1 year ago

legoktm commented 1 year ago

Confusingly the pip download command in build-sync-wheels tries to build the packages we're downloading, except it does not use our bootstrap, so if the package is currently broken when building from source (as pyyaml currently is - https://github.com/freedomofpress/securedrop-client/issues/1681) then the download command will fail!

The solution is to pass --no-build-isolation to the download command so it will build the package using our bootstrapped virtualenv, which already contains all the build dependencies at the versions we want.

Collecting pyyaml==5.4.1
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     |████████████████████████████████| 175 kB 4.7 MB/s            
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /src/securedrop-builder/.venv/bin/python3 /src/securedrop-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpf1ky53m4
       cwd: /tmp/pip-download-hkiw_qvw/pyyaml_3005373dbf0e462abbc745dffa4f8bf5
  Complete output (62 lines):
  /tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
  !!

          ********************************************************************************
          The license_file parameter is deprecated, use license_files instead.

          By 2023-Oct-30, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.

          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************

  !!
    parsed = self.parsers.get(option_name, lambda x: x)(value)
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/src/securedrop-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/src/securedrop-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/src/securedrop-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 314, in run
      self.find_sources()
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
      mm.run()
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 551, in run
      self.add_defaults()
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
      super().add_defaults()
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "<string>", line 201, in get_source_files
    File "/tmp/pip-build-env-qfleh5s5/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz#sha256=607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e (from https://pypi.org/simple/pyyaml/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*). Command errored out with exit status 1: /src/securedrop-builder/.venv/bin/python3 /src/securedrop-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpf1ky53m4 Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pyyaml==5.4.1 (from versions: 3.10, 3.11, 3.12, 3.13b1, 3.13rc1, 3.13, 4.2b1, 4.2b2, 4.2b4, 5.1b1, 5.1b3, 5.1b5, 5.1, 5.1.1, 5.1.2, 5.2b1, 5.2, 5.3b1, 5.3, 5.3.1, 5.4b1, 5.4b2, 5.4, 5.4.1, 6.0b1, 6.0, 6.0.1)
ERROR: No matching distribution found for pyyaml==5.4.1
WARNING: You are using pip version 21.3.1; however, version 23.2.1 is available.
You should consider upgrading via the '/src/securedrop-builder/.venv/bin/python3 -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/src/securedrop-builder/./scripts/build-sync-wheels", line 154, in <module>
    main()
  File "/src/securedrop-builder/./scripts/build-sync-wheels", line 110, in main
    subprocess.check_call(cmd)
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip3', 'download', '--no-binary', ':all:', '--require-hashes', '--dest', '/tmp/tmp4hr5dlqp', '--requirement', '../securedrop-proxy/requirements/requirements.txt']' returned non-zero exit status 1.
make: *** [Makefile:51: build-wheels] Error 1
rocodes commented 1 year ago

Thanks for investigating this. Do you think there are any additional guardrails we can/should add (eg in the sync-wheels script, in developer documentation, etc) to avoid potentially contaminated or stale build environments? AIUI making sure the virtual environment is up to date before building is still a manual process.

legoktm commented 1 year ago

Yes -- I think that should be part of containerizing the build process (https://github.com/freedomofpress/securedrop-engineering/pull/20) so we can just automate/script that the venv is up-to-date, it's using a non-root user, etc., instead of implementing checks everywhere and complaining.

rocodes commented 1 year ago

Makes sense. I'm going to approve https://github.com/freedomofpress/securedrop-builder/pull/458 without any additional guardrails then since the readme mentions removing and rebuilding the venv already, and since the container.sh stuff exists and can be adapted. The no-build-isolation feels a bit like an extra footgun, but we have a proposed direction in mind already to mitigate. Thank you!