When doing a deploy of repo.ros2.org today, the pulp migration stage failed to boot up with the following stacktrace:
Traceback (most recent call last):
File "/usr/local/lib/pulp/bin/pulpcore-manager", line 11, in <module>
sys.exit(manage())
File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage
execute_from_command_line(sys.argv)
File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 325, in execute
settings.INSTALLED_APPS
File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
self._setup(name)
File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
self._wrapped = Settings(settings_module)
File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/app/settings.py", line 81, in <module>
plugin_app_config = entry_point.load()
File "/usr/local/lib/pulp/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2464, in load
self.require(*args, **kwargs)
File "/usr/local/lib/pulp/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2487, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/lib/pulp/lib/python3.6/site-packages/pkg_resources/__init__.py", line 782, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (redis 3.5.3 (/usr/lib/python3.6/site-packages), Requirement.parse('redis>=4.0.0'), {'rq'})
Our docker build process sources many dependencies from the RHEL 8 and EPEL 8 repositories but although we pin the pip-installed packages we have not pinned any of their dependencies and it seems like a bump in the rq version (annoyingly from minor 1.13 to 1.15 although I'm not their semver dad) has introduced a conflict with the system-installed version of python3-redis. I didn't capture good notes about why jsonschema was upgraded away from the system version (it was copied over during the puppet to chef port) but I didn't take that tactic here because whacking individual packages is just going to create longer term churn and as we're dropping pulp in favor of createrepo-agent we really just need this to run for the sake of the deployment process while we're still decommissioning it.
I was able to capture the pip freeze output from the previously deployed version of pulp_image and I've copied that into the constraints.txt file that is now being considered when running pip install during the docker image build with the intent that a working configuration of python3 packages is locked in for the lifetime of the pulp workflow.
When doing a deploy of repo.ros2.org today, the pulp migration stage failed to boot up with the following stacktrace:
Our docker build process sources many dependencies from the RHEL 8 and EPEL 8 repositories but although we pin the pip-installed packages we have not pinned any of their dependencies and it seems like a bump in the
rq
version (annoyingly from minor 1.13 to 1.15 although I'm not their semver dad) has introduced a conflict with the system-installed version of python3-redis. I didn't capture good notes about why jsonschema was upgraded away from the system version (it was copied over during the puppet to chef port) but I didn't take that tactic here because whacking individual packages is just going to create longer term churn and as we're dropping pulp in favor of createrepo-agent we really just need this to run for the sake of the deployment process while we're still decommissioning it.I was able to capture the
pip freeze
output from the previously deployed version ofpulp_image
and I've copied that into the constraints.txt file that is now being considered when running pip install during the docker image build with the intent that a working configuration of python3 packages is locked in for the lifetime of the pulp workflow.I have also grabbed a copy of the same
pip freeze
output for the image that won't boot a pulp process for future study and saved it here https://gist.github.com/nuclearsandwich/f423d4a679163e395e3898152b05f745 although I don't know that there will be enough of a reason to return to it.