Podman push fails on partial_update

[lubosmj]

Version 2.13 (https://github.com/pulp/pulp_container/commit/29b83b536da78c60d4ab0cc09d6eb58c01bce644#diff-1b944e15118f82233aa03a435e9e05c188ad0d371b71be43260506b4a5eeddffR637)

After issuing the following commands, an error is raised:

pulp container remote create --name lubosmj --url https://registry-1.docker.io --upstream-name lubosmj/mirror
SYNC_REPOSITORY=$(pulp container repository create --name lubosmj --remote lubosmj | jq -r ".pulp_href")
pulp container repository sync --name lubosmj

podman pull ghcr.io/pulp/test-fixture-1:manifest_d
podman tag ghcr.io/pulp/test-fixture-1:manifest_d localhost:24817/pulp/fixture:manifest_d
podman login localhost:24817 --tls-verify=false -u admin -p password
podman push localhost:24817/pulp/fixture:manifest_d --tls-verify=false

Traceback:

File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner
  response = get_response(request)
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response
  response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
  return view_func(*args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
  return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
  response = self.handle_exception(exc)
File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 253, in handle_exception
  response = super().handle_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
  self.raise_uncaught_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
  raise exc
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
  response = handler(request, *args, **kwargs)
File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 637, in partial_update
  length = int(request.headers["Content-Length"])
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/http/request.py", line 424, in __getitem__
  return super().__getitem__(key.replace('_', '-'))
File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/utils/datastructures.py", line 316, in __getitem__
  return self._store[key.lower()][1]
KeyError: 'content-length'

[ipanova]

content-length headers is present during the request. please provide podman version.

[lubosmj]

podman version 3.4.7

[lubosmj]

Same for podman v4.2.0-rc1:

Login Succeeded!
Getting image source signatures
Copying blob f9dd3c50d6a0 done  
Copying blob ace0eda3e3be done  
Error: writing blob: uploading layer chunked: received unexpected HTTP status: 500 Internal Server Error

[ipanova]

(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman --version
podman version 3.4.7
$ podman pull  ghcr.io/pulp/test-fixture-1:manifest_d 
Trying to pull ghcr.io/pulp/test-fixture-1:manifest_d...
Getting image source signatures
Copying blob 197e2155e009 skipped: already exists  
Copying blob d7ab49ce0b75 skipped: already exists  
Copying config 7e0088f881 done  
Writing manifest to image destination
Storing signatures
7e0088f8811ded1257c8e0b7611ffe3f8fa9ad9d72842d51ab86776343170e73
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman tag  ghcr.io/pulp/test-fixture-1:manifest_d pulp3-source-fedora34.fluffy.example.com/man-d
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman login pulp3-source-fedora34.fluffy.example.com
Authenticating with existing credentials for pulp3-source-fedora34.fluffy.example.com
Existing credentials are valid. Already logged in to pulp3-source-fedora34.fluffy.example.com
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman push pulp3-source-fedora34.fluffy.example.com/man-d
Getting image source signatures
Copying blob f9dd3c50d6a0 done  
Copying blob ace0eda3e3be done  
Copying config 7e0088f881 done  
Writing manifest to image destination
Storing signatures
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ echo $?
0

[lubosmj]

I am receiving the reported error on the pulpcore's and pulp_container's main branches. Could there be an issue with TLS? Since you are not using it in your example.

[lubosmj]

Yep, the failures are gone when I use the content URL instead of localhost:24817:

(pulp) [vagrant@pulp3-source-fedora35 tmp]$ podman push pulp3-source-fedora35.localhost.example.com/pulp/test-fixture-1:manifest_a
Getting image source signatures
Copying blob ace0eda3e3be done  
Copying blob c86ba25590b0 done  
Copying config 2ea2d39846 done  
Writing manifest to image destination
Storing signatures

(pulp) [vagrant@pulp3-source-fedora35 tmp]$ podman push localhost:24817/pulp/test-fixture-1:manifest_a --tls-verify=false
Getting image source signatures
Copying blob c86ba25590b0 done  
Copying blob ace0eda3e3be done  
Error: writing blob: uploading layer chunked: received unexpected HTTP status: 500 Internal Server Error

So, this needs to be addressed in some way.

[ipanova]

It appears that content-length header is mandatory for requests with entity-body https://www.oreilly.com/library/view/http-the-definitive/1565925092/ch15s02.html Regardless, this can be easily fixed by extracting the length from artifact.size https://github.com/pulp/pulp_container/blob/main/pulp_container/app/registry_api.py#L664

[newswangerd]

I just ran into this issue as well.

root@32c294636579:/app# podman --version
podman version 3.0.1

Commands:

podman pull alpine
podman image tag alpine localhost:5001/alpine
podman image push --creds admin:admin --tls-verify=false localhost:5001/alpine

[newswangerd]

Here's my traceback btw

api_1          | KeyError: 'content-length'
api_1          | pulp [None]: django.request:ERROR: Internal Server Error: /v2/alpine/blobs/uploads/132f0741-9a84-432b-ae65-70c3bf1fca44
api_1          | Traceback (most recent call last):
api_1          |   File "/venv/lib64/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
api_1          |     response = get_response(request)
api_1          |   File "/venv/lib64/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
api_1          |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
api_1          |   File "/venv/lib64/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
api_1          |     return view_func(*args, **kwargs)
api_1          |   File "/venv/lib64/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
api_1          |     return self.dispatch(request, *args, **kwargs)
api_1          |   File "/venv/lib64/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
api_1          |     response = self.handle_exception(exc)
api_1          |   File "/venv/lib64/python3.8/site-packages/pulp_container/app/registry_api.py", line 253, in handle_exception
api_1          |     response = super().handle_exception(exc)
api_1          |   File "/venv/lib64/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
api_1          |     self.raise_uncaught_exception(exc)
api_1          |   File "/venv/lib64/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
api_1          |     raise exc
api_1          |   File "/venv/lib64/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
api_1          |     response = handler(request, *args, **kwargs)
api_1          |   File "/venv/lib64/python3.8/site-packages/pulp_container/app/registry_api.py", line 637, in partial_update
api_1          |     length = int(request.headers["Content-Length"])
api_1          |   File "/venv/lib64/python3.8/site-packages/django/http/request.py", line 424, in __getitem__
api_1          |     return super().__getitem__(key.replace('_', '-'))
api_1          |   File "/venv/lib64/python3.8/site-packages/django/utils/datastructures.py", line 316, in __getitem__
api_1          |     return self._store[key.lower()][1]
api_1          | KeyError: 'content-length'

[newswangerd]

@ipanova this is blocking our ability to test the pulpcore 3.20 upgrade in galaxy_ng

[ipanova]

@newswangerd thank you for confirming this is reproducible in other environments too. We will work on the fix.