search

singularityhub / sregistry

server for storage and management of singularity images
https://singularityhub.github.io/sregistry
Mozilla Public License 2.0
103 stars 42 forks source link

Unable to push to sregistry via command line #350

Open 659-bit opened 3 years ago

659-bit commented 3 years ago

Hi, this is my first bug report on Github, so please tell me, if I missed anything. :) We encountered the following problem with our Sregistry installation: When pushing images to the sregistry via the command line from another host, we receive an http status code 500:

$ singularity push -U cow_latest.sif library://user.name/test/cow:test
WARNING: Skipping container verifying
 0 B / 87.95 MiB [-------------------------------------------------------------------------------------------------------------------------------------------------------------------]   0.00% 0s
FATAL:   Unable to push image to library: request did not succeed: http status code: 500

However, uploading the same images via browser and pulling them images via the command line works without problems.

Sregistry version: 1.1.18 Excerpt of docker-compose logs:

nginx_1      | 10.66.20.3 - - [04/Mar/2021:10:48:55 +0000] "GET /assets/config/config.prod.json HTTP/1.1" 200 521 "-" "Singularity/3.5.3 (Linux amd64) Go/1.12.12" "-"
nginx_1      | 10.66.20.3 - - [04/Mar/2021:10:48:55 +0000] "GET /assets/config/config.prod.json HTTP/1.1" 200 521 "-" "Singularity/3.5.3 (Linux amd64) Go/1.12.12" "-"
uwsgi_1      | [pid: 39|app: 0|req: 25/77] 10.66.20.3 () {32 vars in 442 bytes} [Thu Mar  4 04:48:55 2021] GET /assets/config/config.prod.json => generated 521 bytes in 1 msecs (HTTP/1.1 200) 3 headers in 109 bytes (1 switches on core 0)
uwsgi_1      | GET NamedEntityView
uwsgi_1      | <QueryDict: {}>
uwsgi_1      | user.name
nginx_1      | 10.66.20.3 - - [04/Mar/2021:10:48:56 +0000] "GET /v1/entities/user.name HTTP/1.1" 200 313 "-" "Go-http-client/1.1" "-"
uwsgi_1      | [pid: 37|app: 0|req: 15/78] 10.66.20.3 () {34 vars in 475 bytes} [Thu Mar  4 04:48:56 2021] GET /v1/entities/user.name => generated 313 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 0)
uwsgi_1      | GET GetNamedCollectionView
nginx_1      | 10.66.20.3 - - [04/Mar/2021:10:48:56 +0000] "GET /v1/collections/user.name/test HTTP/1.1" 200 383 "-" "Go-http-client/1.1" "-"
uwsgi_1      | [pid: 39|app: 0|req: 26/79] 10.66.20.3 () {34 vars in 491 bytes} [Thu Mar  4 04:48:56 2021] GET /v1/collections/user.name/test => generated 383 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 1)
uwsgi_1      | GET GetNamedContainerView
nginx_1      | 10.66.20.3 - - [04/Mar/2021:10:48:56 +0000] "GET /v1/containers/user.name/test/cow_old HTTP/1.1" 200 559 "-" "Go-http-client/1.1" "-"
uwsgi_1      | [pid: 39|app: 0|req: 27/80] 10.66.20.3 () {34 vars in 505 bytes} [Thu Mar  4 04:48:56 2021] GET /v1/containers/user.name/test/cow_old => generated 559 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 2)
uwsgi_1      | GET PushNamedContainerView
nginx_1      | 10.66.20.3 - - [04/Mar/2021:10:48:56 +0000] "GET /v1/images/user.name/test/cow_old:sha256.c04264e3dcf5225d24435e4120007f7788c6ac872e93283762a568d183db538c?arch=amd64 HTTP/1.1" 200 559 "-" "Go-http-client/1.1" "-"
uwsgi_1      | [pid: 39|app: 0|req: 28/81] 10.66.20.3 () {34 vars in 662 bytes} [Thu Mar  4 04:48:56 2021] GET /v1/images/user.name/test/cow_old:sha256.c04264e3dcf5225d24435e4120007f7788c6ac872e93283762a568d183db538c?arch=amd64 => generated 559 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 3)
nginx_1      | 10.66.20.3 - - [04/Mar/2021:10:48:56 +0000] "GET /version HTTP/1.1" 200 58 "-" "Go-http-client/1.1" "-"
uwsgi_1      | [pid: 37|app: 0|req: 16/82] 10.66.20.3 () {34 vars in 441 bytes} [Thu Mar  4 04:48:56 2021] GET /version => generated 58 bytes in 1 msecs (HTTP/1.1 200) 5 headers in 141 bytes (1 switches on core 3)
uwsgi_1      | POST RequestPushImageFileView
nginx_1      | 10.66.20.3 - - [04/Mar/2021:10:48:56 +0000] "POST /v2/imagefile/11/_multipart HTTP/1.1" 500 850 "-" "Go-http-client/1.1" "-"
uwsgi_1      | Internal Server Error: /v2/imagefile/11/_multipart
uwsgi_1      | Traceback (most recent call last):
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner
uwsgi_1      |     response = get_response(request)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 115, in _get_response
uwsgi_1      |     response = self.process_exception_by_middleware(e, request)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 113, in _get_response
uwsgi_1      |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
uwsgi_1      |     return view_func(*args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/views/generic/base.py", line 71, in view
uwsgi_1      |     return self.dispatch(request, *args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/ratelimit/mixins.py", line 58, in dispatch
uwsgi_1      |     )(super(RatelimitMixin, self).dispatch)(*args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/ratelimit/decorators.py", line 30, in _wrapped
uwsgi_1      |     return fn(*args, **kw)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 505, in dispatch
uwsgi_1      |     response = self.handle_exception(exc)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 465, in handle_exception
uwsgi_1      |     self.raise_uncaught_exception(exc)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
uwsgi_1      |     raise exc
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 502, in dispatch
uwsgi_1      |     response = handler(request, *args, **kwargs)
uwsgi_1      |   File "./shub/apps/library/views/images.py", line 96, in post
uwsgi_1      |     container = Container.objects.get(id=container_id)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 82, in manager_method
uwsgi_1      |     return getattr(self.get_queryset(), name)(*args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 399, in get
uwsgi_1      |     clone = self.filter(*args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 892, in filter
uwsgi_1      |     return self._filter_or_exclude(False, *args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 910, in _filter_or_exclude
uwsgi_1      |     clone.query.add_q(Q(*args, **kwargs))
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1290, in add_q
uwsgi_1      |     clause, _ = self._add_q(q_object, self.used_aliases)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1318, in _add_q
uwsgi_1      |     split_subq=split_subq, simple_col=simple_col,
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1251, in build_filter
uwsgi_1      |     condition = self.build_lookup(lookups, col, value)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1116, in build_lookup
uwsgi_1      |     lookup = lookup_class(lhs, rhs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/lookups.py", line 20, in __init__
uwsgi_1      |     self.rhs = self.get_prep_lookup()
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/lookups.py", line 70, in get_prep_lookup
uwsgi_1      |     return self.lhs.output_field.get_prep_value(self.rhs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 972, in get_prep_value
uwsgi_1      |     return int(value)
uwsgi_1      | ValueError: invalid literal for int() with base 10: '11/_multipart'
uwsgi_1      | 04:48:56: Internal Server Error: /v2/imagefile/11/_multipart
uwsgi_1      | Traceback (most recent call last):
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner
uwsgi_1      |     response = get_response(request)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 115, in _get_response
uwsgi_1      |     response = self.process_exception_by_middleware(e, request)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 113, in _get_response
uwsgi_1      |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
uwsgi_1      |     return view_func(*args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/views/generic/base.py", line 71, in view
uwsgi_1      |     return self.dispatch(request, *args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/ratelimit/mixins.py", line 58, in dispatch
uwsgi_1      |     )(super(RatelimitMixin, self).dispatch)(*args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/ratelimit/decorators.py", line 30, in _wrapped
uwsgi_1      |     return fn(*args, **kw)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 505, in dispatch
uwsgi_1      |     response = self.handle_exception(exc)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 465, in handle_exception
uwsgi_1      |     self.raise_uncaught_exception(exc)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
uwsgi_1      |     raise exc
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 502, in dispatch
uwsgi_1      |     response = handler(request, *args, **kwargs)
uwsgi_1      |   File "./shub/apps/library/views/images.py", line 96, in post
uwsgi_1      |     container = Container.objects.get(id=container_id)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 82, in manager_method
uwsgi_1      |     return getattr(self.get_queryset(), name)(*args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 399, in get
uwsgi_1      |     clone = self.filter(*args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 892, in filter
uwsgi_1      |     return self._filter_or_exclude(False, *args, **kwargs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 910, in _filter_or_exclude
uwsgi_1      |     clone.query.add_q(Q(*args, **kwargs))
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1290, in add_q
uwsgi_1      |     clause, _ = self._add_q(q_object, self.used_aliases)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1318, in _add_q
uwsgi_1      |     split_subq=split_subq, simple_col=simple_col,
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1251, in build_filter
uwsgi_1      |     condition = self.build_lookup(lookups, col, value)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1116, in build_lookup
uwsgi_1      |     lookup = lookup_class(lhs, rhs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/lookups.py", line 20, in __init__
uwsgi_1      |     self.rhs = self.get_prep_lookup()
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/lookups.py", line 70, in get_prep_lookup
uwsgi_1      |     return self.lhs.output_field.get_prep_value(self.rhs)
uwsgi_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 972, in get_prep_value
uwsgi_1      |     return int(value)
uwsgi_1      | ValueError: invalid literal for int() with base 10: '11/_multipart'
uwsgi_1      | [pid: 38|app: 0|req: 19/83] 10.66.20.3 () {36 vars in 507 bytes} [Thu Mar  4 04:48:56 2021] POST /v2/imagefile/11/_multipart => generated 850 bytes in 12 msecs (HTTP/1.1 500) 4 headers in 142 bytes (1 switches on core 0)
vsoch commented 3 years ago

That looks like a very old version - and I think I might recognize that bug from a long time ago (but can't remember the issue). Could you please try a newer version, 1.1.32 is the current? I believe that but was fixed.