Closed regisb closed 3 years ago
See discussion: https://discuss.overhang.io/t/issue-with-scorm/1126
The tutor-minio package is enabled. When uploading a Scorm package, I sometimes get the following error:
cms_1 | 2020-12-30 10:10:28,212 ERROR 8 [django.request] [user 3] [ip 172.18.0.1] log.py:222 - Internal Server Error: /xblock/block-v1:overhangio+scorm101+alpha+type@scorm+block@4 570ce0de76d40158507b19ea58f8d69/handler/studio_submit cms_1 | Traceback (most recent call last): cms_1 | File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner cms_1 | response = get_response(request) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response cms_1 | response = self.process_exception_by_middleware(e, request) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response cms_1 | response = wrapped_callback(request, *callback_args, **callback_kwargs) cms_1 | File "/opt/pyenv/versions/3.8.6/lib/python3.8/contextlib.py", line 75, in inner cms_1 | return func(*args, **kwds) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view cms_1 | return view_func(request, *args, **kwargs) cms_1 | File "./cms/djangoapps/contentstore/views/component.py", line 479, in component_handler cms_1 | resp = handler_descriptor.handle(handler, req, suffix) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/xblock/mixins.py", line 85, in handle cms_1 | return self.runtime.handle(self, handler_name, request, suffix) cms_1 | File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1453, in handle cms_1 | return super(MetricsMixin, self).handle(block, handler_name, request, suffix=suffix) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/xblock/runtime.py", line 1060, in handle cms_1 | results = handler(request, suffix) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/openedxscorm/scormxblock.py", line 217, in studio_submit cms_1 | self.extract_package(package_file) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/openedxscorm/scormxblock.py", line 275, in extract_package cms_1 | self.storage.save( cms_1 | File "/openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save cms_1 | return self._save(name, content) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save cms_1 | self._save_content(obj, content, parameters=parameters) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content cms_1 | obj.upload_fileobj(content, ExtraArgs=put_parameters) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj cms_1 | return self.meta.client.upload_fileobj( cms_1 | File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj cms_1 | return future.result() cms_1 | File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result cms_1 | return self._coordinator.result() cms_1 | File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result cms_1 | raise self._exception cms_1 | File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__ cms_1 | return self._execute_main(kwargs) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main cms_1 | return_value = self._main(**kwargs) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main cms_1 | client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call cms_1 | return self._make_api_call(operation_name, kwargs) cms_1 | File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 588, in _make_api_call cms_1 | request_dict = self._convert_to_request_dict( cms_1 | File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 643, in _convert_to_request_dict cms_1 | request_dict = self._serializer.serialize_to_request( cms_1 | File "/openedx/venv/lib/python3.8/site-packages/botocore/validate.py", line 291, in serialize_to_request cms_1 | raise ParamValidationError(report=report.generate_report()) cms_1 | botocore.exceptions.ParamValidationError: Parameter validation failed: cms_1 | Invalid type for parameter ContentType, value: b'text/javascript', type: <class 'bytes'>, valid types: <class 'str'>
This is not consistently reproduced. The error occurs in the unzipping step, on js assets.
Further debugging shows that when this error occurs, the mimetypes.guess_type function returns a bytes object, and not a str:
mimetypes.guess_type
bytes
str
import mimetypes # This returns b'text/javascript' mimetypes.guess_type("somefile.js")[0]
This in turns causes s3boto3 request validation to fail.
The b'text/javascript' mimetype is introduced by the django-pipeline package: https://github.com/jazzband/django-pipeline/blob/1.7.0/pipeline/conf.py#L80
b'text/javascript'
The bug was introduced in this PR: https://github.com/jazzband/django-pipeline/pull/297 It turns out that other people face the exact same bug: https://github.com/jazzband/django-pipeline/pull/297#issuecomment-264416094 This bug was later fixed in v2.0.3: https://github.com/jazzband/django-pipeline/pull/715
However, edx-platform depends on django-pipeline<2.0.0: https://github.com/edx/edx-platform/blob/open-release/koa.1/requirements/constraints.txt#L26 It looks like this constraint was introduced for compatibility withg python 3.5, but we don't need this anymore in Koa (python 3.8): https://github.com/edx/edx-platform/pull/24056
Thus, the fix would consist in upgrading django-pipeline to 2.0.3+ upstream.
See discussion: https://discuss.overhang.io/t/issue-with-scorm/1126
The tutor-minio package is enabled. When uploading a Scorm package, I sometimes get the following error:
This is not consistently reproduced. The error occurs in the unzipping step, on js assets.
Further debugging shows that when this error occurs, the
mimetypes.guess_typefunction returns abytesobject, and not astr:This in turns causes s3boto3 request validation to fail.
The
b'text/javascript'mimetype is introduced by the django-pipeline package: https://github.com/jazzband/django-pipeline/blob/1.7.0/pipeline/conf.py#L80The bug was introduced in this PR: https://github.com/jazzband/django-pipeline/pull/297 It turns out that other people face the exact same bug: https://github.com/jazzband/django-pipeline/pull/297#issuecomment-264416094 This bug was later fixed in v2.0.3: https://github.com/jazzband/django-pipeline/pull/715
However, edx-platform depends on django-pipeline<2.0.0: https://github.com/edx/edx-platform/blob/open-release/koa.1/requirements/constraints.txt#L26 It looks like this constraint was introduced for compatibility withg python 3.5, but we don't need this anymore in Koa (python 3.8): https://github.com/edx/edx-platform/pull/24056
Thus, the fix would consist in upgrading django-pipeline to 2.0.3+ upstream.