Scheduler and Worker containers stop after a minio.error.AccessDenied

[Aneoshun]

Hi @vsoch,

I have noticed that after a couple of hours, the Scheduler and Worker containers crash, both throwing this exception: minio.error.AccessDenied: AccessDenied: message: Access Denied

However, uploading and downloading seem to still work.

Do you have an idea of what could be causing this issue? Maybe I missed something in the configuration?

Thanks,

Here are the complete traceback: Scheduler:

  File "/code/manage.py", line 9, in <module>

    execute_from_command_line(sys.argv)

  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line

    utility.execute()

  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 375, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 323, in run_from_argv

    self.execute(*args, **cmd_options)

  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 361, in execute

    self.check()

  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 390, in check

    include_deployment_checks=include_deployment_checks,

  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 377, in _run_checks

    return checks.run_checks(**kwargs)

  File "/usr/local/lib/python3.5/site-packages/django/core/checks/registry.py", line 72, in run_checks

    new_errors = check(app_configs=app_configs)

  File "/usr/local/lib/python3.5/site-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique

    all_namespaces = _load_all_namespaces(resolver)

  File "/usr/local/lib/python3.5/site-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces

    url_patterns = getattr(resolver, 'url_patterns', [])

  File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in __get__

    res = instance.__dict__[self.name] = self.func(instance)

  File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 584, in url_patterns

    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)

  File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in __get__

    res = instance.__dict__[self.name] = self.func(instance)

  File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 577, in urlconf_module

    return import_module(self.urlconf_name)

  File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 985, in _gcd_import

  File "<frozen importlib._bootstrap>", line 968, in _find_and_load

  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 697, in exec_module

  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed

  File "/code/shub/urls.py", line 19, in <module>

    from shub.apps.library import urls as library_urls

  File "/code/shub/apps/library/urls.py", line 12, in <module>

    from shub.apps.library import views

  File "/code/shub/apps/library/views/__init__.py", line 2, in <module>

    from .images import (

  File "/code/shub/apps/library/views/images.py", line 36, in <module>

    from .minio import (

  File "/code/shub/apps/library/views/minio.py", line 59, in <module>

    if not minioClient.bucket_exists(MINIO_BUCKET):

  File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 453, in bucket_exists

    self._url_open('HEAD', bucket_name=bucket_name)

  File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 2080, in _url_open

    object_name).get_exception()

minio.error.AccessDenied: AccessDenied: message: Access Denied

Worker:

Traceback (most recent call last):

  File "/code/manage.py", line 9, in <module>

    execute_from_command_line(sys.argv)

  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line

    utility.execute()

  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 375, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 323, in run_from_argv

    self.execute(*args, **cmd_options)

  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 361, in execute

    self.check()

  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 390, in check

    include_deployment_checks=include_deployment_checks,

  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 377, in _run_checks

    return checks.run_checks(**kwargs)

  File "/usr/local/lib/python3.5/site-packages/django/core/checks/registry.py", line 72, in run_checks

    new_errors = check(app_configs=app_configs)

  File "/usr/local/lib/python3.5/site-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique

    all_namespaces = _load_all_namespaces(resolver)

  File "/usr/local/lib/python3.5/site-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces

    url_patterns = getattr(resolver, 'url_patterns', [])

  File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in __get__

    res = instance.__dict__[self.name] = self.func(instance)

  File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 584, in url_patterns

    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)

  File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in __get__

    res = instance.__dict__[self.name] = self.func(instance)

  File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 577, in urlconf_module

    return import_module(self.urlconf_name)

  File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 985, in _gcd_import

  File "<frozen importlib._bootstrap>", line 968, in _find_and_load

  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 697, in exec_module

  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed

  File "/code/shub/urls.py", line 19, in <module>

    from shub.apps.library import urls as library_urls

  File "/code/shub/apps/library/urls.py", line 12, in <module>

    from shub.apps.library import views

  File "/code/shub/apps/library/views/__init__.py", line 2, in <module>

    from .images import (

  File "/code/shub/apps/library/views/images.py", line 36, in <module>

    from .minio import (

  File "/code/shub/apps/library/views/minio.py", line 59, in <module>

    if not minioClient.bucket_exists(MINIO_BUCKET):

  File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 453, in bucket_exists

    self._url_open('HEAD', bucket_name=bucket_name)

  File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 2080, in _url_open

    object_name).get_exception()

minio.error.AccessDenied: AccessDenied: message: Access Denied```

[vsoch]

Hmm, it might be happening because every couple of hours they run a task, and that's the first time there is an import. Have you tried taking the error message verbatim - that the access is denied, and perhaps we need to bind the same credential exports for the environment? Actually, I think that would work - let me quickly open a PR for you to test.

[vsoch]

Here you go! https://github.com/singularityhub/sregistry/pull/302 Please test that out and let me knowif it fixes the issue. This was likely an oversight because I never left it running long enough to see the containers run any task and poop out :)

[Aneoshun]

Thanks! It's running. I will let you know if it survives until tomorrow morning.

[vsoch]

Closed with #302