lms pod failing in tutor k8s quickstart

[RajputVaibhav]

Bug description With reference to https://github.com/overhangio/tutor/issues/249#issuecomment-614421217 I am trying to run tutor k8s quickstart and the lms pod keeps on failing. Stated below is the error log I get on running the quickstart command.

Initialising lms...
Waiting for a lms pod to be ready...
kubectl wait --namespace openedx --selector=app.kubernetes.io/instance=openedx-Ddy6VpVQZBuZ5hpH8VudcFdo,app.kubernetes.io/name=lms --for=condition=ContainersReady --timeout=600s pod
pod/lms-5db54cbd7f-gwmc4 condition met
Finding pod name for lms deployment...
kubectl get --namespace openedx --selector=app.kubernetes.io/instance=openedx-Ddy6VpVQZBuZ5hpH8VudcFdo,app.kubernetes.io/name=lms pods -o=jsonpath={.items[0].metadata.name}
kubectl exec --namespace openedx lms-5db54cbd7f-gwmc4 -- sh -e -c dockerize -wait tcp://mysql:3306 -timeout 20s

export DJANGO_SETTINGS_MODULE=$SERVICE_VARIANT.envs.$SETTINGS
echo "Loading settings $DJANGO_SETTINGS_MODULE"

./manage.py lms migrate

./manage.py lms create_oauth2_client \
    "http://androidapp.com" "http://androidapp.com/redirect" public \
    --client_id android --client_secret cB4RjX0Qogdyk4mtG4PsiFmm \
    --trusted

# Fix incorrect uploaded file path
if [ -d /openedx/data/uploads/ ]; then
  if [ -n "$(ls -A /openedx/data/uploads/)" ]; then
    echo "Migrating LMS uploaded files to shared directory"
    mv /openedx/data/uploads/* /openedx/media/
    rm -rf /openedx/data/uploads/
  fi
fi
2020/04/16 05:14:05 Waiting for: tcp://mysql:3306
Loading settings lms.envs.tutor.production
2020/04/16 05:14:10 Connected to tcp://mysql:3306
WARNING:py.warnings:/openedx/edx-platform/lms/djangoapps/courseware/__init__.py:5: DeprecationWarning: Importing 'lms.djangoapps.courseware' as 'courseware' is no longer supported
  warnings.warn("Importing 'lms.djangoapps.courseware' as 'courseware' is no longer supported", DeprecationWarning)

2020-04-16 05:14:14,103 WARNING 238 [enterprise.utils] utils.py:50 - Could not import Registry from third_party_auth.provider
2020-04-16 05:14:14,103 WARNING 238 [enterprise.utils] utils.py:51 - cannot import name _LTI_BACKENDS
command terminated with exit code 137
Error: Command failed with status 137: kubectl exec --namespace openedx lms-5db54cbd7f-gwmc4 -- sh -e -c dockerize -wait tcp://mysql:3306 -timeout 20s

export DJANGO_SETTINGS_MODULE=$SERVICE_VARIANT.envs.$SETTINGS
echo "Loading settings $DJANGO_SETTINGS_MODULE"

./manage.py lms migrate

./manage.py lms create_oauth2_client \
    "http://androidapp.com" "http://androidapp.com/redirect" public \
    --client_id android --client_secret cB4RjX0Qogdyk4mtG4PsiFmm \
    --trusted

# Fix incorrect uploaded file path
if [ -d /openedx/data/uploads/ ]; then
  if [ -n "$(ls -A /openedx/data/uploads/)" ]; then
    echo "Migrating LMS uploaded files to shared directory"
    mv /openedx/data/uploads/* /openedx/media/
    rm -rf /openedx/data/uploads/
  fi
fi

How to reproduce tutor k8s quickstart

Environment Kubernetes cluster setup using 3 EC2 instances with kubeadm.

Additional context lms-worker and mysql pods are in Running state. Following is the log from the lms pod

WARNING:py.warnings:/openedx/edx-platform/lms/djangoapps/courseware/__init__.py:5: DeprecationWarning: Importing 'lms.djangoapps.courseware' as 'courseware' is no longer supported
  warnings.warn("Importing 'lms.djangoapps.courseware' as 'courseware' is no longer supported", DeprecationWarning)

WARNING:py.warnings:/openedx/edx-platform/lms/djangoapps/courseware/__init__.py:5: DeprecationWarning: Importing 'lms.djangoapps.courseware' as 'courseware' is no longer supported
  warnings.warn("Importing 'lms.djangoapps.courseware' as 'courseware' is no longer supported", DeprecationWarning)

2020-04-16 05:14:08,344 WARNING 11 [enterprise.utils] utils.py:50 - Could not import Registry from third_party_auth.provider
2020-04-16 05:14:08,344 WARNING 11 [enterprise.utils] utils.py:51 - cannot import name _LTI_BACKENDS
2020-04-16 05:14:08,520 WARNING 10 [enterprise.utils] utils.py:50 - Could not import Registry from third_party_auth.provider
2020-04-16 05:14:08,520 WARNING 10 [enterprise.utils] utils.py:51 - cannot import name _LTI_BACKENDS
Traceback (most recent call last):
  File "/openedx/venv/bin/gunicorn", line 11, in <module>
    sys.exit(run())
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 166, in run
    super(Application, self).run()
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 71, in run
    Arbiter(self).run()
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 196, in run
    self.halt(reason=inst.reason, exit_status=inst.exit_status)
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 292, in halt
    self.stop()
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 342, in stop
    self.kill_workers(sig)
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 546, in kill_workers
    self.kill_worker(pid, sig)
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 556, in kill_worker
    os.kill(pid, sig)
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 209, in handle_chld
    self.reap_workers()
  File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 459, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

[regisb]

I think we can now close this, as it should be resolved in Koa (Tutor v11.0.0). Please re-open this if I'm wrong.