Closed DiegoGallegos4 closed 5 years ago
regisb
commented
5 years ago @DiegoGallegos4 thanks for reporting this! As you mentioned, this can be "solved" by running the k8s quickstart again later. Of course, this is not a proper fix, and we need a more appropriate way of deploying to k8s.
DiegoGallegos4
commented
5 years ago Its also a little more than that, I haven't been able to run it at all.
First I run:
tutor k8s quickstart and it fails as described above.
Then trying to run migrations tutor k8s databases it timeouts:
2019-03-21 09:32:27,819 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10a6ae4e0>: Failed to establish a new connection: [Errno 60] Operation timed out',)': /api/v1/namespaces/openedx/pods?labelSelector=app%3Dlms
2019-03-21 09:33:46,650 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10a6ae5f8>: Failed to establish a new connection: [Errno 60] Operation timed out',)': /api/v1/namespaces/openedx/pods?labelSelector=app%3Dlms
2019-03-21 09:35:02,918 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10a6ae6d8>: Failed to establish a new connection: [Errno 60] Operation timed out',)': /api/v1/namespaces/openedx/pods?labelSelector=app%3Dlms
Traceback (most recent call last):
File "urllib3/connection.py", line 159, in _new_conn
File "urllib3/util/connection.py", line 80, in create_connection
File "urllib3/util/connection.py", line 70, in create_connection
TimeoutError: [Errno 60] Operation timed out
natea
commented
5 years ago First I got this error, similar to the one @DiegoGallegos4 got:
$ tutor k8s quickstart
==================================================
Interactive platform configuration
==================================================
Your website domain name for students (LMS) [www.myopenedx.com] www.openedx.com
Your website domain name for teachers (CMS) [studio.openedx.com]
Your platform name/title [My Open edX] Open edX
Your public contact email address [nate@appsembler.com]
The default language code for the platform [en]
Activate SSL/TLS certificates for HTTPS access? Important note:this will NOT work in a development environment. [yN]
Activate Student Notes service (https://open.edx.org/features/student-notes)? [yN]
Activate Xqueue for external grader services (https://github.com/edx/xqueue)? [yN]
Configuration saved to /Users/nateaune/Library/Application Support/tutor/config.yml
Environment generated in /Users/nateaune/Library/Application Support/tutor/env
==============================================
Stopping any existing platform
==============================================
kubectl delete deployments,services,ingress,configmaps --all --namespace openedx
No resources found
=====================================
Starting the platform
=====================================
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/namespace.yml
namespace/openedx created
kubectl create configmap nginx-config --from-file /Users/nateaune/Library/Application Support/tutor/env/apps/nginx --namespace openedx
configmap/nginx-config created
kubectl create configmap mysql-config --from-env-file /Users/nateaune/Library/Application Support/tutor/env/apps/mysql/auth.env --namespace openedx
configmap/mysql-config created
kubectl create configmap openedx-settings-lms --from-file /Users/nateaune/Library/Application Support/tutor/env/apps/openedx/settings/lms --namespace openedx
configmap/openedx-settings-lms created
kubectl create configmap openedx-settings-cms --from-file /Users/nateaune/Library/Application Support/tutor/env/apps/openedx/settings/cms --namespace openedx
configmap/openedx-settings-cms created
kubectl create configmap openedx-config --from-file /Users/nateaune/Library/Application Support/tutor/env/apps/openedx/config --namespace openedx
configmap/openedx-config created
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/volumes.yml --namespace openedx
persistentvolumeclaim/cms-data created
persistentvolumeclaim/lms-data created
persistentvolumeclaim/elasticsearch created
persistentvolumeclaim/mysql created
persistentvolumeclaim/openedx-staticfiles created
persistentvolumeclaim/rabbitmq created
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/ingress.yml --namespace openedx
ingress.extensions/web created
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/services.yml --namespace openedx
service/cms created
service/forum created
service/lms created
service/elasticsearch created
service/memcached created
service/mongodb created
service/mysql created
service/nginx created
service/rabbitmq created
service/smtp created
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/deployments.yml --namespace openedx
deployment.apps/cms created
deployment.apps/forum created
deployment.apps/lms created
deployment.apps/elasticsearch created
deployment.apps/memcached created
deployment.apps/mongodb created
deployment.apps/mysql created
deployment.apps/smtp created
deployment.apps/nginx created
deployment.apps/rabbitmq created
===============================================================================================================
Running migrations. NOTE: this might fail. If it does, please retry 'tutor k9s databases later'
===============================================================================================================
Creating lms/cms databases...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEIruqUV9/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it lms-6c479589fd-22x4s -- bash -e -c dockerize -wait tcp://mysql:3306 -timeout 20s
mysql -u root --password="NME7mivi" --host "mysql" -e 'CREATE DATABASE IF NOT EXISTS openedx;'
mysql -u root --password="NME7mivi" --host "mysql" -e 'GRANT ALL ON openedx.* TO "openedx"@"%" IDENTIFIED BY "NME7mivi";'
Error from server (BadRequest): pod lms-6c479589fd-22x4s does not have a host assigned
Running lms migrations...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEIruqUV9/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it lms-6c479589fd-22x4s -- bash -e -c dockerize -wait tcp://mysql:3306 -timeout 20s && ./manage.py lms migrate
Error from server (BadRequest): pod lms-6c479589fd-22x4s does not have a host assigned
Running cms migrations...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEIruqUV9/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it cms-6b47564c5f-kfqtm -- bash -e -c dockerize -wait tcp://mysql:3306 -timeout 20s && ./manage.py cms migrate
Error from server (BadRequest): pod cms-6b47564c5f-kfqtm does not have a host assigned
Running forum migrations...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEIruqUV9/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it forum-55bfb875cf-qmtrg -- bash -e -c bundle exec rake search:initialize && bundle exec rake search:rebuild_index
error: unable to upgrade connection: container not found ("forum")
Creating oauth2 users...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEIruqUV9/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it lms-6c479589fd-22x4s -- bash -e -c ./manage.py lms create_oauth2_client \
"http://androidapp.com" "http://androidapp.com/redirect" public \
--client_id android --client_secret MgEEmkW6IM2WWHiYvEPialnE \
--trusted
Error from server (BadRequest): pod lms-6c479589fd-22x4s does not have a host assigned
Databases ready.then I re-ran the tutor k8s quickstart command as suggested above, and it got further, but still had an error:
$ tutor k8s quickstart
==================================================
Interactive platform configuration
==================================================
Your website domain name for students (LMS) [www.openedx.com]
Your website domain name for teachers (CMS) [studio.openedx.com]
Your platform name/title [Open edX]
Your public contact email address [nate@appsembler.com]
The default language code for the platform [en]
Activate SSL/TLS certificates for HTTPS access? Important note:this will NOT work in a development environment. [yN]
Activate Student Notes service (https://open.edx.org/features/student-notes)? [yN]
Activate Xqueue for external grader services (https://github.com/edx/xqueue)? [yN]
Configuration saved to /Users/nateaune/Library/Application Support/tutor/config.yml
Environment generated in /Users/nateaune/Library/Application Support/tutor/env
==============================================
Stopping any existing platform
==============================================
kubectl delete deployments,services,ingress,configmaps --all --namespace openedx
deployment.extensions "cms" deleted
deployment.extensions "elasticsearch" deleted
deployment.extensions "forum" deleted
deployment.extensions "lms" deleted
deployment.extensions "memcached" deleted
deployment.extensions "mongodb" deleted
deployment.extensions "mysql" deleted
deployment.extensions "nginx" deleted
deployment.extensions "rabbitmq" deleted
deployment.extensions "smtp" deleted
service "cms" deleted
service "elasticsearch" deleted
service "forum" deleted
service "lms" deleted
service "memcached" deleted
service "mongodb" deleted
service "mysql" deleted
service "nginx" deleted
service "rabbitmq" deleted
service "smtp" deleted
ingress.extensions "web" deleted
configmap "mysql-config" deleted
configmap "nginx-config" deleted
configmap "openedx-config" deleted
configmap "openedx-settings-cms" deleted
configmap "openedx-settings-lms" deleted
=====================================
Starting the platform
=====================================
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/namespace.yml
Error from server (AlreadyExists): error when creating "/Users/nateaune/Library/Application Support/tutor/env/k8s/namespace.yml": namespaces "openedx" already exists
kubectl create configmap nginx-config --from-file /Users/nateaune/Library/Application Support/tutor/env/apps/nginx --namespace openedx
configmap/nginx-config created
kubectl create configmap mysql-config --from-env-file /Users/nateaune/Library/Application Support/tutor/env/apps/mysql/auth.env --namespace openedx
configmap/mysql-config created
kubectl create configmap openedx-settings-lms --from-file /Users/nateaune/Library/Application Support/tutor/env/apps/openedx/settings/lms --namespace openedx
configmap/openedx-settings-lms created
kubectl create configmap openedx-settings-cms --from-file /Users/nateaune/Library/Application Support/tutor/env/apps/openedx/settings/cms --namespace openedx
configmap/openedx-settings-cms created
kubectl create configmap openedx-config --from-file /Users/nateaune/Library/Application Support/tutor/env/apps/openedx/config --namespace openedx
configmap/openedx-config created
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/volumes.yml --namespace openedx
Error from server (AlreadyExists): error when creating "/Users/nateaune/Library/Application Support/tutor/env/k8s/volumes.yml": persistentvolumeclaims "cms-data" already exists
Error from server (AlreadyExists): error when creating "/Users/nateaune/Library/Application Support/tutor/env/k8s/volumes.yml": persistentvolumeclaims "lms-data" already exists
Error from server (AlreadyExists): error when creating "/Users/nateaune/Library/Application Support/tutor/env/k8s/volumes.yml": persistentvolumeclaims "elasticsearch" already exists
Error from server (AlreadyExists): error when creating "/Users/nateaune/Library/Application Support/tutor/env/k8s/volumes.yml": persistentvolumeclaims "mysql" already exists
Error from server (AlreadyExists): error when creating "/Users/nateaune/Library/Application Support/tutor/env/k8s/volumes.yml": persistentvolumeclaims "openedx-staticfiles" already exists
Error from server (AlreadyExists): error when creating "/Users/nateaune/Library/Application Support/tutor/env/k8s/volumes.yml": persistentvolumeclaims "rabbitmq" already exists
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/ingress.yml --namespace openedx
ingress.extensions/web created
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/services.yml --namespace openedx
service/cms created
service/forum created
service/lms created
service/elasticsearch created
service/memcached created
service/mongodb created
service/mysql created
service/nginx created
service/rabbitmq created
service/smtp created
kubectl create -f /Users/nateaune/Library/Application Support/tutor/env/k8s/deployments.yml --namespace openedx
deployment.apps/cms created
deployment.apps/forum created
deployment.apps/lms created
deployment.apps/elasticsearch created
deployment.apps/memcached created
deployment.apps/mongodb created
deployment.apps/mysql created
deployment.apps/smtp created
deployment.apps/nginx created
deployment.apps/rabbitmq created
===============================================================================================================
Running migrations. NOTE: this might fail. If it does, please retry 'tutor k9s databases later'
===============================================================================================================
Creating lms/cms databases...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEILRSXT1/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it lms-6c479589fd-pvw2k -- bash -e -c dockerize -wait tcp://mysql:3306 -timeout 20s
mysql -u root --password="NME7mivi" --host "mysql" -e 'CREATE DATABASE IF NOT EXISTS openedx;'
mysql -u root --password="NME7mivi" --host "mysql" -e 'GRANT ALL ON openedx.* TO "openedx"@"%" IDENTIFIED BY "NME7mivi";'
error: unable to upgrade connection: container not found ("lms")
Running lms migrations...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEILRSXT1/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it lms-6c479589fd-pvw2k -- bash -e -c dockerize -wait tcp://mysql:3306 -timeout 20s && ./manage.py lms migrate
error: unable to upgrade connection: container not found ("lms")
Running cms migrations...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEILRSXT1/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it cms-6b47564c5f-kfqtm -- bash -e -c dockerize -wait tcp://mysql:3306 -timeout 20s && ./manage.py cms migrate
error: unable to upgrade connection: container not found ("cms")
Running forum migrations...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEILRSXT1/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it forum-55bfb875cf-npfsn -- bash -e -c bundle exec rake search:initialize && bundle exec rake search:rebuild_index
error: unable to upgrade connection: container not found ("forum")
Creating oauth2 users...
/var/folders/py/_tyj2dfn2yqc5mvgqlwj_d6w0000gn/T/_MEILRSXT1/kubernetes/config/kube_config.py:496: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
kubectl exec --namespace openedx -it lms-6c479589fd-pvw2k -- bash -e -c ./manage.py lms create_oauth2_client \
"http://androidapp.com" "http://androidapp.com/redirect" public \
--client_id android --client_secret MgEEmkW6IM2WWHiYvEPialnE \
--trusted
error: unable to upgrade connection: container not found ("lms")
Databases ready.This should be solved in v3.4.0.
Steps to reproduce
tutor k8s quickstartUnexpected behavior
Fails on creating databases
Additional info (IMPORTANT)
tutor, version 3.1.0I know fix is on the commandline prompt however I added here as if it fails, it should be a two step process instead of trying to run migrations after service creations.