regisb
commented
6 years ago Yes, the database associated with the xqueue service needs to be created. Please run make provision, make migrate and try again.
Closed natea closed 6 years ago
regisb
commented
6 years ago Yes, the database associated with the xqueue service needs to be created. Please run make provision, make migrate and try again.
regisb
commented
6 years ago Concerning the 400 error, what is the exact domain name you used to configure the LMS?
natea
commented
6 years ago i ran the two commands you suggested, but I'm still getting the same error.
i just realized that when i re-ran the ./configure command i just accepted the defaults which were localhost:8000 instead of www.myopenedx.com. and localhost:8001 for Studio instead of studio.myopenedx.com.
But when I go to localhost:8000 and localhost:8001, it doesn't return the site.
regisb
commented
6 years ago Did you stop all services prior to running both commands? (sorry, I should have mentioned that earlier)
To test the images locally, specify "localhost" and "studio.localhost" during configuration. I realised this is a common mistake, so I modified the message displayed on configuration.
natea
commented
6 years ago No, I don't believe I had stopped all the services prior to running those commands. Will do so now.
If the default for testing locally is localhost and studio.localhost wouldn't you want to make those the defaults, rather than localhost:8000 and localhost:8001, or are the defaults for running it in production?
natea
commented
6 years ago The docs are still referring to localhost for local development: https://github.com/regisb/openedx-docker#for-developers
regisb
commented
6 years ago During configuration, the defaults are "www.myopenedx.com" and "studio.myopenedx.com". I wanted to set defaults for a production environment, which explains why I didn't set "localhost" or "localhost:8000" as defaults.
Let's see what's going on here (and then I'll add more information to the README because you're not the first one with this problem): when the platform is running and we try to access it on port 80, we reach the Nginx server, which then forwards the request to the gunicorn process. The production settings of the gunicorn process are defined to allow access only to the configured domain name, through the ALLOWED_HOSTS django setting: https://github.com/regisb/openedx-docker/blob/master/config/openedx/universal/lms/production.py#L21
This is a security setting. But it is not the case in development: development settings allow access from any domain name (ALLOWED_HOSTS = ['*'] is set in aws.py): https://github.com/regisb/openedx-docker/blob/master/config/openedx/universal/lms/development.py
This is an issue that confuses many people, and I'm not 100% sure how we can improve the situation. A good start would be to allow access to "localhost:8000" and "localhost:8001", even in production. It's not a big deal, in terms of security, because ports 8000 and 8001 are not open from the docker-compose configuration. But I wonder if this is going to generate other issues, for instance with static assets.
natea
commented
6 years ago I just re-ran the commands setting these environment variables. Before I was using the default settings file, which is for production, not for development.
export EDX_PLATFORM_PATH=/Users/nateaune/Dropbox/code/natea-edx-platform-ginkgo
export EDX_PLATFORM_SETTINGS=developmentNow the LMS is responding at localhost and Studio is responding at studio.localhost but I still see the django.db.utils.OperationalError: (1045, "Access denied for user 'xqueue'@'172.18.0.13' (using password: YES)") error message in the logs.
regisb
commented
6 years ago Nate, I just made changes to the production settings (in nginx and django) which should allow anyone to access "localhost" and "studio.localhost" both in development and producton mode. I also fixed the documentation.
The xqueue error message that you are seeing means that the xqueue MySQL database was not created, or not with the right credentials. I suspect that multiple mysql containers have been run concurrently. And here we reach the bottom of my ignorance about docker :sweat_smile: For some reason, when running make stop, most containers (including mysql) remain up. Can you please try to docker stop ... and docker rm ... each one of them manually? No container should be left running. Then run make provision again, and try to run the platform.
regisb
commented
6 years ago (I just updated the make stop command to make sure all containers are really stopped and removed)
natea
commented
6 years ago Yeah, the xqueue docker container was not stopped for some reason, which is probably why i was getting that error before:
Stopping openedx-docker_lms_worker_1 ... done
Stopping openedx-docker_cms_worker_1 ... done
Stopping openedx-docker_lms_1 ... done
Stopping openedx-docker_cms_1 ... done
Stopping openedx-docker_xqueue_1 ... done
Stopping openedx-docker_xqueue_consumer_1 ... done
Stopping openedx-docker_mysql_1 ...
Stopping openedx-docker_mysql_1 ... done
Stopping openedx-docker_forum_1 ... done
Stopping openedx-docker_elasticsearch_1 ... done
Stopping openedx-docker_smtp_1 ... done
Stopping openedx-docker_rabbitmq_1 ... done
Stopping openedx-docker_mongodb_1 ... done
Stopping openedx-docker_memcached_1 ...
Stopping openedx-docker_memcached_1 ... done
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ git pull
remote: Counting objects: 55, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 55 (delta 30), reused 55 (delta 30), pack-reused 0
Unpacking objects: 100% (55/55), done.
From https://github.com/regisb/openedx-docker
8ba0887724..9d4743f00e master -> origin/master
Updating 8ba0887724..9d4743f00e
Fast-forward
.gitignore | 4 ++++
Makefile | 8 ++++----
README.md | 13 ++-----------
config/mysql/templates/auth.env.templ | 2 +-
config/nginx/templates/cms.conf.templ | 2 +-
config/nginx/templates/lms.conf.templ | 2 +-
config/openedx/universal/cms/production.py | 3 +++
config/openedx/universal/lms/production.py | 2 ++
config/xqueue/templates/universal.py.templ | 1 +
configure | 16 +++++++++++-----
openedx/Dockerfile | 7 ++++++-
openedx/bin/wait-for-greenlight.sh | 10 ----------
12 files changed, 36 insertions(+), 34 deletions(-)
delete mode 100755 openedx/bin/wait-for-greenlight.sh
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4778136496cc regis/openedx-xqueue:ginkgo "./manage.py run_con…" 7 hours ago Restarting (0) About a minute ago openedx-docker_xqueue_consumer_1
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ make stop
docker-compose rm --stop --force
Stopping openedx-docker_xqueue_consumer_1 ... done
Going to remove openedx-docker_lms_worker_1, openedx-docker_cms_worker_1, openedx-docker_lms_1, openedx-docker_cms_1, openedx-docker_xqueue_1, openedx-docker_xqueue_consumer_1, openedx-docker_mysql_1, openedx-docker_forum_1, openedx-docker_elasticsearch_1, openedx-docker_smtp_1, openedx-docker_rabbitmq_1, openedx-docker_mongodb_1, openedx-docker_nginx_1, openedx-docker_memcached_1
Removing openedx-docker_lms_worker_1 ... done
Removing openedx-docker_cms_worker_1 ... done
Removing openedx-docker_lms_1 ... done
Removing openedx-docker_cms_1 ... done
Removing openedx-docker_xqueue_1 ... done
Removing openedx-docker_xqueue_consumer_1 ... done
Removing openedx-docker_mysql_1 ... done
Removing openedx-docker_forum_1 ... done
Removing openedx-docker_elasticsearch_1 ... done
Removing openedx-docker_smtp_1 ... done
Removing openedx-docker_rabbitmq_1 ... done
Removing openedx-docker_mongodb_1 ... done
Removing openedx-docker_nginx_1 ... done
Removing openedx-docker_memcached_1 ... done
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)Should I run the make update command or use make build as before? Is there any advantage to building the images myself, besides waiting for the slow images download on my home internet connection. ;)
natea
commented
6 years ago I pulled down the images from your Docker Hub, and re-ran all the commands, and now getting this error:
xqueue_consumer_1 | Traceback (most recent call last):
xqueue_consumer_1 | File "./manage.py", line 10, in <module>
xqueue_consumer_1 | execute_from_command_line(sys.argv)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
xqueue_consumer_1 | utility.execute()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute
xqueue_consumer_1 | self.fetch_command(subcommand).run_from_argv(self.argv)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
xqueue_consumer_1 | self.execute(*args, **cmd_options)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
xqueue_consumer_1 | self.check()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 482, in check
xqueue_consumer_1 | include_deployment_checks=include_deployment_checks,
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 72, in run_checks
xqueue_consumer_1 | new_errors = check(app_configs=app_configs)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/checks/model_checks.py", line 28, in check_all_models
xqueue_consumer_1 | errors.extend(model.check(**kwargs))
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1205, in check
xqueue_consumer_1 | errors.extend(cls._check_fields(**kwargs))
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1282, in _check_fields
xqueue_consumer_1 | errors.extend(field.check(**kwargs))
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 934, in check
xqueue_consumer_1 | errors = super(AutoField, self).check(**kwargs)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 207, in check
xqueue_consumer_1 | errors.extend(self._check_backend_specific_checks(**kwargs))
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 306, in _check_backend_specific_checks
xqueue_consumer_1 | return connection.validation.check_field(self, **kwargs)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/validation.py", line 18, in check_field
xqueue_consumer_1 | field_type = field.db_type(connection)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 614, in db_type
xqueue_consumer_1 | return connection.data_types[self.get_internal_type()] % data
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 36, in __getattr__
xqueue_consumer_1 | return getattr(connections[DEFAULT_DB_ALIAS], item)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1 | res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 196, in data_types
xqueue_consumer_1 | if self.features.supports_microsecond_precision:
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1 | res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/features.py", line 52, in supports_microsecond_precision
xqueue_consumer_1 | return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1 | res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 371, in mysql_version
xqueue_consumer_1 | with self.temporary_connection():
xqueue_consumer_1 | File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
xqueue_consumer_1 | return self.gen.next()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 462, in temporary_connection
xqueue_consumer_1 | cursor = self.cursor()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
xqueue_consumer_1 | cursor = self.make_cursor(self._cursor())
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
xqueue_consumer_1 | self.ensure_connection()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1 | self.connect()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
xqueue_consumer_1 | six.reraise(dj_exc_type, dj_exc_value, traceback)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1 | self.connect()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
xqueue_consumer_1 | self.connection = self.get_new_connection(conn_params)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 276, in get_new_connection
xqueue_consumer_1 | conn = Database.connect(**conn_params)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
xqueue_consumer_1 | return Connection(*args, **kwargs)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 193, in __init__
xqueue_consumer_1 | super(Connection, self).__init__(*args, **kwargs2)
xqueue_consumer_1 | django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'mysql' (111)")What is the output of the make provision command?
(and building or pulling should just be a matter of personal preference: in theory, there shouldn't be any difference.)
natea
commented
6 years ago For some reason, I can't stop the xqueue_consumer container.
^CGracefully stopping... (press Ctrl+C again to force)
Stopping openedx-docker_lms_worker_1 ... done
Stopping openedx-docker_lms_1 ... done
Stopping openedx-docker_forum_1 ... done
Stopping openedx-docker_cms_worker_1 ... done
Stopping openedx-docker_xqueue_1 ... done
Stopping openedx-docker_cms_1 ... done
Stopping openedx-docker_xqueue_consumer_1 ... done
Stopping openedx-docker_mysql_1 ... done
Stopping openedx-docker_nginx_1 ... done
Stopping openedx-docker_smtp_1 ... done
Stopping openedx-docker_memcached_1 ... done
Stopping openedx-docker_elasticsearch_1 ... done
Stopping openedx-docker_rabbitmq_1 ... done
Stopping openedx-docker_mongodb_1 ... done
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b00de9457a66 regis/openedx-xqueue:ginkgo "./manage.py run_con…" 14 hours ago Restarting (0) 45 minutes ago openedx-docker_xqueue_consumer_1
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker stop b00de9457a66
b00de9457a66
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b00de9457a66 regis/openedx-xqueue:ginkgo "./manage.py run_con…" 14 hours ago Restarting (0) 45 minutes ago openedx-docker_xqueue_consumer_1I just removed it, so now it's not running anymore:
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker rm b00de9457a66
b00de9457a66
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)Here's the output of the make provision command:
$ make provision
docker-compose run --rm lms bash -c "dockerize -wait tcp://mysql:3306 -timeout 20s && bash /openedx/config/provision.sh"
Starting openedx-docker_mysql_1 ... done
Starting openedx-docker_rabbitmq_1 ... done
Starting openedx-docker_elasticsearch_1 ... done
Starting openedx-docker_mongodb_1 ... done
Starting openedx-docker_smtp_1 ... done
Starting openedx-docker_memcached_1 ... done
Starting openedx-docker_forum_1 ... done
2018/06/09 16:08:17 Waiting for: tcp://mysql:3306
2018/06/09 16:08:17 Problem with dial: dial tcp 172.18.0.4:3306: connect: connection refused. Sleeping 1s
2018/06/09 16:08:18 Problem with dial: dial tcp 172.18.0.4:3306: connect: connection refused. Sleeping 1s
2018/06/09 16:08:19 Connected to tcp://mysql:3306
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.Hmmm and you're still getting that xqueue error? I need to think about this...
natea
commented
6 years ago Any more thoughts? Should I just delete all the containers and start over with rebuilding them from scratch?
On Sat, Jun 9, 2018 at 10:03 AM, Régis B. notifications@github.com wrote:
Hmmm and you're still getting that xqueue error? I need to think about this...
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/regisb/openedx-docker/issues/34#issuecomment-395984242, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHQyH5BL9aQCYc60GENHxwS9s4sytQjks5t6__pgaJpZM4UgqW7 .
--
Website: http://www.appsembler.com Blog: http://www.appsembler.com/blog Twitter: http://twitter.com/natea LinkedIn: http://linkedin.com/in/natea Phone: +1 (617) 702-4331
regisb
commented
6 years ago Let's try to diagnose this better. The error you are getting is still the following, right?
xqueue_consumer_1 | django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'mysql' (111)")Let's try the following steps:
Log in to the xqueue_consumer container:
docker-compose run --rm xqueue_consumer bash
Try to ping the mysql container:
apt install iputils-ping ping mysql
Try to connect to mysql instance: (replace "mypassword" by the XQUEUE_MYSQL_PASSWORD value you will find inside config/config.json)
apt install mysql-client mysql --user=xqueue --password=mypassword --host=mysql xqueue
Which steps failed?
natea
commented
6 years ago @regisb i'm wondering if this had something to do with it... Notice the two xqueue containers, one which is restarting every few minutes?
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c592fc649d21 regis/openedx-xqueue:ginkgo "./manage.py run_con…" 3 days ago Restarting (0) 17 seconds ago openedx-docker_xqueue_consumer_1
4e92a05909c2 regis/openedx:ginkgo "docker-entrypoint.s…" 4 days ago Up 31 hours 8000/tcp openedx-docker_lms_worker_1
ca1d03eeb986 regis/openedx:ginkgo "docker-entrypoint.s…" 4 days ago Up 31 hours 8000/tcp openedx-docker_lms_1
a713493aab1f regis/openedx-forum:ginkgo "/bin/sh -c './bin/u…" 4 days ago Up 31 hours 4567/tcp openedx-docker_forum_1
7f99c44df023 regis/openedx:ginkgo "docker-entrypoint.s…" 4 days ago Up 31 hours 8000/tcp openedx-docker_cms_worker_1
b6c48330d883 regis/openedx-xqueue:ginkgo "/bin/sh -c 'gunicor…" 4 days ago Up 31 hours 8040/tcp openedx-docker_xqueue_1
70ac45dc4093 regis/openedx:ginkgo "docker-entrypoint.s…" 4 days ago Up 31 hours 8000/tcp openedx-docker_cms_1
b2fd3947c711 mysql:5.6.36 "docker-entrypoint.s…" 4 days ago Up 31 hours 3306/tcp openedx-docker_mysql_1
20e910dbfa1a nginx:1.13 "nginx -g 'daemon of…" 4 days ago Up 31 hours 0.0.0.0:80->80/tcp openedx-docker_nginx_1
94a075800bf2 namshi/smtp "/bin/entrypoint.sh …" 4 days ago Up 31 hours 25/tcp openedx-docker_smtp_1
261a354f8f3e memcached:1.4.38 "docker-entrypoint.s…" 4 days ago Up 31 hours 11211/tcp openedx-docker_memcached_1
938abd73c96f elasticsearch:1.5.2 "/docker-entrypoint.…" 4 days ago Up 31 hours 9200/tcp, 9300/tcp openedx-docker_elasticsearch_1
4cc6b6bf6d24 mongo:3.0.14 "docker-entrypoint.s…" 4 days ago Up 31 hours 27017/tcp openedx-docker_mongodb_1so I stopped and removed that container:
$ docker stop c592fc649d21
c592fc649d21
nateaune at natemacbook in ~/Dropbox/code/docker-vscode (master●)
$ docker rm c592fc649d21
c592fc649d21
nateaune at natemacbook in ~/Dropbox/code/docker-vscode (master●)
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4e92a05909c2 regis/openedx:ginkgo "docker-entrypoint.s…" 4 days ago Up 31 hours 8000/tcp openedx-docker_lms_worker_1
ca1d03eeb986 regis/openedx:ginkgo "docker-entrypoint.s…" 4 days ago Up 31 hours 8000/tcp openedx-docker_lms_1
a713493aab1f regis/openedx-forum:ginkgo "/bin/sh -c './bin/u…" 4 days ago Up 31 hours 4567/tcp openedx-docker_forum_1
7f99c44df023 regis/openedx:ginkgo "docker-entrypoint.s…" 4 days ago Up 31 hours 8000/tcp openedx-docker_cms_worker_1
b6c48330d883 regis/openedx-xqueue:ginkgo "/bin/sh -c 'gunicor…" 4 days ago Up 31 hours 8040/tcp openedx-docker_xqueue_1
70ac45dc4093 regis/openedx:ginkgo "docker-entrypoint.s…" 4 days ago Up 31 hours 8000/tcp openedx-docker_cms_1
b2fd3947c711 mysql:5.6.36 "docker-entrypoint.s…" 4 days ago Up 31 hours 3306/tcp openedx-docker_mysql_1
20e910dbfa1a nginx:1.13 "nginx -g 'daemon of…" 4 days ago Up 31 hours 0.0.0.0:80->80/tcp openedx-docker_nginx_1
94a075800bf2 namshi/smtp "/bin/entrypoint.sh …" 4 days ago Up 31 hours 25/tcp openedx-docker_smtp_1
261a354f8f3e memcached:1.4.38 "docker-entrypoint.s…" 4 days ago Up 31 hours 11211/tcp openedx-docker_memcached_1
938abd73c96f elasticsearch:1.5.2 "/docker-entrypoint.…" 4 days ago Up 31 hours 9200/tcp, 9300/tcp openedx-docker_elasticsearch_1
4cc6b6bf6d24 mongo:3.0.14 "docker-entrypoint.s…" 4 days ago Up 31 hours 27017/tcp openedx-docker_mongodb_1I switched back to universal.production settings rather than my development settings to rule out that as the culprit. And sure enough once I did that, then the LMS and CMS came up fine without any problems. So I need to go debug my local environment to find out why it's not working.
I did run the commands you suggested, and everything worked as expected:
$ docker-compose run --rm xqueue_consumer bash
Starting openedx-docker_mysql_1 ... done
root@d5765b4b9e86:/openedx/xqueue# apt install iputils-ping
Reading package lists... Done
Building dependency tree
Processing triggers for libc-bin (2.23-0ubuntu10) ...
root@d5765b4b9e86:/openedx/xqueue# ping mysql
PING mysql (172.18.0.13) 56(84) bytes of data.
64 bytes from openedx-docker_mysql_1.openedx-docker_default (172.18.0.13): icmp_seq=1 ttl=64 time=0.229 ms
64 bytes from openedx-docker_mysql_1.openedx-docker_default (172.18.0.13): icmp_seq=2 ttl=64 time=0.102 ms
64 bytes from openedx-docker_mysql_1.openedx-docker_default (172.18.0.13): icmp_seq=3 ttl=64 time=0.109 ms
^C
--- mysql ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2027ms
rtt min/avg/max/mdev = 0.102/0.146/0.229/0.059 ms
root@d5765b4b9e86:/openedx/xqueue# apt install mysql-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libaio1 libnuma1 mysql-client-5.7 mysql-client-core-5.7
The following NEW packages will be installed:
libaio1 libnuma1 mysql-client mysql-client-5.7 mysql-client-core-5.7
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 8136 kB of archives.
After this operation, 65.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnuma1 amd64 2.0.11-1ubuntu1.1 [21.0 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 libaio1 amd64 0.3.110-2 [6356 B]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 mysql-client-core-5.7 amd64 5.7.22-0ubuntu0.16.04.1 [6425 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 mysql-client-5.7 amd64 5.7.22-0ubuntu0.16.04.1 [1673 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 mysql-client all 5.7.22-0ubuntu0.16.04.1 [10.0 kB]
Fetched 8136 kB in 11s (691 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libnuma1:amd64.
(Reading database ... 16714 files and directories currently installed.)
Preparing to unpack .../libnuma1_2.0.11-1ubuntu1.1_amd64.deb ...
Unpacking libnuma1:amd64 (2.0.11-1ubuntu1.1) ...
Selecting previously unselected package libaio1:amd64.
Preparing to unpack .../libaio1_0.3.110-2_amd64.deb ...
Unpacking libaio1:amd64 (0.3.110-2) ...
Selecting previously unselected package mysql-client-core-5.7.
Preparing to unpack .../mysql-client-core-5.7_5.7.22-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-client-core-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package mysql-client-5.7.
Preparing to unpack .../mysql-client-5.7_5.7.22-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-client-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package mysql-client.
Preparing to unpack .../mysql-client_5.7.22-0ubuntu0.16.04.1_all.deb ...
Unpacking mysql-client (5.7.22-0ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up libnuma1:amd64 (2.0.11-1ubuntu1.1) ...
Setting up libaio1:amd64 (0.3.110-2) ...
Setting up mysql-client-core-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Setting up mysql-client-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Setting up mysql-client (5.7.22-0ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
root@d5765b4b9e86:/openedx/xqueue# mysql --user=xqueue --password=HNpmPB1j --host=mysql xqueue
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 611
Server version: 5.6.36 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>The xqueue_consumer keeps restasrting because it is configured to restart on failure: see restart: unless-stopped in docker-compose.yml.
Can you post your development settings?
natea
commented
6 years ago This one seems to have reared its ugly head again.
xqueue_consumer_1 | Traceback (most recent call last):
xqueue_consumer_1 | File "./manage.py", line 10, in <module>
xqueue_consumer_1 | execute_from_command_line(sys.argv)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
xqueue_consumer_1 | utility.execute()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute
xqueue_consumer_1 | self.fetch_command(subcommand).run_from_argv(self.argv)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
xqueue_consumer_1 | self.execute(*args, **cmd_options)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
xqueue_consumer_1 | self.check()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 482, in check
xqueue_consumer_1 | include_deployment_checks=include_deployment_checks,
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 72, in run_checks
xqueue_consumer_1 | new_errors = check(app_configs=app_configs)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/checks/model_checks.py", line 28, in check_all_models
xqueue_consumer_1 | errors.extend(model.check(**kwargs))
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1205, in check
xqueue_consumer_1 | errors.extend(cls._check_fields(**kwargs))
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1282, in _check_fields
xqueue_consumer_1 | errors.extend(field.check(**kwargs))
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 934, in check
xqueue_consumer_1 | errors = super(AutoField, self).check(**kwargs)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 207, in check
xqueue_consumer_1 | errors.extend(self._check_backend_specific_checks(**kwargs))
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 306, in _check_backend_specific_checks
xqueue_consumer_1 | return connection.validation.check_field(self, **kwargs)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/validation.py", line 18, in check_field
xqueue_consumer_1 | field_type = field.db_type(connection)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 614, in db_type
xqueue_consumer_1 | return connection.data_types[self.get_internal_type()] % data
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 36, in __getattr__
xqueue_consumer_1 | return getattr(connections[DEFAULT_DB_ALIAS], item)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1 | res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 196, in data_types
xqueue_consumer_1 | if self.features.supports_microsecond_precision:
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1 | res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/features.py", line 52, in supports_microsecond_precision
xqueue_consumer_1 | return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1 | res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 371, in mysql_version
xqueue_consumer_1 | with self.temporary_connection():
xqueue_consumer_1 | File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
xqueue_consumer_1 | return self.gen.next()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 462, in temporary_connection
xqueue_consumer_1 | cursor = self.cursor()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
xqueue_consumer_1 | cursor = self.make_cursor(self._cursor())
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
xqueue_consumer_1 | self.ensure_connection()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1 | self.connect()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
xqueue_consumer_1 | six.reraise(dj_exc_type, dj_exc_value, traceback)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1 | self.connect()
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
xqueue_consumer_1 | self.connection = self.get_new_connection(conn_params)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 276, in get_new_connection
xqueue_consumer_1 | conn = Database.connect(**conn_params)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
xqueue_consumer_1 | return Connection(*args, **kwargs)
xqueue_consumer_1 | File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 193, in __init__
xqueue_consumer_1 | super(Connection, self).__init__(*args, **kwargs2)
xqueue_consumer_1 | django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'mysql' (111)")I got a mysql> prompt by following the steps you outlined above, so the xqueue consumer appears to be able to ping and connect to the MySQL server using the password as defined in config.json.
Why then the django.db.utils.OperationalError?
Does anything seem strange about the containers that are running:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
73ab6d877376 regis/openedx-xqueue:ginkgo "./manage.py run_con…" 5 days ago Restarting (0) 13 seconds ago openedx-docker_xqueue_consumer_1
12ca516d4d9d elasticsearch:1.5.2 "/docker-entrypoint.…" 5 days ago Up 18 minutes 9200/tcp, 9300/tcp openedx-docker_elasticsearch_1
9a439a8f3a8a regis/openedx:ginkgo "docker-entrypoint.s…" 5 days ago Up 18 minutes 8000/tcp openedx-docker_lms_worker_1
cfbb6da369e3 regis/openedx:ginkgo "docker-entrypoint.s…" 5 days ago Up 18 minutes 8000/tcp openedx-docker_cms_worker_1
bd2d928b0880 regis/openedx:ginkgo "docker-entrypoint.s…" 5 days ago Up 18 minutes 8000/tcp openedx-docker_lms_1
b602f3a0bd29 regis/openedx:ginkgo "docker-entrypoint.s…" 5 days ago Up 18 minutes 8000/tcp openedx-docker_cms_1
83518f0378a2 regis/openedx-xqueue:ginkgo "/bin/sh -c 'gunicor…" 5 days ago Up 18 minutes 8040/tcp openedx-docker_xqueue_1
1f65380c911b regis/openedx-forum:ginkgo "/bin/sh -c './bin/u…" 5 days ago Up 17 minutes 4567/tcp openedx-docker_forum_1
20a1f4529090 memcached:1.4.38 "docker-entrypoint.s…" 5 days ago Up 18 minutes 11211/tcp openedx-docker_memcached_1
661853b842bc namshi/smtp "/bin/entrypoint.sh …" 5 days ago Up 18 minutes 25/tcp openedx-docker_smtp_1
453065872e92 nginx:1.13 "nginx -g 'daemon of…" 5 days ago Up 18 minutes 0.0.0.0:80->80/tcp openedx-docker_nginx_1
616ca8b2e307 mysql:5.6.36 "docker-entrypoint.s…" 5 days ago Up 18 minutes 3306/tcp openedx-docker_mysql_1
a1d400358a44 rabbitmq:3.6.10 "docker-entrypoint.s…" 5 days ago Up 18 minutes 4369/tcp, 5671-5672/tcp, 25672/tcp openedx-docker_rabbitmq_1
d9c3110a26a2 mongo:3.0.14 "docker-entrypoint.s…" 5 days ago Up 18 minutes 27017/tcp openedx-docker_mongodb_1You have two different containers that run xqueue: 83518f0378a2 and 73ab6d877376. The problem is that you should have only one. From my understanding, the container with the mysql error is not inside the open edx "cluster" created from all the containers (lms, cms, mysql, etc.), so it does not have access to mysql. Try to kill the xqueue containers and restart it.
By the way, do you need the xqueue container? I'm thinking of removing it from the project.
Hi Regis,
I did a
git pull,configureandmake buildand now when I runmake upi'm seeing this permissions error with the xqueue service.And when I go to www.myopenedx.com and studio.myopenedx.com, I'm seeing "400 Bad Request".