search

microsoft / oxa-tools

Open edX on Azure Tools
MIT License
36 stars 103 forks source link

Mongdb services are down #394

Closed sanjay186 closed 5 years ago

sanjay186 commented 5 years ago

Hi @eltoncarr

Suddenly my cms was down, its showing "The Studio servers encountered an error". When checked the cms log under /edx/var/log/cms/edx.log

openedxjpvmss000000 [service_variant=cms][root][env:sandbox] ERROR [openedxjpvmss000000 31729] [exceptions.py:16] - Uncaught exception from <class 'dj$ Traceback (most recent call last): File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, callback_kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner return func(*args, *kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/dogapi/stats/dog_stats_api.py", line 184, in wrapped result = func(args, kwargs) File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/heartbeat/views.py", line 24, in heartbeat output = modulestore().heartbeat() File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/django.py", line 240, in modulestore contentstore(), File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/contentstore/django.py", line 28, in contentstore CONTENTSTORE[name] = class(options) File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/contentstore/mongo.py", line 42, in init port=port, tz_aware=tz_aware, user=user, password=password, proxy=proxy, kwargs File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/mongo_utils.py", line 42, in connect_to_mongodb **kwargs File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 425, in init raise ConnectionFailure(str(e)) ConnectionFailure: [Errno 111] Connection refused, [Errno -2] Name or service not known, [Errno 111] Connection refused

When i accessed to mongodb vm and checked the mongodb status its shows its not running.

Last login: Mon Oct 15 10:53:50 2018 from 10.0.0.4 openedxadmin@openedxjpmongo1:~$ sudo service mongod status ● mongod.service - High-performance, schema-free document-oriented database Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2018-10-15 10:56:28 UTC; 1h 51min ago Docs: https://docs.mongodb.org/manual Process: 7328 ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf (code=exited, status=1/FAILURE) Main PID: 7328 (code=exited, status=1/FAILURE)

Oct 15 10:56:28 openedxjpmongo1 systemd[1]: Started High-performance, schema-free document-oriented database. Oct 15 10:56:28 openedxjpmongo1 mongod[7328]: about to fork child process, waiting until server is ready for connections. Oct 15 10:56:28 openedxjpmongo1 mongod[7328]: forked process: 7346 Oct 15 10:56:28 openedxjpmongo1 systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE Oct 15 10:56:28 openedxjpmongo1 systemd[1]: mongod.service: Unit entered failed state. Oct 15 10:56:28 openedxjpmongo1 systemd[1]: mongod.service: Failed with result 'exit-code'.

please help us in solving this issue.

sanjay186 commented 5 years ago

@eltoncarr

my mongo db VM restarted due to some outage in central us region. will this be a reason for mongodb services to go down??

eltoncarr commented 5 years ago

I am not sure of the specifics but you may want to check the services and see what's happening. Check the error logs and mitigate as necessary. We also have services in south central and didn't see what you are indicating...