Endpoints degraded after migration from unmanaged to managed disk ..

[sanjay186]

hi @eltoncarr

We deployed openedx ficus version on azure, the site was working fine. And then we migrated the disks of all the VMs from unmanaged to managed in Azure. After changing this, all the VMs has got restarted again. Then we have built the assets of lms and cms and restarted all the services. After restarting all the VMs, the endpoints got degraded and the LMS is completely down now and CMS is showing server error after we login. It would be really good if you can help us out to resolve this issue, by giving your suggestion. As we are running this as production side, your valuable help would really mean to us.

Thanks in advance

[sanjay186]

i found this logs under /edx/var/log/cms/edx.log

Sep 20 12:14:52 TrainJPvmss000000 [service_variant=cms][django.request][env:sandbox] ERROR [TrainJPvmss000000 22225] [base.py:256] - Internal Server Error: /heartbeat Traceback (most recent call last): File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, callback_kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner return func(*args, *kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/dogapi/stats/dog_stats_api.py", line 184, in wrapped result = func(args, kwargs) File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/heartbeat/views.py", line 24, in heartbeat output = modulestore().heartbeat() File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/django.py", line 240, in modulestore contentstore(), File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/contentstore/django.py", line 28, in contentstore CONTENTSTORE[name] = class(options) File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/contentstore/mongo.py", line 42, in init port=port, tz_aware=tz_aware, user=user, password=password, proxy=proxy, kwargs File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/mongo_utils.py", line 42, in connect_to_mongodb **kwargs File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 425, in init raise ConnectionFailure(str(e)) ConnectionFailure: [Errno -2] Name or service not known, [Errno 111] Connection refused, [Errno 111] Connection refused

[sanjay186]

when i checked the status i found this

[sanjay186]

under /edx/var/log/supervisor/lms-stderr.log

[eltoncarr]

Hi Sanjay,

I will take a look into this issue later today and if possible provide a response. As a general rule of thumb, you should not perform any operation on your production environment that hasn't already been tested in a test environment.

[sanjay186]

HI elton

How to create a test environment ?? Should i deploy a new instance in azure and keep that as test environment?? or is there any way that i can create a test environment in the same instance that i have in azure??

Please Guide me

[eltoncarr]

@sanjay186, yes. You'd first setup a cluster similar to what you are deploying to production and run tests there. When done, you can always tear it down. Regarding the error, you may want to double check that your backend services are all operational. Connection error suggests that a service (MongoDB) isn't running.