Open sivashankerN opened 8 years ago
As per requirement 2, here, I am dividing the tasks and also adding estimated time for each task.
Adding the requirement : Requirement 2: Basic Scalable OpenEdX Deployment.
Current Status: LMS, CMS and DB are installed in one instance. This will cause severe challenges in scaling, performance. And it is getting manifested in rigidity of the system.
User <-> {LMS, CMS, DB ++ All in one box}
What Next?
A basic distributed structure will look like as below:
User <-> Load Balancer <-> LMS #1 --
LMS #2 -- DB Server Farm
:
LMS #N --
:
CMS #1 --
We will have 2 C3/C4 machines for LMS
1 C3/C4 machine for CMS
1 M3/M4 machine for DB Server
1 Elastic Load Balancer or HA Proxy (as needed)
Note: There is no clustering. If it works as expected, we can extend it.
LMS serves OpenEdX pages to Students (Learners)
CMS serves managing the courses.
CMS need not be a cluster / farm. A single machine suffice.
LMS need to be under Load Balancing.
Time frame: 1 Week of Experimentation needed to stabilise platform at least.
Objective: Getting control of OpenEdX deployment and to a reasonable level of scale. Say at least 500 concurrent students working.
Testing: It is really a tough task. We need to take some external help to develop Selenium, PhantomJS test suits. Maybe for some critical paths.
How it helps in Lab Porting? The proposed set up is Platform Architecture. It should not affect current activities the porting of LABs.
Note : Core Machine, is a machine with complete open edx platform setup. This machine will have all the services of open edx platoform.
Updated the entries in lms.env.json and lms.auth.json files for Memcached service. So that LMS machines will use Memcached server (May be Core Machine). There is a CACHES section in =lms.env.json= and replaced localhost with Memcached server IP(Core Machine.
And also replaced localhost with Core Machine ip in some places where backends are mysql and mongodb.
RabbitMQ parameter value need to update to get connected with RabbitMQ service which is running on RabbitMQ server( could be in Core Machine). Configuration file location /edx/app/xqueue/xqueue.*json
RABBITMQ_HOST=<RabbitMQ-server-IP>
Discussion page not loading in any of the courses.
sudo tail -f /edx/var/log/lms/edx.log
Aug 17 09:17:48 ip-172-31-17-157 [service_variant=lms][django.request][env:sandbox] ERROR [ip-172-31-17-157 2410] [base.py:256] - Internal Server Error: /courses/course-v1:IIIT-Hyderabad+CSE01+Anytime/discussion/forum Traceback (most recent call last): File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner return func(*args, **kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view return view_func(request, *args, **kwargs) File "/edx/app/edxapp/edx-platform/lms/djangoapps/django_comment_client/forum/views.py", line 183, in wrapped_view return view_func(request, course_key, *args, **kwargs) File "/edx/app/edxapp/edx-platform/lms/djangoapps/django_comment_client/forum/views.py", line 234, in forum_form_discussion user_info = user.to_dict() File "/edx/app/edxapp/edx-platform/lms/lib/comment_client/models.py", line 60, in to_dict self.retrieve() File "/edx/app/edxapp/edx-platform/lms/lib/comment_client/models.py", line 65, in retrieve self._retrieve(*args, **kwargs) File "/edx/app/edxapp/edx-platform/lms/lib/comment_client/user.py", line 151, in _retrieve metric_tags=self._metric_tags, File "/edx/app/edxapp/edx-platform/lms/lib/comment_client/utils.py", line 117, in perform_request content=response.text[:100] CommentClientError: u'Comments service returned invalid JSON for request d7871e5d-9bba-4f9c-a872-28f7846d500c; first 100 characters: \'<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor="white">\r\n<center><h1>502 Bad Gat\''
sudo tail -f /edx/var/log/nginx/error.log
2016/08/17 09:17:48 [crit] 1170#1170: *282 connect() to unix:/edx/var/forum/forum.sock failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: forum.*, request: "GET /api/v1/users/5?complete=True&request_id=d7871e5d-9bba-4f9c-a872-28f7846d500c HTTP/1.1", upstream: "http://unix:/edx/var/forum/forum.sock:/api/v1/users/5?complete=True&request_id=d7871e5d-9bba-4f9c-a872-28f7846d500c", host: "localhost:18080"
I created forum.sock manually, given www-data ownership permissions and tried still the problem exist.
sudo tail -f /edx/var/log/supervisor/forum-stderr.log
/edx/app/forum/.gem/ruby/1.9.1/gems/elasticsearch-transport-0.4.11/lib/elasticsearch/transport/transport/base.rb:132:in `__raise_transport_error': [404] {"error":"IndexMissingException[[content] missing]","status":404} (Elasticsearch::Transport::Transport::Errors::NotFound)
For this , I edited this /etc/elasticsearch/elasticsearch.yaml file for updating following parameter/directive values
network.bind_host: 0.0.0.0 network.publish_host: 0.0.0.0 network.host: 0.0.0.0 transport.tcp.port: 9300 http.port: 9200
Restart elasticsearch
sudo /etc/init.d/elasticsearch restart
Then the above "Not Found" error disappeared but still forum page is not loading( still Problem exist)
I could not find out @ where the problem is, So i thought of creating new VM(0) with ubuntu-14.04 and install a fresh open edx platform.
I installed Open edX platform on it by commenting some roles in edx_sandbox.yml
nginx_sites: -certs
- role: ecommerce when: SANDBOX_ENABLE_ECOMMERCE
- role: ecomworker when: SANDBOX_ENABLE_ECOMMERCE
- analytics_api
- insights
- oauth_client_setup
- certs
and there discussion page is loading for demo course.
Uncommented above roles and ran edx_sanbox.yml ansible playform for full setup.
OPTIONAL: Here we may not be needed insights and analytics_api
as we are not using these for anaylytics. So optionally we can comment them.
After second installation , again tested Discussion forum page in demo curse . It is loading without any issue.
Take snasphot of working one. So that with this we can create volumes and can be attached to any new VM with installed open edx platform.
From now on, Configure one Server at a time check discussion page and its scaling configuration Servers on VM0 ( Core Machine or Open edX platform):
- MySql
- MongoD
- RabbitMQ
- Memcached
etc I will add more if i miss any server Configuration on VM1 - LMS machine:
- /edx/app/edxapp/lms.env.json and /edx/app/edxapp/lms.auth.json
- /edx/app/forum/ if needed
- /edx/app/rabbitmq if needed
- /edx/app/xqueue/xqueue.env.json and /edx/app/xqueue/xqueue.auth.json
Took snapshot, For snapshot test, I created volume out of this snapshot and attached to another VM. Discussion page is loading. Correct log lines from "/edx/var/log/lms/edx.log". (just for understanding i posting here)
Aug 18 11:06:41 ip-172-31-25-197 [service_variant=lms][requests.packages.urllib3.connectionpool][env:sandbox] INFO [ip-172-31-25-197 2819] [connectionpool.py:207] - Starting new HTTP connection (1): localhost Aug 18 11:06:41 ip-172-31-25-197 [service_variant=lms][lms.lib.comment_client.utils][env:sandbox] INFO [ip-172-31-25-197 2819] [utils.py:49] - comment_client_request_log: request_id=217ade28-5b73-4b04-a942-44edc87ac317, method=get, url=http://localhost:18080/api/v1/users/5, duration=0.00850105285645 Aug 18 11:06:41 ip-172-31-25-197 [service_variant=lms][requests.packages.urllib3.connectionpool][env:sandbox] INFO [ip-172-31-25-197 2819] [connectionpool.py:207] - Starting new HTTP connection (1): localhost Aug 18 11:06:41 ip-172-31-25-197 [service_variant=lms][lms.lib.comment_client.utils][env:sandbox] INFO [ip-172-31-25-197 2819] [utils.py:49] - comment_client_request_log: request_id=d1857155-c48d-4d79-8605-cfc0c9a677f5, method=get, url=http://localhost:18080/api/v1/users/5, duration=0.00607895851135 Aug 18 11:06:41 ip-172-31-25-197 [service_variant=lms][requests.packages.urllib3.connectionpool][env:sandbox] INFO [ip-172-31-25-197 2819] [connectionpool.py:207] - Starting new HTTP connection (1): localhost Aug 18 11:06:41 ip-172-31-25-197 [service_variant=lms][lms.lib.comment_client.utils][env:sandbox] INFO [ip-172-31-25-197 2819] [utils.py:49] - comment_client_request_log: request_id=fcb84227-7dd5-47d6-8fae-5cf2a99ede35, method=get, url=http://localhost:18080/api/v1/threads, duration=0.00647687911987
Also you will be seeing as follows in above logs
Aug 18 11:11:08 ip-172-31-25-197 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-172-31-25-197 1557] [consumer.py:364] - consumer: Cannot connect to amqp://celery:@127.0.0.1:5672//: [Errno 104] Connection reset by peer. Trying again in 32.00 seconds... Aug 18 11:11:08 ip-172-31-25-197 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-172-31-25-197 1558] [consumer.py:364] - consumer: Cannot connect to amqp://celery:@127.0.0.1:5672//: [Errno 104] Connection reset by peer. Trying again in 32.00 seconds..
No need to worry about these, we can solve this by modifying CELERY_BROKER_HOSTNAME parameter value in =/edx/app/edxapp/[lms|cms].[env.|auth].json files
Ported one more course into VM0 to check discussion forum page. It is loading. no issues for now. Now, will have to start configuring servers mentioned above one by one and test discussion forum page
Default values of mysql server and its databases after installation of platform on VM0 Mysql server configuration: -> Edited the /etc/mysql/my.cnf commented bind-address directive value
bind-addres = 127.0.0.1
sudo service mysql restart
Test mysql server configuration: Now we should be able to accept connection from remote machines
From VM1
telent -h 172.31.25.109 3306
Output
ubuntu@ip-172-31-24-144:~$ telnet 172.31.25.109 3306 Trying 172.31.25.109... Connected to 172.31.25.109. Escape character is '^]'. dHost 'ip-172-31-24-144.us-west-1.compute.internal' is not allowed to connect to this MySQL serverConnection closed by foreign host.
Mysql databases information.
sudo mysql mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | ecommerce | | edx_notes_api | | edxapp | | edxapp_csmh | | mysql | | performance_schema | | xqueue | +--------------------+ 8 rows in set (0.00 sec)
Mysql user privileges are
use mysql mysql> select user,host from user; +------------------+------------------+ | user | host | +------------------+------------------+ | root | 127.0.0.1 | | root | ::1 | | root | ip-172-31-25-109 | | admin | localhost | | debian-sys-maint | localhost | | ecomm001 | localhost | | edxapp001 | localhost | | migrate | localhost | | notes001 | localhost | | read_only | localhost | | root | localhost | | xqueue001 | localhost | +------------------+------------------+ 12 rows in set (0.00 sec)
root user can access the databases from localhost as well as from 172.31.25.109(which is actually IP based) Same as root user, we have to do for other users as well. So that , VM1 can access theses databases on particular user.
Providing privileges
mysql> GRANT ALL PRIVILEGES ON . TO 'edxapp001'@'172.31.17.157' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.01 sec)
mysql> GRANT ALL PRIVILEGES ON . TO 'admin'@'172.31.17.157' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.01 sec)
mysql> GRANT ALL PRIVILEGES ON . TO 'migrate'@'172.31.17.157' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON . TO 'notes001'@'172.31.17.157' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON . TO 'read_only'@'172.31.17.157' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON . TO 'reports001'@'172.31.17.157' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON . TO 'xqueue001'@'172.31.17.157' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON . TO 'ecomm001'@'172.31.17.157' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec)
Then updated user information on mysql database as follows
sudo myql use mysql mysql> select user,host from user; +------------------+------------------+ | user | host | +------------------+------------------+ | root | 127.0.0.1 | | admin | 172.31.17.157 | | ecomm001 | 172.31.17.157 | | edxapp001 | 172.31.17.157 | | migrate | 172.31.17.157 | | notes001 | 172.31.17.157 | | read_only | 172.31.17.157 | | xqueue001 | 172.31.17.157 | | root | ::1 | | root | ip-172-31-25-109 | | admin | localhost | | debian-sys-maint | localhost | | ecomm001 | localhost | | edxapp001 | localhost | | migrate | localhost | | notes001 | localhost | | read_only | localhost | | root | localhost | | xqueue001 | localhost | +------------------+------------------+ 19 rows in set (0.00 sec)
I checked discussion forum page, it is working without any issue
MongoDB server configuration on VM0 Nothing to do for MongoDB server . Mongodb allows remote machines by default.
/etc/mongod.conf #mongodb configuration file Run bellow command on VM1 to check mongodb connection mongo 172.31.25.109:27017/cs_comments_service -u cs_comments_service -p password
Will get the following output.
MongoDB shell version: 2.6.12 connecting to: 172.31.25.109:27017/cs_comments_service Welcome to the MongoDB shell. For interactive help, type "help". For more comprehensive documentation, see http://docs.mongodb.org/ Questions? Try the support group http://groups.google.com/group/mongodb-user ^C bye
That means we are able to connect MongoDB server
Memcached server on VM0
vim /etc/memcached.conf
and change the interface value and restart service
-l 127.0.0.1 to -l 0.0.0.0 sudo service mysql restart
Now Memcached server allows remote machines to connect. Check from VM1 by typing this command
telnet 172.31.25.109 11211
OUTPUT
Trying 172.31.25.109... Connected to 172.31.25.109. Escape character is '^]'.
RabbitMQ server configuration on VM0
netstat -tunlp | grep 5672 tcp 0 0 0.0.0.0:15672 0.0.0.0:* LISTEN 1406/beam.smp
tcp 0 0 0.0.0.0:55672 0.0.0.0:* LISTEN 1406/beam.smp
tcp 0 0 127.0.0.1:5672 0.0.0.0:* LISTEN 1406/beam.smp
Edit
vim /etc/rabbitmq/rabbitmq-env.conf
and update
RABBITMQ_NODE_IP_ADDRESS=127.0.0.1 to RABBITMQ_NODE_IP_ADDRESS=0.0.0.0 to accept the rabbitmq-client connections from remote machines. Then restart RabbitMQ sudo service rabbitmq-server restart
Check it from VM1:
telnet 172.31.25.109 5672
OUTPUT
Trying 172.31.25.109... Connected to 172.31.25.109. Escape character is '^]'.
For now all are working without any issues. even discussion forum page too.
Stopped mysql service in VM1
Now update .json files for fetching courses from VM0. Edit /edx/app/edxapp/lms.auth.json file , Look for Mysql Databases
"DATABASES": { "default": {
..... "HOST": "172.31.25.109", .... }, ... This time it was showing created courses on VM0. that means we are able to see the course on VM1. These course are coming from VM0.
But we click on any course it is showing page not found. at this time log files from
sudo tail -f /edx/var/log/lms/edx.log Aug 19 09:52:57 ip-172-31-17-157 [service_variant=lms][openedx.core.lib.edx_api_utils][env:sandbox] WARNING [ip-172-31-17-157 2224] [edx_api_utils.py:42] - programs configuration is disabled.
Ignore it for now since we are not updated all mysql database related parameters in lms.auth.json file
so, let us go ahead and update For Celery issue just update bellow parameter in lms.env.json file
"CELERY_BROKER_HOSTNAME": "172.31.25.109",
restart edx services
No change happened after updating the entries.
"read_replica": { "ENGINE": "django.db.backends.mysql", "HOST": "172.31.25.109", "NAME": "edxapp", "PASSWORD": "password", "PORT": "3306", "USER": "edxapp001" }, "student_module_history": { "ENGINE": "django.db.backends.mysql", "HOST": "172.31.25.109", "NAME": "edxapp_csmh", "PASSWORD": "password", "PORT": "3306", "USER": "edxapp001" } let us go ahead update mongodb related parameters
Stopped mongodb service on VM1 and updated "host" parameter
"CONTENTSTORE": { "ADDITIONAL_OPTIONS": {}, "DOC_STORE_CONFIG": { "collection": "modulestore", "connectTimeoutMS": 2000, "db": "edxapp", "host": [ "localhost" ], "password": "password", "port": 27017, "socketTimeoutMS": 3000, "ssl": false, "user": "edxapp" }, "ENGINE": "xmodule.contentstore.mongo.MongoContentStore", "OPTIONS": { "db": "edxapp", "host": [ "localhost" ], "password": "password", "port": 27017, "ssl": false, "user": "edxapp" } },
Updated caches also
"CACHES": { "celery": { "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", "KEY_FUNCTION": "util.memcache.safe_key", "KEY_PREFIX": "celery", "LOCATION": [ "172.31.25.109:11211" ], "TIMEOUT": "7200" }, "configuration": { "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", "KEY_FUNCTION": "util.memcache.safe_key", "KEY_PREFIX": "ip-172-31-25-109", "LOCATION": [ "172.31.25.109:11211" ] }, "course_structure_cache": { "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", "KEY_FUNCTION": "util.memcache.safe_key", "KEY_PREFIX": "course_structure", "LOCATION": [ "172.31.25.109:11211" ], "TIMEOUT": "7200" }, "default": { "BACKEND": "django.core.cache.backends.memcached.MemcachedCache", "KEY_FUNCTION": "util.memcache.safe_key", "KEY_PREFIX": "default", "LOCATION": [ "172.31.25.109:11211" ], "VERSION": "1" },
restart edx services
This time we are able to see all courses but except discussion page .
Found where the problem is. It is mongod databases related parameter values. I had a doubt, is really mongod causing the issue?. Yes, it is.
Checked lms/edx.log file this we got different logs
sudo tail -f /edx/var/log/lms/edx.log Aug 19 10:50:22 ip-172-31-17-157 [service_variant=lms][django.request][env:sandbox] ERROR [ip-172-31-17-157 2326] [base.py:256] - Internal Server Error: /courses/course-v1:IIIT-Hyderabad+CCNSB03+Anytime/discussion/forum Traceback (most recent call last): File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, _callback_args, _callback_kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner return func(_args, _kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view return view_func(request, _args, _kwargs) File "/edx/app/edxapp/edx-platform/lms/djangoapps/django_comment_client/forum/views.py", line 183, in wrapped_view return view_func(request, course_key, _args, _kwargs) File "/edx/app/edxapp/edx-platform/lms/djangoapps/django_comment_client/forum/views.py", line 234, in forum_form_discussion user_info = user.to_dict() File "/edx/app/edxapp/edx-platform/lms/lib/comment_client/models.py", line 60, in to_dict self.retrieve() File "/edx/app/edxapp/edx-platform/lms/lib/comment_client/models.py", line 65, in retrieve self._retrieve(_args, _kwargs) File "/edx/app/edxapp/edx-platform/lms/lib/comment_client/user.py", line 151, in _retrieve metric_tags=self._metric_tags, File "/edx/app/edxapp/edx-platform/lms/lib/comment_client/utils.py", line 89, in perform_request timeout=config.connection_timeout File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/requests/api.py", line 53, in request return session.request(method=method, url=url, _kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request resp = self.send(prep, *_send_kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send r = adapter.send(request, **kwargs) File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/requests/adapters.py", line 449, in send raise ReadTimeout(e, request=request) ReadTimeout: HTTPConnectionPool(host='localhost', port=18080): Read timed out. (read timeout=5.0)
So, i updated following parameter in lms.env.json
COMMENTS_SERVICE_URL": "http://localhost:18080", to COMMENTS_SERVICE_URL": "http://172.31.25.109:18080",
Restarted edx services Then Discussion forum started opening again. we can not conclude here that forum page is opening so we are done.
because still there are some more things to do
XQueue app configuration on VM1 Edit xqueue.env.json file of VM1
"RABBIT_HOST": "localhost", to "RABBIT_HOST": "172.31.25.109",
restarted edx services and checked for discussion page. it is loading (no issues till now).
Edit xqueue.auth.json file of VM1
"HOST" : "localhost" to "HOST": "172.31.25.109"
restarted edx services and checked for discussion page. it is loading (no issues till now).
Successfully configured xqueue app.
Forum app configuration on VM1 Update /edx/app/forum/forum_env
export MONGOHQ_URL="mongodb://cs_comments_service:password@localhost:27017/cs_comments_service"
to
export MONGOHQ_URL="mongodb://cs_comments_service:password@172.31.25.109:27017/cs_comments_service"
restarted edx services and checked for discussion page. it is loading without any issues.
stopped CMS service
/edx/bin/supervisorctl stop edxapp:cms
Checked discussion page LMS works.
Stopped following service too. because as we are using VM0's servers.
sudo service rabbitmq-server stop sudo service memcached stop
Checked for Discussion forum page. it is loading.
From all above comments, summary is: If we install complete Open edX platform in one machine (VM0- Core machine)( all edx services) and Only LMS in one machine (VM1) then update following files On VM0
/etc/mysql/my.cf ----- bind-address /etc/rabbitmq-env.conf -- RABBITMQ_NODE_IP_ADDRESS /etc/memcached.conf -- -l Grant privileges on databases to users from remote machine (VM1)
See the above comments for more understanding.
On VM1 In /edx/app/edxapp/lms.env.json update following parameters
In "CACHES" : section # for memcached service "host" : "CELERY_BROKER_HOSTNAME" : "VM0-IP or Domain name" "COMMENTS_SERVICE_URL" : "http://VM0-IP:18080 or http://Domain name:18080 "SITE_NAME" : "VM1-IP or Domain name "
- In /edx/app/edxapp/lms.auth.json, just replace all "localhost" with VM0-IP or VM0 domain name.
- /edx/app/forum/forum_env
- /edx/app/xqueue/xqueue.env.json and /edx/app/xqueue/xqueue.auth.json
- /edx/app/certs/auth.json and env.json #if needed , # i did not touch this in this setup.
- Then restart lms service Please read above comments for more understanding.
If all goes well, your are done with Basic Scalable open edX platform (with Two machine setup).
That means we should able to see the courses on VM1 without any issues, actually these courses are created/ported on/to VM0.
Feedback from Release 2 :
Setting this issue to milestone : Release III
Content of this issue is private and hence removed from here.