search

twitter-archive / mysos

Cotton (formerly known as Mysos)
https://incubator.apache.org/projects/cotton.html
588 stars 67 forks source link

is mysos runable? #55

Closed mingqi closed 9 years ago

mingqi commented 9 years ago

I run up the mesos scheduler and listener on 5500 port. The page of http://myhost:5500 always display "Mysos scheduler is still connecting...". Below is scheduler's logs. Is somethings wrong?

My command is:

python ./scheduler.py \
    --mesos_master=zk://172.31.15.246:2181/mesos \
    --port=5500 \
    --framework_user=mysos \
    --executor_uri=/home/vagrant/mysos/dist/mysos-0.1.0-dev0.zip \
    --executor_cmd=/home/vagrant/mysos/vagrant/bin/mysos_executor.sh \
    --zk_url=zk://172.31.15.246:2181/mysos \
    --admin_keypath=/home/ubuntu/mysos/vagrant/etc/admin_keyfile.yml \
    --framework_failover_timeout=1m \
    --framework_role=mysos \
    --scheduler_keypath=/home/ubuntu/mysos/vagrant/etc/scheduler_keyfile.txt \
    --executor_source_prefix='vagrant.devcluster' \
    --executor_environ='[{"name": "MYSOS_DEFAULTS_FILE", "value": "/etc/mysql/conf.d/my5.6.cnf"}]'

and the scheduler.py content is:

#!/usr/bin/python

from os.path import join, abspath
import sys

ROOT_DIR = abspath('./')
sys.path.insert(0, ROOT_DIR)

from mysos.scheduler.mysos_scheduler import proxy_main

if __name__ == '__main__':
    proxy_main()

output is:

root@mingqi-dev:~/mysos# ./scheduler.sh
I0603 15:34:57.013078 17733 mysos_scheduler.py:177] Options in use: {'framework_failover_timeout': '1m', 'twitter_common_log_simple'
: False, 'verbose': None, 'twitter_common_app_daemon_stdout': '/dev/null', 'twitter_common_log_scribe_category': 'python_default', '
api_port': 5500, 'twitter_common_log_log_dir': '/var/tmp', 'twitter_common_app_daemonize': False, 'twitter_common_app_ignore_rc_file
': False, 'twitter_common_app_profiling': False, 'work_dir': '/tmp/mysos', 'twitter_common_app_pidfile': None, 'twitter_common_log_s
cribe_buffer': False, 'executor_source_prefix': 'vagrant.devcluster', 'election_timeout': '60s', 'twitter_common_app_rc_filename': False, 'framework_role': 'mysos', 'executor_environ': '[{"name": "MYSOS_DEFAULTS_FILE", "value": "/etc/mysql/conf.d/my5.6.cnf"}]', 'twitter_common_log_scribe_log_level': 'NONE', 'executor_uri': '/home/vagrant/mysos/dist/mysos-0.1.0-dev0.zip', 'twitter_common_log_disk_log_level': 'NONE', 'twitter_common_log_stderr_log_level': 'ERROR', 'framework_authentication_file': None, 'state_storage': 'zk', 'executor_cmd': '/home/vagrant/mysos/vagrant/bin/mysos_executor.sh', 'twitter_common_app_profile_output': None, 'framework_user': 'mysos', 'zk_url': 'zk://172.31.15.246:2181/mysos', 'twitter_common_app_debug': False, 'twitter_common_log_scribe_port': 1463, 'twitter_common_log_scribe_host': 'localhost', 'scheduler_keypath': '/home/ubuntu/mysos/vagrant/etc/scheduler_keyfile.txt', 'installer_args': None, 'backup_store_args': None, 'mesos_master': 'zk://172.31.15.246:2181/mesos', 'admin_keypath': '/home/ubuntu/mysos/vagrant/etc/admin_keyfile.yml', 'twitter_common_app_daemon_stderr': '/dev/null'}
I0603 15:34:57.016001 17733 mysos_scheduler.py:219] Extracted web assets into /tmp/mysos
I0603 15:34:57.016411 17733 mysos_scheduler.py:244] Starting Mysos scheduler
I0603 15:34:57.027303 17733 connection.py:566] Connecting to 172.31.15.246:2181
I0603 15:34:57.028554 17733 connection.py:276] Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None)
I0603 15:34:57.031117 17733 client.py:378] Zookeeper connection established, state: CONNECTED
I0603 15:34:57.034564 17733 mysos_scheduler.py:250] Using ZooKeeper (path: /mysos) for state storage
I0603 15:34:57.034933 17733 connection.py:276] Sending request(xid=1): GetData(path='/mysos/state/scheduler', watcher=None)
I0603 15:34:57.036081 17733 connection.py:360] Received response(xid=1): ("ccopy_reg\n_reconstructor\np1\n(cmysos.scheduler.state\nScheduler\np2\nc__builtin__\nobject\np3\nNtRp4\n(dp5\nS'framework_info'\np6\ncmesos.interface.mesos_pb2\nFrameworkInfo\np7\n(tRp8\n(dp9\nS'serialized'\np10\nS'\\n\\x05mysos\\x12\\x05mysos!\\x00\\x00\\x00\\x00\\x00\\x00N@(\\x012\\x05mysosB\\x05mysos'\np11\nsbsS'clusters'\np12\ng1\n(ctwitter.common.collections.orderedset\nOrderedSet\np13\ng3\nNtRp14\n(dp15\nS'map'\np16\n(dp17\nsS'end'\np18\n(lp19\nNag19\nag19\nasbsb.", ZnodeStat(czxid=17, mzxid=17, ctime=1433316463490, mtime=1433316463490, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=422, numChildren=0, pzxid=17))
I0603 15:34:57.036864 17733 mysos_scheduler.py:262] Successfully restored scheduler state
I0603 15:34:57.039196 17733 sched.cpp:139] Version: 0.20.1
2015-06-03 15:34:57,041:17733(0x7f90d37fe700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-06-03 15:34:57,053:17733(0x7f90d37fe700):ZOO_INFO@log_env@716: Client environment:host.name=mingqi-dev
2015-06-03 15:34:57,053:17733(0x7f90d37fe700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-06-03 15:34:57,054:17733(0x7f90d37fe700):ZOO_INFO@log_env@724: Client environment:os.arch=3.13.0-44-generic
2015-06-03 15:34:57,054:17733(0x7f90d37fe700):ZOO_INFO@log_env@725: Client environment:os.version=#73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014
2015-06-03 15:34:57,054:17733(0x7f90d37fe700):ZOO_INFO@log_env@733: Client environment:user.name=ubuntu
2015-06-03 15:34:57,055:17733(0x7f90d37fe700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2015-06-03 15:34:57,055:17733(0x7f90d37fe700):ZOO_INFO@log_env@753: Client environment:user.dir=/home/ubuntu/mysos
2015-06-03 15:34:57,055:17733(0x7f90d37fe700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=172.31.15.246:2181 sessionTimeout=10000 watcher=0x7f90e525bcc0 sessionId=0 sessionPasswd=<null> context=0x7f90cc0013a0 flags=0
2015-06-03 15:34:57,057:17733(0x7f90d09c9700):ZOO_INFO@check_events@1703: initiated connection to server [172.31.15.246:2181]
2015-06-03 15:34:57,059:17733(0x7f90d09c9700):ZOO_INFO@check_events@1750: session establishment complete on server [172.31.15.246:2181], sessionId=0x14db7918a73000e, negotiated timeout=10000
I0603 15:34:57.060551 17745 group.cpp:313] Group process (group(1)@127.0.0.1:38704) connected to ZooKeeper
I0603 15:34:57.060619 17745 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0603 15:34:57.060670 17745 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
Bottle v0.11.6 server starting up (using CherryPyServer())...
Listening on http://0.0.0.0:5500/
I0603 15:34:57.076939 17745 detector.cpp:138] Detected a new leader: (id='1')
I0603 15:34:57.077203 17745 group.cpp:658] Trying to get '/mesos/info_0000000001' in ZooKeeper
Hit Ctrl-C to quit.

I0603 15:34:57.085907 17745 detector.cpp:426] A new leading master (UPID=master@172.31.15.246:5050) is detected
I0603 15:34:57.086035 17745 sched.cpp:235] New master detected at master@172.31.15.246:5050
I0603 15:34:57.086138 17745 sched.cpp:243] No credentials provided. Attempting to register without authentication
xujyan commented 9 years ago

Has the issue been resolved? Feel free to post issues here or questions to dev@mysos.incubator.apache.org

mingqi commented 9 years ago

OK, thanks. Yes, this issue was resolved. The problem is I use role "mysos" at first time but this role not in mesos cluster. I changed role to default "*" but not take effect. The cause is mysos restore state from Zookeeper.

pangbingqiang commented 9 years ago

hi,all,ihave same question. my command is: 1 #!/bin/sh 2 3 ZK_HOST=xx.xxx.xxx.xxx 4 API_PORT=55001 5 6 # NOTE: In --executor_environ we are pointing MYSOS_DEFAULTS_FILE to an empty MySQL defaults file. 7 # The file 'my5.6.cnf' is pre-installed by the 'mysql-server-5.6' package on the VM. 8 mysos_scheduler \ 9 --port=$API_PORT \ 10 --framework_user=vagrant \ 11 --mesos_master=zk://$ZK_HOST:2184/mesos \ 12 --executor_uri=/home/huajianfeng/.tox/distshare/mysos-0.1.0-dev0.zip \ 13 --executor_cmd=/home/huajianfeng/incubator-cotton-master/vagrant/bin/mysos_executor.sh \ 14 --zk_url=zk://$ZK_HOST:2184/mysos \ 15 --admin_keypath=/home/huajianfeng/incubator-cotton-master/vagrant/etc/admin_keyfile.yml \ 16 --framework_failover_timeout=1m \ 17 --framework_role=* \ 18 --scheduler_keypath=/home/huajianfeng/incubator-cotton-master/vagrant/etc/scheduler_keyfile.txt \
19 --executor_source_prefix='vagrant.devcluster' \ 20 --executor_environ='[{"name": "MYSOS_DEFAULTS_FILE", "value": "/etc/mysql/conf.d/my5.6.cnf"}]'

and output is: I1124 00:13:35.048789 178455 mysos_scheduler.py:219] Extracted web assets into /tmp/mysos I1124 00:13:35.048928 178455 mysos_scheduler.py:244] Starting Mysos scheduler I1124 00:13:35.050659 178455 connection.py:566] Connecting to 10.175.100.231:2184 I1124 00:13:35.051512 178455 connection.py:276] Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) I1124 00:13:35.055763 178455 client.py:378] Zookeeper connection established, state: CONNECTED I1124 00:13:35.058011 178455 mysos_scheduler.py:250] Using ZooKeeper (path: /mysos) for state storage I1124 00:13:35.058290 178455 connection.py:276] Sending request(xid=1): GetData(path='/mysos/state/scheduler', watcher=None) I1124 00:13:35.059406 178455 connection.py:360] Received response(xid=1): ("ccopy_reg\n_reconstructor\np1\n(cmysos.scheduler.state\nScheduler\np2\ncbuiltin\nobject\np3\nNtRp4\n(dp5\nS'framework_info'\np6\ncmesos.interface.mesos_pb2\nFrameworkInfo\np7\n(tRp8\n(dp9\nS'serialized'\np10\nS'\n\x07vagrant\x12\x05mysos!\x00\x00\x00\x00\x00\x00N@(\x012\x01*'\np11\nsbsS'clusters'\np12\ng1\n(ctwitter.common.collections.orderedset\nOrderedSet\np13\ng3\nNtRp14\n(dp15\nS'map'\np16\n(dp17\nsS'end'\np18\n(lp19\nNag19\nag19\nasbsb.", ZnodeStat(czxid=259437, mzxid=259437, ctime=1448290870564, mtime=1448290870564, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=410, numChildren=0, pzxid=259437)) I1124 00:13:35.059720 178455 mysos_scheduler.py:262] Successfully restored scheduler state 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@log_env@716: Client environment:host.name=hare229 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@log_env@724: Client environment:os.arch=3.16.0-30-generic 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@log_env@725: Client environment:os.version=#40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 I1124 00:13:35.066289 178455 sched.cpp:164] Version: 0.25.0 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@log_env@733: Client environment:user.name=pangbingqiang 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@log_env@741: Client environment:user.home=/root 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@log_env@753: Client environment:user.dir=/home/huajianfeng/incubator-cotton-master 2015-11-24 00:13:35,066:178455(0x7f36717f2700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=10.175.100.231:2184 sessionTimeout=10000 watcher=0x7f3690e18692 sessionId=0 sessionPasswd= context=0x7f3630000e90 flags=0 2015-11-24 00:13:35,070:178455(0x7f365ffff700):ZOO_INFO@check_events@1703: initiated connection to server [10.175.100.231:2184] 2015-11-24 00:13:35,073:178455(0x7f365ffff700):ZOO_INFO@check_events@1750: session establishment complete on server [10.175.100.231:2184], sessionId=0x15134d48dc00014, negotiated timeout=10000 I1124 00:13:35.073601 178493 group.cpp:331] Group process (group(1)@10.175.102.229:58721) connected to ZooKeeper I1124 00:13:35.073673 178493 group.cpp:805] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I1124 00:13:35.073709 178493 group.cpp:403] Trying to create path '/mesos' in ZooKeeper I1124 00:13:35.076272 178487 detector.cpp:156] Detected a new leader: (id='56') I1124 00:13:35.076438 178492 group.cpp:674] Trying to get '/mesos/info_0000000056' in ZooKeeper W1124 00:13:35.077445 178479 detector.cpp:444] Leading master master@10.175.102.230:5050 is using a Protobuf binary format when registering with ZooKeeper (info): this will be deprecated as of Mesos 0.24 (see MESOS-2340) I1124 00:13:35.077500 178479 detector.cpp:481] A new leading master (UPID=master@10.175.102.230:5050) is detected I1124 00:13:35.077623 178476 sched.cpp:262] New master detected at master@10.175.102.230:5050 I1124 00:13:35.077847 178476 sched.cpp:272] No credentials provided. Attempting to register without authentication Bottle v0.11.6 server starting up (using CherryPyServer())... Listening on http://0.0.0.0:55001/ Hit Ctrl-C to quit.