search

twitter-archive / mysos

Cotton (formerly known as Mysos)
https://incubator.apache.org/projects/cotton.html
588 stars 67 forks source link

Cannot build package #40

Open eirslett opened 9 years ago

eirslett commented 9 years ago

Or it might just be me you cannot figure out how.

Steps to reproduce:

First, start with a clean Ubuntu 14.04 x64 server (e.g. from digitalocean.com) Then, run these commands:

# ===========
# Follow instructions from https://docs.mesosphere.com/getting-started/datacenter/install/
# Setup
apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]')
CODENAME=$(lsb_release -cs)

# Add the repository
echo "deb http://repos.mesosphere.io/${DISTRO} ${CODENAME} main" | \
  sudo tee /etc/apt/sources.list.d/mesosphere.list
apt-get update

apt-get install mesos marathon
# ===========

apt-get install git
git clone https://github.com/twitter/mysos.git
cd mysos
apt-get install python-pip
pip install virtualenv
virtualenv venv
source venv/bin/activate
cd 3rdparty
wget http://downloads.mesosphere.io/master/ubuntu/14.04/mesos-0.22.1-py2.7-linux-x86_64.egg
wheel convert mesos-0.22.1-py2.7-linux-x86_64.egg
cd ..

python setup.py install
mysos_scheduler

Expected: The scheduler should start? Actual:

Traceback (most recent call last):
  File "/root/mysos/venv/bin/mysos_scheduler", line 9, in <module>
    load_entry_point('mysos==0.1.0.dev0', 'console_scripts', 'mysos_scheduler')()
  File "/root/mysos/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 552, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/root/mysos/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2672, in load_entry_point
    return ep.load()
  File "/root/mysos/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2344, in load
    self.require(*args, **kwargs)
  File "/root/mysos/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2361, in require
    items = working_set.resolve(reqs, env, installer)
  File "/root/mysos/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 833, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pynacl<1,>=0.3.0' distribution was not found and is required by the application
xujyan commented 9 years ago

I am sorry that documentation on this is lacking but basically the scheduler requires extra dependencies which can be installed via pip install mysos[scheduler].

https://github.com/twitter/mysos/blob/master/vagrant/bin/mysos_scheduler.sh has the example for installing mysos scheduler.

In this example, /home/vagrant/mysos/dist has the mysos package and /home/vagrant/mysos/deps contains the mesos.native package.

BTW mysos currently requires Mesos native egg version to be 0.20.1 (there's a ticket for upgrading it) and the egg being named mesos.native-*. See how https://github.com/twitter/mysos/blob/master/vagrant/provision-dev-cluster.sh#L55 downloads the native egg.

eirslett commented 9 years ago

Ok, I downloaded native binaries for 0.20.1. I get a little further, now there's another error...

I ran

apt-get upgrade # get python version 2.7.6
cp -r 3rdparty dist
python setup.py install
pip install mysos[scheduler]

and this is the error that pip shows:

Collecting mesos.native==0.20.1 (from mysos[scheduler])
/root/mysos/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Could not find a version that satisfies the requirement mesos.native==0.20.1 (from mysos[scheduler]) (from versions: )
  Some externally hosted files were ignored as access to them may be unreliable (use --allow-external mesos.native to allow).
No matching distribution found for mesos.native==0.20.1 (from mysos[scheduler])

I'm not sure if it's the urllib3/SSLContext error that is causing problems, I've never seen that error before. But it looks like it cannot find the native package...

Edit I also tried running pip install mysos[scheduler] --allow-external mesos.native (like the log suggested) but that didn't help at all. Edit 2 I fixed the SSL warning by following instructions at the link. I also fixed the mesos.native warning by running

cd 3rdparty
pip install pip install mesos.native-0.20.1-cp27-none-linux_x86_64.whl

which works, but I'm not sure if that's the "correct" way to do it.

But if you're going to run Mysos in production, you would build a .pex file for the scheduler, and a .pex file for the executor - like in the vagrant file?

yuriksan commented 9 years ago

I also had difficulties getting mysos to connect [Mysos scheduler is still connecting...]. I'm using the playa-mesos vagrant box as a start point. Single node master/slave on same server.

Question how is mysos-0.1.0-dev0.zip built?

Here is my startup script

!/bin/sh

10.0.2.15

ZK_HOST=10.141.141.10 API_PORT=55001

NOTE: In --executor_environ we are pointing MYSOS_DEFAULTS_FILE to an empty MySQL defaults file.

The file 'my5.6.cnf' is pre-installed by the 'mysql-server-5.6' package on the VM.

mysos_scheduler \ --verbose=on \ --port=$API_PORT \ --framework_user=vagrant \ --mesos_master=zk://$ZK_HOST:2181/mesos/master \ --admin_keypath=/home/vagrant/mysos/vagrant/etc/admin_keyfile.yml \ --scheduler_keypath=/home/vagrant/mysos/vagrant/etc/scheduler_keyfile.txt \ --executor_uri=/home/vagrant/mysos/dist/mysos-0.1.0-dev0.zip \ --executor_cmd=/home/vagrant/mysos/vagrant/bin/mysos_executor.sh \ --zk_url=zk://$ZK_HOST:2181/mysos \ --framework_failover_timeout=1m \ --executor_environ='[{"name": "MYSOS_DEFAULTS_FILE", "value": "/etc/mysql/conf.d/my5.6.cnf"}]'

Which outputs... /usr/lib/python2.7/dist-packages/pkg_resources.py:1031: UserWarning: /home/vagrant/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable). warnings.warn(msg, UserWarning) I0619 12:07:27.604989 4360 mysos_scheduler.py:177] Options in use: {'framework_failover_timeout': '1m', 'twitter_common_log_simple': False, 'verbose': 'on', 'twitter_common_app_daemon_stdout': '/dev/null', 'twitter_common_log_scribe_category': 'python_default', 'api_port': 55001, 'twitter_common_log_log_dir': '/var/tmp', 'twitter_common_app_daemonize': False, 'twitter_common_app_ignore_rc_file': False, 'twitter_common_app_profiling': False, 'work_dir': '/tmp/mysos', 'twitter_common_app_pidfile': None, 'twitter_common_log_scribe_buffer': False, 'executor_source_prefix': None, 'election_timeout': '60s', 'twitter_common_app_rc_filename': False, 'framework_role': '*', 'executor_environ': '[{"name": "MYSOS_DEFAULTS_FILE", "value": "/etc/mysql/conf.d/my5.6.cnf"}]', 'twitter_common_log_scribe_log_level': 'NONE', 'executor_uri': '/home/vagrant/mysos/dist/mysos-0.1.0-dev0-py2.7.egg', 'twitter_common_log_disk_log_level': 'NONE', 'twitter_common_log_stderr_log_level': 'ERROR', 'framework_authentication_file': None, 'state_storage': 'zk', 'executor_cmd': '/home/vagrant/mysos/vagrant/bin/mysos_executor.sh', 'twitter_common_app_profile_output': None, 'framework_user': 'vagrant', 'zk_url': 'zk://10.141.141.10:2181/mysos', 'twitter_common_app_debug': False, 'twitter_common_log_scribe_port': 1463, 'twitter_common_log_scribe_host': 'localhost', 'scheduler_keypath': '/home/vagrant/mysos/vagrant/etc/scheduler_keyfile.txt', 'installer_args': None, 'backup_store_args': None, 'mesos_master': 'zk://10.141.141.10:2181/mesos/master', 'admin_keypath': '/home/vagrant/mysos/vagrant/etc/admin_keyfile.yml', 'twitter_common_app_daemon_stderr': '/dev/null'} I0619 12:07:27.631294 4360 mysos_scheduler.py:219] Extracted web assets into /tmp/mysos I0619 12:07:27.631490 4360 mysos_scheduler.py:244] Starting Mysos scheduler D0619 12:07:27.635765 4360 connection.py:456] ZK loop started D0619 12:07:27.636231 4360 client.py:374] Skipping state change I0619 12:07:27.636354 4360 connection.py:566] Connecting to 10.141.141.10:2181 D0619 12:07:27.636468 4360 connection.py:570] Using session_id: None session_passwd: 00000000000000000000000000000000 I0619 12:07:27.637034 4360 connection.py:276] Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) D0619 12:07:27.640809 4360 connection.py:261] Read response Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=94019154140135465, passwd='V\x9e6Lf vmD\xd1\x88\xaa6\xb7R\x0b', read_only=False) D0619 12:07:27.640971 4360 connection.py:603] Session created, session_id: 94019154140135465 session_passwd: 569e364c6620766d44d188aa36b7520b negotiated session timeout: 10000 connect timeout: 10000 read timeout: 6666.66666667 I0619 12:07:27.641103 4360 client.py:378] Zookeeper connection established, state: CONNECTED I0619 12:07:27.644429 4360 mysos_scheduler.py:250] Using ZooKeeper (path: /mysos) for state storage I0619 12:07:27.645031 4360 connection.py:276] Sending request(xid=1): GetData(path='/mysos/state/scheduler', watcher=None) D0619 12:07:27.646624 4360 connection.py:401] Reading for header ReplyHeader(xid=1, zxid=486, err=0) I0619 12:07:27.646917 4360 connection.py:360] Received response(xid=1): ("ccopy_reg\n_reconstructor\np1\n(cmysos.scheduler.state\nScheduler\np2\ncbuiltin\nobject\np3\nNtRp4\n(dp5\nS'framework_info'\np6\ncmesos.interface.mesos_pb2\nFrameworkInfo\np7\n(tRp8\n(dp9\nS'serialized'\np10\nS'\n\x07vagrant\x12\x05mysos!\x00\x00\x00\x00\x00\x00N@(\x012\x05mysosB\x05mysos'\np11\nsbsS'clusters'\np12\ng1\n(ctwitter.common.collections.orderedset\nOrderedSet\np13\ng3\nNtRp14\n(dp15\nS'map'\np16\n(dp17\nsS'end'\np18\n(lp19\nNag19\nag19\nasbsb.", ZnodeStat(czxid=118, mzxid=118, ctime=1434628707473, mtime=1434628707473, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=424, numChildren=0, pzxid=118)) I0619 12:07:27.647811 4360 mysos_scheduler.py:262] Successfully restored scheduler state I0619 12:07:27.660214 4360 sched.cpp:139] Version: 0.20.1 2015-06-19 12:07:27,660:4360(0x7fdaa3fff700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2015-06-19 12:07:27,660:4360(0x7fdaa3fff700):ZOO_INFO@log_env@716: Client environment:host.name=mesos 2015-06-19 12:07:27,660:4360(0x7fdaa3fff700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2015-06-19 12:07:27,660:4360(0x7fdaa3fff700):ZOO_INFO@log_env@724: Client environment:os.arch=3.13.0-55-generic 2015-06-19 12:07:27,660:4360(0x7fdaa3fff700):ZOO_INFO@log_env@725: Client environment:os.version=#92-Ubuntu SMP Sun Jun 14 18:32:20 UTC 2015 2015-06-19 12:07:27,672:4360(0x7fdaa3fff700):ZOO_INFO@log_env@733: Client environment:user.name=vagrant 2015-06-19 12:07:27,672:4360(0x7fdaa3fff700):ZOO_INFO@log_env@741: Client environment:user.home=/home/vagrant 2015-06-19 12:07:27,672:4360(0x7fdaa3fff700):ZOO_INFO@log_env@753: Client environment:user.dir=/home/vagrant/mysos 2015-06-19 12:07:27,672:4360(0x7fdaa3fff700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=10.141.141.10:2181 sessionTimeout=10000 watcher=0x7fdab498acc0 sessionId=0 sessionPasswd= context=0x7fda8c0010d0 flags=0 2015-06-19 12:07:27,675:4360(0x7fda9bfff700):ZOO_INFO@check_events@1703: initiated connection to server [10.141.141.10:2181] 2015-06-19 12:07:27,682:4360(0x7fda9bfff700):ZOO_INFO@check_events@1750: session establishment complete on server [10.141.141.10:2181], sessionId=0x14e05ec2c9b002a, negotiated timeout=10000 I0619 12:07:27.683342 4369 group.cpp:313] Group process (group(1)@127.0.1.1:56730) connected to ZooKeeper I0619 12:07:27.683377 4369 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0619 12:07:27.683393 4369 group.cpp:385] Trying to create path '/mesos/master' in ZooKeeper Bottle v0.11.6 server starting up (using CherryPyServer())... Listening on http://0.0.0.0:55001/ Hit Ctrl-C to quit.

D0619 12:07:30.685837 4360 connection.py:276] Sending request(xid=-2): Ping() D0619 12:07:30.687498 4360 connection.py:386] Received Ping D0619 12:07:34.005599 4360 connection.py:276] Sending request(xid=-2): Ping() D0619 12:07:34.007508 4360 connection.py:386] Received Ping D0619 12:07:37.255810 4360 connection.py:276] Sending request(xid=-2): Ping() D0619 12:07:37.258078 4360 connection.py:386] Received Ping D0619 12:07:40.526374 4360 connection.py:276] Sending request(xid=-2): Ping() D0619 12:07:40.528445 4360 connection.py:386] Received Ping I0619 12:07:41.477478 4360 mysos_scheduler.py:331] Interrupted, exiting. 2015-06-19 12:07:41,482:4360(0x7fdab9edf740):ZOO_INFO@zookeeper_close@2505: Closing zookeeper sessionId=0x14e05ec2c9b002a to [10.141.141.10:2181]

I have this in ZK under /mysos/state/scheduler

ccopy_reg _reconstructor p1 (cmysos.scheduler.state Scheduler p2 cbuiltin object p3 NtRp4 (dp5 S'framework_info' p6 cmesos.interface.mesos_pb2 FrameworkInfo p7 (tRp8 (dp9 S'serialized' p10 S'\n\x07vagrant\x12\x05mysos!\x00\x00\x00\x00\x00\x00N@(\x012\x05mysosB\x05mysos' p11 sbsS'clusters' p12 g1 (ctwitter.common.collections.orderedset OrderedSet p13 g3 NtRp14 (dp15 S'map' p16 (dp17 sS'end' p18 (lp19 Nag19 ag19 asbsb.