search

mesos / mr-redis

Redis on Apache Mesos
Apache License 2.0
127 stars 32 forks source link

Not able to deploy mr-redis Cluster on mesos. #52

Open deepakkashyap opened 7 years ago

deepakkashyap commented 7 years ago

OUTPUT:

curl -X "POST" mrredis.mesos:5656/v1/CREATE/redisCluster/100/1/2 2016/11/29 17:21:35 Instance Name=redisCluster, Capacity=100, masters=1, slaves=2 ConfigJson={1} ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/ ZOO K=/MrRedis/Instances/redisCluster/Type V=MS ZOO K=/MrRedis/Instances/redisCluster/Masters V=0 ZOO K=/MrRedis/Instances/redisCluster/Slaves V=0 ZOO K=/MrRedis/Instances/redisCluster/Capacity V=100 ZOO K=/MrRedis/Instances/redisCluster/ExpMasters V=1 ZOO K=/MrRedis/Instances/redisCluster/ExpSlaves V=2 ZOO K=/MrRedis/Instances/redisCluster/Status V=CREATING ZOO K=/MrRedis/Instances/redisCluster/Mname V= ZOO K=/MrRedis/Instances/redisCluster/DValue V=1 ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/Snames/ ZOO CREATE SECTION K=/MrRedis/Instances/redisCluster/Procs/ 2016/11/29 17:21:35 [router.go:854] [D] | POST | /v1/CREATE/redisCluster/100/1/2 | 56.157575ms | match | /v1/CREATE/:INSTANCENAME/:CAPACITY/:MASTERS/:SLAVES | 2016/11/29 17:21:35 Received offer {true 0xc4200de360 1} Request Accepted, Instance will be created.2016/11/29 17:21:35 Created 1 master offers for Instance redisCluster root@mesos-master1:/usr/local/go/src/github.com/mesos/mr-redis/sched# 2016/11/29 17:21:35 Received Offer with CPU=7.5 MEM=13982 OfferID=215731b7-c3e2-4ad9-8333-c3c7231af861-O5849452 2016/11/29 17:21:35 Launched 1 tasks from this offer 2016/11/29 17:21:35 Received Offer with CPU=8 MEM=14014 OfferID=215731b7-c3e2-4ad9-8333-c3c7231af861-O5849453 2016/11/29 17:21:35 Launched 0 tasks from this offer 2016/11/29 17:21:35 MrRedis Receives offer 2016/11/29 17:21:35 MrRedis Task Update received 2016/11/29 17:21:35 Status={redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 TASK_ERROR 215731b7-c3e2-4ad9-8333-c3c7231af861-S3 []} 2016/11/29 17:21:35 Received a Task update from the channel &{redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 TASK_ERROR 215731b7-c3e2-4ad9-8333-c3c7231af861-S3 []} 2016/11/29 17:21:35 Invalid Key /MrRedis/Instances/redisCluster/Procs/92b8edb7-96ee-4f1e-5f28-58a1a50b11a1, Cannot load 2016/11/29 17:21:35 Invalid Key /MrRedis/Instances/redisCluster/Procs/, Cannot load 2016/11/29 17:21:35 Task redisCluster::92b8edb7-96ee-4f1e-5f28-58a1a50b11a1 is Error

curl hostname:port/v1/STATUS/redisCluster {"Name":"redisCluster","Type":"MS","Status":"CREATING","Capacity":100,"Master":null,"Slaves":null}

Not able to debug whats wrong I'm doing.Can you please help?

dhilipkumars commented 7 years ago

Thanks for reporting, it is always useful with the Logs. Clearly something is wrong with the ZooKeeper drivers.
Looks like the Executor tries to write to Zookeeper and that write is not successful, do you have the executor logs? I can try and replicate this in my environment.

Is the zookeeper reachable from the slaves?

deepakkashyap commented 7 years ago

Yes, zookeeper is reachable Logs: Starting MrRedis Executor ZOO CREATE SECTION K=/MrRedis ZOO CREATE SECTION K=/MrRedis/Instances ZOO CREATE SECTION K=/MrRedis/Config InterfaceAddress = x.x.x.x Executor process has started and running. Registered Executor on slave Launching task trial::8ff66757-ea0d-4ca7-4a63-8dd870008369 with command The Redmon object = {0xc8201c2000 0 x.x.x.x 6380 false 0xc8200942a0 0xc82000aa50 redis:3.0-alpine 0xc820170640} Error creating a container error during connect: Post https://%2Fvar%2Frun%2Fdocker.sock/v1.22/containers/create?name=8ff66757-ea0d-4ca7-4a63-8dd870008369: http: server gave HTTP response to HTTPS client Total tasks launched 1 Executor Finished, Delete all the containers executor terminated

*I1208 18:23:16.718339 114025 fetcher.cpp:424] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/83371dc2-4360-42c6-bb7c-35006d4de0fe-S22\/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"http:\/\/x.x.x.x:8090\/MrRedisExecutor"}}],"sandbox_directory":"\/data\/mesos\/slaves\/83371dc2-4360-42c6-bb7c-35006d4de0fe-S22\/frameworks\/83371dc2-4360-42c6-bb7c-35006d4de0fe-1640\/executors\/default\/runs\/3ce8e1c8-09f4-4bbd-bc87-3aa33fbd11cf","user":"root"} I1208 18:23:16.719645 114025 fetcher.cpp:379] Fetching URI 'http://x.x.x.x:8090/MrRedisExecutor' I1208 18:23:16.719657 114025 fetcher.cpp:250] Fetching directly into the sandbox directory I1208 18:23:16.719669 114025 fetcher.cpp:187] Fetching URI 'http://x.x.x.x:8090/MrRedisExecutor' I1208 18:23:16.719676 114025 fetcher.cpp:134] Downloading resource from 'http://x.x.x.x:8090/MrRedisExecutor' to '/data/mesos/slaves/83371dc2-4360-42c6-bb7c-35006d4de0fe-S22/frameworks/83371dc2-4360-42c6-bb7c-35006d4de0fe-1640/executors/default/runs/3ce8e1c8-09f4-4bbd-bc87-3aa33fbd11cf/MrRedisExecutor' I1208 18:23:16.886005 114025 fetcher.cpp:456] Fetched 'http://x.x.x.x:8090/MrRedisExecutor' to '/data/mesos/slaves/83371dc2-4360-42c6-bb7c-35006d4de0fe-S22/frameworks/83371dc2-4360-42c6-bb7c-35006d4de0fe-1640/executors/default/runs/3ce8e1c8-09f4-4bbd-bc87-3aa33fbd11cf/MrRedisExecutor' 2016/12/08 18:23:16 Connected to x.x.x.x:2181 2016/12/08 18:23:16 Authenticated: id=96998585951780933, timeout=40000 I1208 18:23:16.945909 114030 executor.go:133] Init mesos executor driver I1208 18:23:16.945921 114030 executor.go:134] Protocol Version: 0.24.0 I1208 18:23:16.945996 114030 executor.go:524] Starting the executor driver I1208 18:23:16.946038 114030 http_transporter.go:383] listening on x.x.x.x port 34061 I1208 18:23:16.946104 114030 executor.go:559] Mesos executor is started with PID= executor(1)@x.x.x.x:34061 I1208 18:23:16.946113 114030 executor.go:626] Waiting for the executor driver to stop I1208 18:23:16.948471 114030 executor.go:305] Executor driver registered I1208 18:23:16.948479 114030 executor.go:318] Registered on slave &SlaveID{Value:83371dc2-4360-42c6-bb7c-35006d4de0fe-S22,XXX_unrecognized:[],} I1208 18:23:16.950577 114030 executor.go:402] Executor driver runTask I1208 18:23:16.950593 114030 executor.go:416] Executor asked to run task '&TaskID{Value:trial::8ff66757-ea0d-4ca7-4a63-8dd870008369,XXX_unrecognized:[],}' I1208 18:23:23.577681 114030 executor.go:687] Executor sending status update &StatusUpdate{FrameworkId:&FrameworkID{Value:83371dc2-4360-42c6-bb7c-35006d4de0fe-1640,XXX_unrecognized:[],},ExecutorId:&ExecutorID{Value:default,XXX_unrecognized:[],},SlaveId:&SlaveID{Value:83371dc2-4360-42c6-bb7c-35006d4de0fe-S22,XXX_unrecognized:[],},Status:&TaskStatus{TaskId:&TaskID{Value:trial::8ff66757-ea0d-4ca7-4a63-8dd870008369,XXX_unrecognized:[],},State:TASK_ERROR,Data:nil,Message:nil,SlaveId:&SlaveID{Value:83371dc2-4360-42c6-bb7c-35006d4de0fe-S22,XXX_unrecognized:[],},Timestamp:1.481201603e+09,ExecutorId:nil,Healthy:nil,Source:nil,Reason:nil,Uuid:nil,Labels:nil,ContainerStatus:nil,XXX_unrecognized:[],},Timestamp:1.481201603e+09,Uuid:[77 185 135 129 189 69 17 230 154 65 6 241 36 82 183 249],LatestState:nil,XXX_unrecognized:[],} I1208 18:23:23.580006 114030 executor.go:445] Executor statusUpdateAcknowledgement I1208 18:23:23.580015 114030 executor.go:448] Receiving status update acknowledgement &StatusUpdateAcknowledgementMessage{SlaveId:&SlaveID{Value:83371dc2-4360-42c6-bb7c-35006d4de0fe-S22,XXX_unrecognized:[],},FrameworkId:&FrameworkID{Value:83371dc2-4360-42c6-bb7c-35006d4de0fe-1640,XXX_unrecognized:[],},TaskId:&TaskID{Value:trial::8ff66757-ea0d-4ca7-4a63-8dd870008369,XXX_unrecognized:[],},Uuid:[77 185 135 129 189 69 17 230 154 65 6 241 36 82 183 249],XXX_unrecognized:[],} I1208 18:43:24.014788 114030 executor.go:489] Executor driver received shutdown I1208 18:43:24.014807 114030 executor.go:496] Executor driver is asked to shutdown I1208 18:43:24.014816 114030 executor.go:572] Stopping the executor driver I1208 18:43:24.014822 114030 messenger.go:295] stopping messenger executor(1)@x.x.x.x:34061.. I1208 18:43:24.014838 114030 http_transporter.go:447] stopping HTTP transport**

dhilipkumars commented 7 years ago

@deepakkashyap This is interesting. Mr-redis comes with custom executor, it appears that it is unable to connect to the docker daemon at the mesos-agent.

The customer executor will automatically perform a docker pull by communicating with the docker daemon, and then perform a docker run.

dhilipkumars commented 7 years ago

in the config file , could you supply the actual "username" which has permission to access the docker unix:///var/run/docker.sock socket file? If you have installed docker using a sudo command then its safe to supply 'root' for the UserName attribute in the config file.

$cat config.json
{
   "UserName": "root",
   "FrameworkName": "MrRedis",
   "Master": "127.0.0.1:5050",
   "ExecutorPath": "./MrRedisExecutor",
   "RedisImage": "redis:3.0-alpine",
   "DBType": "zookeeper",, 
   "DBEndPoint": "127.0.0.1:2181"", 
   "DBType": "etcd", 
   "DBEndPoint": "127.0.0.1:2379", 
   "LogFile": "stderr",
   "ArtifactIP": "127.0.0.1",
   "ArtifactPort": "5454",
   "HTTPPort": "5656"
}

Ofcourse I'm assume that docker daemon is pre-installed on all your mesos Agents. :-)

deepakkashyap commented 7 years ago

I am running framework as root user only. Config file: { "UserName": "root", "Master": "127.0.0.1:5050", "FrameworkName": "MrRedis", "ArtifactIP": "127.0.0.1", "ExecutorPath": "/usr/local/go/src/github.com/mesos/mr-redis/exec/MrRedisExecutor", "RedisImage": "redis:3.0-alpine", "ArtifactPort": "8089", "HTTPPort": "5656", "DBType": "zookeeper", "DBEndPoint": "127.0.0.1:2181" }

dhilipkumars commented 7 years ago

@deepakkashyap what is the docker version running on the 'Agent'? Do you have any redis containers already running on that slave? 

$docker ps
deepakkashyap commented 7 years ago

Docker version is 1.11.2 and Redis container is not running on any of the slave.