westfood
commented
6 years ago I guess I have an idea, i wanted to bind Worker instances with Elastic IP. So I updated user data. But I was not sure how local IP is used in Worker nodes, so I exported it again after IP assigment.
"# guide-aws\n",
"docker run --label com.docker.editions.system --log-driver=json-file --log-opt max-size=50m --name=guide-aws --restart=always -d -e DYNAMODB_TABLE=$DYNAMODB_TABLE -e NODE_TYPE=$NODE_TYPE -e REGION=$AWS_REGION -e STACK_NAME=$STACK_NAME -e INSTANCE_NAME=$INSTANCE_NAME -e VPC_ID=$VPC_ID -e STACK_ID=\"$STACK_ID\" -e ACCOUNT_ID=$ACCOUNT_ID -e SWARM_QUEUE=\"$SWARM_QUEUE\" -e CLEANUP_QUEUE=\"$CLEANUP_QUEUE\" -e RUN_VACUUM=$RUN_VACUUM -e DOCKER_FOR_IAAS_VERSION=$DOCKER_FOR_IAAS_VERSION -e EDITION_ADDON=$EDITION_ADDON -e HAS_DDC=$HAS_DDC -e CHANNEL=$CHANNEL -v /var/run/docker.sock:/var/run/docker.sock docker4x/guide-aws:$DOCKER_FOR_IAAS_VERSION\n",
"\n",
"# AC Elastic IP association\n",
"export ELASTIC_ID=$(docker exec guide-aws aws ec2 describe-addresses --region us-west-2 --query 'Addresses[?AssociationId==null].{AllocationID:AllocationId}[0]' --output text)\n",
"export INSTANCE_ID=$(wget -qO- http://169.254.169.254/latest/meta-data/instance-id)\n",
"docker exec guide-aws aws ec2 associate-address --instance-id ${INSTANCE_ID} --allocation-id ${ELASTIC_ID} --no-allow-reassociation --region us-west-2\n",
"# AC - not sure if this address is used inside host, setting it again. Because Elastic IP could change Private IP.\n",
"export LOCAL_IP=$(wget -qO- http://169.254.169.254/latest/meta-data/local-ipv4)\n",
"\n",
"# cloudstor\n",
"docker plugin install --alias cloudstor:aws --grant-all-permissions docker4x/cloudstor:$DOCKER_FOR_IAAS_VERSION CLOUD_PLATFORM=AWS EFS_ID_REGULAR=$EFS_ID_REGULAR EFS_ID_MAXIO=$EFS_ID_MAXIO AWS_REGION=$AWS_REGION AWS_STACK_ID=$STACK_ID EFS_SUPPORTED=$ENABLE_EFS DEBUG=1\n",
"\n",I checked the EC2 logs, association went fine, but I could not get cloudstor:
{
"AssociationId": "eipassoc-19bff0d4"
}
Error response from daemon: Get https://registry-1.docker.io/v2/docker4x/cloudstor/manifests/17.12.0-ce-aws1: net/http: TLS handshake timeout
[ ok ]
Expected behavior
Worker runs tasks backed with Cloudstor EBS volume.
Actual behavior
Worker wont start task with error: no suitable node (scheduling constraints not satisfied on 5 nodes; missing plugin on 3 nodes.
Information
I updated user-data for manager and worker instances. So all instances has been restarted, all went fine. But two tasks depends on cloudstor EBS volume did not stared. And docker wont start them.
Cloudstor is installed on manager nodes, it seems worker nodes does not have cloudstor installed.
Maybe I miss something obvious. But cloudstor work before restarting all instances.
Steps to reproduce the behavior