search

docker-archive / for-aws

92 stars 26 forks source link

Using cloudstor and l4controller-aws outside of docker4AWS #109

Closed rgardam closed 6 years ago

rgardam commented 7 years ago

Hi, I have started to build my own swarm cluster on AWS using terraform and love the idea of integrating persistent volumes and dynamic elb's into my cluster.

Is there any reason why I can't just deploy these containers to my swarm cluster nodes and have it work?

I can understand there might be some magic that happens in moby to make this work, but i'd like to not use cloudformations if I can avoid it.

Thanks, Rob

ddebroy commented 7 years ago

@rgardam that should be possible. You need to keep l4controller running on the leader and pass it the necessary mounts, config file as you can see in the userdata section of the ASG launch config in the CFN.

Several people do use cloudstor out of band ... you need to just run the plugin install command from the userdata on each node to get that installed. Note that if you want EFS support and deploy your own domain name resolution/DC, you need to make sure EC2 name resolution cam still happen for the EFS DNS.

rgardam commented 7 years ago

ok, this is great to know.

I will try it out and let you know. 👍

FrenchBen commented 7 years ago

@rgardam I would recommend using the version in Infrakit instead: https://github.com/docker/infrakit/blob/master/docs/controller/ingress/example.yml

iget-master commented 6 years ago

I'm trying to use cloudstor outside of d4aws, but I'm stuck on authentication issue.

Everytime I try to create a volume it returns me:

ERROR: create production_storage: VolumeDriver.Create: volume creation failed: NoCredentialProviders: no valid providers in chain. Deprecated.

I didn't found where should I put the AWS IAM credentials to feed the cloudstor. I think that d4aws does it automatically, right?

Where can I found documentation aboud cloudstor? It's a really pain to search about it!

ddebroy commented 6 years ago

@iget-master can you try the following IAM configuration on all nodes where you install cloudstor:

"Action": [
    "ec2:CreateTags",
    "ec2:AttachVolume",
    "ec2:DetachVolume",
    "ec2:CreateVolume",
    "ec2:DeleteVolume",
    "ec2:DescribeVolumes",
    "ec2:DescribeVolumeStatus",
    "ec2:CreateSnapshot",
    "ec2:DeleteSnapshot",
    "ec2:DescribeSnapshots"
],
"Effect": "Allow",
"Resource": "*"

You are correct that the IAM configuration above is done by the Docker4AWS CloudFormation template. So you can take a look at that through https://docs.docker.com/docker-for-aws/#docker-community-edition-ce-for-aws. Any specific reasons for not using Docker4AWS?

iget-master commented 6 years ago

@ddebroy Thanks for the response. I'm running a docker together with a plesk setup, and want to keep volumes on EBS.

I've already created a IAM exactly like your, but I don't know how to attach it to the instance itself. I know how to attach IAM's to users, groups.

The documentation about it is very confusing and difuse. :-(

ddebroy commented 6 years ago

@iget-master you need to attach the IAM to a role and configure the role on the instances/ASGs. See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

iget-master commented 6 years ago

@ddebroy Thank you, you really helped me.

For anyone that want to know how to use Cloustor outside Docker4AWS that's the way:

"Action": [
    "ec2:CreateTags",
    "ec2:AttachVolume",
    "ec2:DetachVolume",
    "ec2:CreateVolume",
    "ec2:DeleteVolume",
    "ec2:DescribeVolumes",
    "ec2:DescribeVolumeStatus",
    "ec2:CreateSnapshot",
    "ec2:DeleteSnapshot",
    "ec2:DescribeSnapshots"
],
"Effect": "Allow",
"Resource": "*"
# Without EFS Support
docker plugin install --alias cloudstor:aws --grant-all-permissions docker4x/cloudstor:17.06.0-ce-aws2 AWS_REGION=us-east-1 CLOUD_PLATFORM=AWS EFS_SUPPORTED=0

# With EFS Support
# Details on https://github.com/docker/for-aws/issues/85
docker plugin install --alias cloudstor:aws --grant-all-permissions docker4x/cloudstor:17.06.0-ce-aws2 CLOUD_PLATFORM=AWS EFS_ID_REGULAR=fs-abcd0123 AWS_REGION=us-east-2 EFS_SUPPORTED=1 DEBUG=1 AWS_STACK_ID=nostack EFS_ID_MAXIO=fs-abcd2222

:-)

mgpoly commented 6 years ago

@ddebroy Re using Docker-for-AWS containers out of band: I already have an EC2 VPC and I don't need any new subnets, security groups, gateways, ELBs, EFSs, etc. I just want to run a Docker swarm with HA. So it looks like I could just cherry pick various docker4x/* containers (and supporting SQS) listed in the user data section of the CF template. It looks like these containers auto remove/add a swarm node when a new EC2 instance is terminated/created:

Most of the services I want to run are meant to be internal: I don't want them exposed by public IPs and I only want to expose one port to our end users (so I definitely do not want l4controller).

matthewmrichter commented 5 years ago

I just stumbled upon this page - I've been having a lot of trouble finding any kind of documentation of how to manually install and configure Cloudstor - am I missing a documentation page somewhere? I only see the one that basically lists step one as "Make sure Cloudstor is already installed" which is not useful.

Unless it's a "Well you should just buy commercial docker support and ask us" thing.

christianacca commented 5 years ago

I found the following resource helpful - albeit specific to azure: https://jmaitrehenry.ca/2017/09/25/building-a-docker-swarm-on-azure/