search

Smithx10 / nomad-driver-triton

HashiCorp Nomad Triton driver plugin
15 stars 6 forks source link

Feature Request: Docker Support #7

Closed axisofentropy closed 5 years ago

axisofentropy commented 5 years ago

Managing Triton containers using Nomad is really valuable, but I think managing Docker containers ON Triton with Nomad could be even better. I tried using Nomad's existing Docker driver without success, since it seems to require the Docker host to be on the same system as the Nomad client. I posted about my experience here https://groups.google.com/forum/#!topic/nomad-tool/DXFQVTo2KyY

Could this Triton Nomad driver support docker containers? Or should I copy and modify the existing Docker Driver to better work with Triton's Docker? Or should I use some other tool like Docker Swarm?

Smithx10 commented 5 years ago

Give me a few days, we will probably need this internally to make some people happy, and I had code in virtual kubelet to do this. I probably can rig something up for ya.

axisofentropy commented 5 years ago

Thanks! Let me know how I can help!

Smithx10 commented 5 years ago

@axisofentropy I didn't forget about ya.

I found some time to work on this issue this morning.

At the current state I've got the follow config working. But will add some more features soon, and hopefully get a branch pushed for you to try.

        taskConfigSpec = hclspec.NewObject(map[string]*hclspec.Spec{
                "api_type": hclspec.NewAttr("api_type", "string", true),
                "docker_api": hclspec.NewBlock("docker_api", false, hclspec.NewObject(map[string]*hclspec.Spec{
                        "cmd":        hclspec.NewAttr("cmd", "list(string)", false),
                        "entrypoint": hclspec.NewAttr("entrypoint", "list(string)", false),
                        "openstdin":  hclspec.NewAttr("openstdin", "bool", false),
                        "stdinonce":  hclspec.NewAttr("stdinonce", "bool", false),
                        "tty":        hclspec.NewAttr("tty", "bool", false),
                        "workingdir": hclspec.NewAttr("workingdir", "string", false),
                        //"labels":         hclspec.NewAttr("labels", "list(map(string))", false),
                        "public_network":  hclspec.NewAttr("public_network", "string", false),
                        "private_network": hclspec.NewAttr("private_network", "string", false),
                        "image": hclspec.NewBlock("image", true, hclspec.NewObject(map[string]*hclspec.Spec{
                                "name":      hclspec.NewAttr("name", "string", true),
                                "auto_pull": hclspec.NewAttr("auto_pull", "bool", false),
                        })),

                        //"restart_policy": hclspec.NewAttr("restart_policy", "string", false),
                })),
                "cloud_api": hclspec.NewBlock("cloud_api", false, hclspec.NewObject(map[string]*hclspec.Spec{
                        "image": hclspec.NewBlock("image", true, hclspec.NewObject(map[string]*hclspec.Spec{
                                "name":        hclspec.NewAttr("name", "string", false),
                                "uuid":        hclspec.NewAttr("uuid", "string", false),
                                "version":     hclspec.NewAttr("version", "string", false),
                                "most_recent": hclspec.NewAttr("most_recent", "bool", false),
                        })),
                        "networks": hclspec.NewBlockList("networks", hclspec.NewObject(map[string]*hclspec.Spec{
                                "name":   hclspec.NewAttr("name", "string", false),
                                "public": hclspec.NewAttr("public", "bool", false),
                                "uuid":   hclspec.NewAttr("uuid", "string", false),
                        })),
                        "user_data":    hclspec.NewAttr("user_data", "string", false),
                        "cloud_config": hclspec.NewAttr("cloud_config", "string", false),
                        "user_script":  hclspec.NewAttr("user_script", "string", false),
                })),
                "tags":      hclspec.NewBlockAttrs("tags", "string", false),
                "affinity":  hclspec.NewAttr("affinity", "list(string)", false),
                "fwenabled": hclspec.NewAttr("fwenabled", "bool", false),
                "fwrules":   hclspec.NewBlockAttrs("fwrules", "string", false),
                "cns":       hclspec.NewAttr("cns", "list(string)", false),
                "package": hclspec.NewBlock("package", true, hclspec.NewObject(map[string]*hclspec.Spec{
                        "name":    hclspec.NewAttr("name", "string", false),
                        "uuid":    hclspec.NewAttr("uuid", "string", false),
                        "version": hclspec.NewAttr("version", "string", false),
                })),
        })
Smithx10 commented 5 years ago

@axisofentropy

https://github.com/Smithx10/nomad-driver-triton/tree/alpha

Has pretty much everything you need to get started.

Here are some known issues:

I delete the instance outside of the docker API so it doesn't clean up the fwrules, I am going to add a go routine for that.

image.auto_pull currently doesn't do anything. I will add that method sometime in the near future. If there are more docker thingies you want on input, just open an issue.

Let me know how it goes.

Thank You, Bruce.

Smithx10 commented 5 years ago

@axisofentropy

I've cleared up those Issues with this commit, https://github.com/Smithx10/nomad-driver-triton/commit/4318c1704c7f6d0dee41deb997426e0778ae920c

I've gotta look at what part of the DockerAPI CloudAPI has implemented, and add all of those options to the docker_api api_type.

Smithx10 commented 5 years ago

@axisofentropy

On the alpha branch I believe the following are working properly. 

        taskConfigSpec = hclspec.NewObject(map[string]*hclspec.Spec{
                "api_type": hclspec.NewAttr("api_type", "string", true),
                "docker_api": hclspec.NewBlock("docker_api", false, hclspec.NewObject(map[string]*hclspec.Spec{
                        "cmd":             hclspec.NewAttr("cmd", "list(string)", false),
                        "entrypoint":      hclspec.NewAttr("entrypoint", "list(string)", false),
                        "openstdin":       hclspec.NewAttr("openstdin", "bool", false),
                        "stdinonce":       hclspec.NewAttr("stdinonce", "bool", false),
                        "tty":             hclspec.NewAttr("tty", "bool", false),
                        "workingdir":      hclspec.NewAttr("workingdir", "string", false),
                        "hostname":        hclspec.NewAttr("hostname", "string", false),
                        "dns":             hclspec.NewAttr("dns", "list(string)", false),
                        "dns_search":      hclspec.NewAttr("dns_search", "list(string)", false),
                        "extra_hosts":     hclspec.NewAttr("extra_hosts", "list(string)", false),
                        "user":            hclspec.NewAttr("user", "string", false),
                        "domain_name":     hclspec.NewAttr("domain_name", "string", false),
                        "labels":          hclspec.NewBlockAttrs("labels", "string", false),
                        "public_network":  hclspec.NewAttr("public_network", "string", false),
                        "private_network": hclspec.NewAttr("private_network", "string", false),
                        "log_config": hclspec.NewBlock("log_config", false, hclspec.NewObject(map[string]*hclspec.Spec{
                                "type":   hclspec.NewAttr("type", "string", false),
                                "config": hclspec.NewBlockAttrs("config", "string", false),
                        })),
                        "ports": hclspec.NewBlock("ports", false, hclspec.NewObject(map[string]*hclspec.Spec{
                                "tcp":         hclspec.NewAttr("tcp", "list(number)", false),
                                "udp":         hclspec.NewAttr("udp", "list(number)", false),
                                "publish_all": hclspec.NewAttr("publish_all", "bool", false),
                        })),
                        "image": hclspec.NewBlock("image", true, hclspec.NewObject(map[string]*hclspec.Spec{
                                "name":      hclspec.NewAttr("name", "string", true),
                                "tag":       hclspec.NewAttr("tag", "string", false),
                                "auto_pull": hclspec.NewAttr("auto_pull", "bool", false),
                        })),
                        "restart_policy": hclspec.NewAttr("restart_policy", "string", false),
                })),
                "cloud_api": hclspec.NewBlock("cloud_api", false, hclspec.NewObject(map[string]*hclspec.Spec{
                        "image": hclspec.NewBlock("image", true, hclspec.NewObject(map[string]*hclspec.Spec{
                                "name":        hclspec.NewAttr("name", "string", false),
                                "uuid":        hclspec.NewAttr("uuid", "string", false),
                                "version":     hclspec.NewAttr("version", "string", false),
                                "most_recent": hclspec.NewAttr("most_recent", "bool", false),
                        })),
                        "networks": hclspec.NewBlockList("networks", hclspec.NewObject(map[string]*hclspec.Spec{
                                "name": hclspec.NewAttr("name", "string", false),
                                "uuid": hclspec.NewAttr("uuid", "string", false),
                        })),
                        "user_data":    hclspec.NewAttr("user_data", "string", false),
                        "cloud_config": hclspec.NewAttr("cloud_config", "string", false),
                        "user_script":  hclspec.NewAttr("user_script", "string", false),
                })),
                "tags":      hclspec.NewBlockAttrs("tags", "string", false),
                "affinity":  hclspec.NewAttr("affinity", "list(string)", false),
                "fwenabled": hclspec.NewAttr("fwenabled", "bool", false),
                "fwrules":   hclspec.NewBlockAttrs("fwrules", "string", false),
                "cns":       hclspec.NewAttr("cns", "list(string)", false),
                "package": hclspec.NewBlock("package", true, hclspec.NewObject(map[string]*hclspec.Spec{
                        "name":    hclspec.NewAttr("name", "string", false),
                        "uuid":    hclspec.NewAttr("uuid", "string", false),
                        "version": hclspec.NewAttr("version", "string", false),
                })),
        })
Smithx10 commented 5 years ago

I've merged this into master, please try it, and file more issues. Happy Dockering.

axisofentropy commented 5 years ago

Neat! Is this implementation talking to Docker API, Cloud API, or both?

Smithx10 commented 5 years ago

Both, I tried to take advantage of Docker, but not neglect the features that triton has.... so the FWRules and tags will be applied to the instance after provisioned, but before it is started.

Check this code block for more clarification. https://github.com/Smithx10/nomad-driver-triton/blob/master/plugin/triton.go#L248 Once this becomes rather stable I'll invest time into writing some Docs / Tests so that it's more user friendly.

Smithx10 commented 5 years ago

@axisofentropy Were you able to try this?

axisofentropy commented 5 years ago

Not yet. We're gonna explore Nomad on KVM/bhyve zones on Triton first. What do you think we'll be missing out on with that approach?

On Mon, Apr 29, 2019, 21:03 Smithx10 notifications@github.com wrote:

@axisofentropy https://github.com/axisofentropy Were you able to try this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Smithx10/nomad-driver-triton/issues/7#issuecomment-487801455, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAQGCYFLEARV6TIUUQNGZ3PS6SFNANCNFSM4HIIX7OQ .

Smithx10 commented 5 years ago

The plugin should schedule and behave pretty decent. We use the Nomad Triton Provider to grow our Nomad Cluster to deploy Docker workloads that don't work in LX branded zones.