Proposal: Driver architecture

[sthulb]

Proposal: Driver architecture

THIS PROPOSAL IS A WORK IN PROGRESS

This is a proposal to change the architecture of drivers. I think it has been universally agreed by now that drivers are going to be a maintenance problem in the future, if you don't already consider them one already.

I would like to see a point in the near future where drivers aren't a part of the machine repository, but instead they live outside in their own repo, maintained by people who work with that provider.

Architecture

Drivers will become self contained binaries, communicating over unix sockets and TCP sockets (windows).

The binaries would be compiled as docker-machine-driver-<provider>, i.e. docker-machine-driver-aws-ec2

I'm not particularly fond of shelling out to things for communication, for that reason, communicating over a socket is nicer. The driver binary would be responsible for sending it's configuration options to docker-machine via the socket, docker-machine would be responsible for interpreting the configuration options to the user using the CLI.

Drivers would still be responsible for creating, reading, updating, and deleting hosts on the provider.

Machine would provide a versioned API to ensure drivers work with certain versions of machine. Sharing a token to ensure compatibility between the driver and machine.

The lifetime of binary would be short, with the driver binary only running whilst needed.

Machine would have to have a framework capable for supporting complex drivers, providing registration, socket communication and generic implementations. Machine would also be responsible for versioning of the API.

[sthulb]

cc: @ehazlett @nathanleclaire @bfirsh

[ehazlett]

Related to #286

[jeffmendoza]

It seems to me like one of machine's goals is to unify and standardize the 'getting started with docker' experience between docker-host options. Example: trying to standardize the distro, docker daemon setup, even the hostname pattern.

Splitting up and loosely coupling the drivers would be counter to that goal. I don't necessarily think this is a bad thing, but rather a larger choice to be made on the tradeoffs of maintenance effort for a consistent user experience, and limiting the pace and breadth of driver development.

[sthulb]

@jeffmendoza If we don't, we end up having to maintain drivers that we can't test. We're seeing drivers for some of the more esoteric providers.

[sthulb]

Obviously, a number of "core" drivers would be maintained by machine devs

[ehazlett]

Thanks for the write up.

I've been trying to think through this as well. My fear of splitting everything out into sub-binaries is what @jeffmendoza said. Once we push all of these out to the providers we could lose a bit of the unification that machine provides. There are those that don't like the "bloat" of machine currently, however, I also feel that this single entrypoint provides a consistent experience across all drivers.

I'm not saying we shouldn't split them out; rather continuing the discussion. One other thing to think about is where would these things reside (maintainer CDN, github, etc) and how would the user get started? Would we provide a distribution system for the separate binaries (docker images perhaps but that won't work for some of the drivers i.e. hyper-v, fusion, virtualbox)?

Perhaps we could accomplish the same concept similar to how Docker currently does it: we don't need to split them all out but rather define maintainers for that portion of the code. I don't think we need to necessarily separate the repo just to have maintainers responsible for their drivers.

[jeffmendoza]

@sthulb Sorry, not obvious. It would be a tough decision to delineate between "core" drivers that reside under the Docker Open Source Project (DOSP, is there a term for this?) and others. I would hope that any driver with an active maintainer would be welcome.

@ehazlett Yes, discovery and install would need to be built into machine. Similar to the way vagrant does it. Vagrant has the benefit of using ruby gems under the hood.

[ehazlett]

No as far as I'm concerned we would not delineate between drivers.

On Monday, February 9, 2015, Jeff Mendoza notifications@github.com wrote:

@sthulb https://github.com/sthulb Sorry, not obvious. It would be a tough decision to delineate between "core" drivers that reside under the Docker Open Source Project (DOSP, is there a term for this?) and others. I would hope that any driver with an active maintainer would be welcome.

@ehazlett https://github.com/ehazlett Yes, discovery and install would need to be built into machine. Similar to the way vagrant does it. Vagrant has the benefit of using ruby gems under the hood.

— Reply to this email directly or view it on GitHub https://github.com/docker/machine/issues/500#issuecomment-73591435.

[sthulb]

I'd say we could utilise GitHub's releases API, which would allow something like driver install foo/bar, which would even allow us to use private repos to add drivers.

For the core drivers, the local ones & the most popular cloud providers (GCE, EC2, Azure — others defined later). Leaving the esoteric ones to be community managed.

The problem with keeping all the drivers as a part of the main repo is what happens when the maintainers drop out and we can't fix issues, something that could potentially delay a release.

[ehazlett]

If a driver isn't maintained then it's pulled from being supported. I don't think it's a good idea to differentiate between those "core" or not. We should simply have a spec and maintainers. If a driver is maintained and adheres to that then it should be in IMO.

On Monday, February 9, 2015, Simon Thulbourn notifications@github.com wrote:

I'd say we could utilise GitHub's releases API, which would allow something like driver install foo/bar, which would even allow us to use private repos to add drivers.

For the core drivers, the local ones & the most popular cloud providers (GCE, EC2, Azure — others defined later). Leaving the esoteric ones to be community managed.

The problem with keeping all the drivers as a part of the main repo is what happens when the maintainers drop out and we can't fix issues, something that could potentially delay a release.

— Reply to this email directly or view it on GitHub https://github.com/docker/machine/issues/500#issuecomment-73608474.

[thaJeztah]

which would allow something like driver install foo/bar

Or even automatically;

$ docker-machine create -d virtualbox dev

#custom driver;
$ docker-machine create -d github.com/my-org/my-repo dev

Which would give a more "batteries included" experience. Manually installing/updating/deleting should also be possible though.

[sthulb]

@ehazlett If we keep storing drivers in this repo, we should still support my model for enterprises that want to have their own providers. Even if they're stored in the same repo, we should perhaps allow users to select which drivers they want.

@thaJeztah I like that syntax :)

[ehazlett]

Another reason I'm not a fan of using external binaries is they would be useless on their own as they would need key creation, etc from machine. It seems odd to separate out just to have the code separated.

I agree we should have common operations in machine and a versioned API for the drivers to adhere.

[nathanleclaire]

I'm not sure about implementation (e.g. having separate binaries, etc.), but I'm convinced we need a way to develop the core of machine relatively independently from drivers. Right now is a critical time for Docker Machine's development and the project needs to be able to move quickly, experiment, iterate, and possibly throw some things out completely later if they don't end up working out.

The current driver structure makes such innovation nigh impossible. It's pretty clear that docker-machine will need to be modified dramatically from its current form, so it should probably be addressed sooner rather than later.

It seems to me like one of machine's goals is to unify and standardize the 'getting started with docker' experience between docker-host options. Example: trying to standardize the distro, docker daemon setup, even the hostname pattern.

I agree with this, but I think having a well-defined specification (and several strong core drivers) will help with most of where this would be an issue. At any rate, I don't see any reason why the core Machine API couldn't take care of provisioning if we're just doing it all over SSH or cloudinit anyway.

It seems more comfortable to me to have "recommended driver plugins" that we would revoke our recommendation for if they fall out of grace rather than having to maintain the code ourselves. 10-20 drivers directly in machine's core is a huge crushing maintenance burden, and doesn't really make sense if 80% of users' use cases would be covered by 5 or 6 of them.

Right now we have a pretty low bar for contributing a driver (which is great!) but a ridiculously high bar to contributing things which affect the core (which is awful). Any individual contributor has pretty much no way to test if their changes have affected other drivers, and even as someone who has contributed a decent bit to machine I haven't ever touched at least half of the drivers.

If a driver isn't maintained then it's pulled from being supported. I don't think it's a good idea to differentiate between those "core" or not.

I'm actually very in favor of having several "core drivers" that we maintain as examples for the community to follow. Right now, I have a feeling that they vary wildly in quality anyway and we're biased towards maintaining the ones we know well. It's a little disingenuous. If we're worried about being accused of playing favorites, we could put it up to community vote and pick the top 5 to include in the core.

I'm not a fan of letting the code in, then kicking it out later if the maintainers can't keep up. I feel like it will only result in hurt feelings and disengaged community members (A merge, to me, implies a commitment to keep the code in for the foreseeable future).

Another reason I'm not a fan of using external binaries is they would be useless on their own as they would need key creation, etc from machine. It seems odd to separate out just to have the code separated.

Long term, this might be an advantage - if it's universal, let's put it in the core API. People should ideally be able to build machine drivers in languages other than Go, too, but that's less of a concern for me in the immediate future.

And now for the crazy idea of the day: We might actually have a huge opportunity here to build something like Powerstrip right in (at least as a development tool). If we provide people with the tools to easily build extensions and have a nice interface for it, I think they will be more motivated to contribute, even if the underlying interfaces and APIs are rapidly shifting.

Anyway, hope this doesn't come across as negative - just want to contribute to the conversation.

[ehazlett]

I personally think picking "core" drivers will lead to us looking like we play favorites whether we intend to or not.

I agree everything is in disarray at the moment as far as testing, core contribution etc.-- we've beaten that horse to death. We all agree. No one is arguing we don't need to refactor -- that's a given.

I think spreading drivers out in separate binaries is a bad idea but if that's the road we want to travel collectively we can. I think putting standalone binaries would make an even worse maintenance -- what do we do for updates? API changes? Do we expect third parties to keep up? How do we make sure the versions are correct? User side? I think that user experience would suffer greatly. If we want to put certain drivers on the outskirts because most people don't use them I will simply go on the record stating it's a bad idea in my opinion and we can push forward.

I'm not a fan of letting the code in, then kicking it out later if the maintainers can't keep up. I feel like it will only result in hurt feelings and disengaged community members (A merge, to me, implies a commitment to keep the code in for the foreseeable future).

I don't see how disabling a driver because of compatibility fails is any more hurtful then saying their driver isn't used enough to be "core".

Looks like we are at a standstill - I'm fine with doing whatever the consensus is. @bfirsh any thoughts here?

[thaJeztah]

Great words, @nathanleclaire making the driver more pluggable sounds like better maintainable, and might reduce the workload of the (core)maintainers, not needing to actively support users when working with drivers. Also, being able to de-couple release cycles of drivers from machine itself, makes it a lot more flexible.

WRT "official" / "non-official" and API compatibility; I think many systems have coped with similar issues. I'm thinking of some kind of a "market place" for drivers, similar to Docker Hub or, probably more close to the situation here, the way WordPress plugins work (I hate WordPress, but the user experience for the plugin system is actually very good);

Individual developers have to keep up with changes in WordPress and mention up to which version the plugin was tested. Badly supported/maintained plugins just get worse reviews/ratings so users can steer away from them.

This also means it would be possible to have multiple drivers for the same provider, and people can decide which one to use, based on quality or features.

And finally, if plugins can be downloaded/installed automatically, there would be no "official" drivers. Yes, you could decide to develop some drivers yourself (as example and to provide the "batteries included"), but still leave room for users to pick an alternative driver.

[ehazlett]

Thanks for replying. I wasn't saying not make it pluggable -- simply trying to point out my concerns on the potential downsides.

I'm a bit scared of creating a "driver marketplace" as well. Pulling it out of the core is fine, but that still needs to be maintained. I'm also worried about a distributed plugin model. I'm all for competition but having 10 different EC2 drivers that each do separate things does not help the user IMO.

So it looks like pulling and spreading these things out is the consensus.

Before moving on to actual design of what this should look like I would like to get @bfirsh's opinion on the following:

• should these be external binaries that are built and pulled by the user (either at runtime or via a separate command)?
• should we have a "core" set of drivers that are recommended? if so, which ones?

The remaining I think we simply have to put into sprint planning and prioritize -- the above is quite a bit of work (which I agree 1000% is needed).

Thanks all.

On Tue, Feb 10, 2015 at 3:59 PM, Sebastiaan van Stijn < notifications@github.com> wrote:

Great words, @nathanleclaire https://github.com/nathanleclaire making the driver more pluggable sounds like better maintainable, and might reduce the workload of the (core)maintainers, not needing to actively support users when working with drivers. Also, being able to de-couple release cycles of drivers from machine itself, makes it a lot more flexible.

WRT "official" / "non-official" and API compatibility; I think many systems have coped with similar issues. I'm thinking of some kind of a "market place" for drivers, similar to Docker Hub or, probably more close to the situation here, the way WordPress plugins work (I hate WordPress, but the user experience for the plugin system is actually very good);

Individual developers have to keep up with changes in WordPress and mention up to which version the plugin was tested. Badly supported/maintained plugins just get worse reviews/ratings to users can steer away from them.

This also means it would be possible to have multiple drivers for the same provider, and people can decide which one to use, based on quality or features.

And finally, if plugins can be downloaded/installed automatically, there would be no "official" drivers. Yes, you could decide to develop some drivers yourself (as example and to provide the "batteries included"), but still leave room for users to pick an alternative driver.

— Reply to this email directly or view it on GitHub https://github.com/docker/machine/issues/500#issuecomment-73782367.

[thaJeztah]

Well (as usual, LOL), I am mainly brainstorming. There could be a lot of (technical) limitations that make it impossible to do and, in the end, you (the maintainers) have the final say, because it's easy for me to say things.

I'm a bit scared of creating a "driver marketplace" as well.

For the time being, this could be GitHub (implemented using my earlier suggestion). Similar to the official images on Docker Hub, a separate repo could be created to collect "official" drivers that people can contribute to. Also similar to the official docker images, that repository can be maintained by a separate maintainer if the workload becomes too much for the machine maintainers themselves.

Drivers from the official repository won't need a github.com/username prefix; this also makes it easier for users to distinguish official drivers from unofficial / 3rd-party drivers.

Docker (Inc.) might be interested to take this further and extend Docker Hub with a "Machine Driver" section. Could be nice, but I don't think that's something for the maintainers here to worry about.

having 10 different EC2 drivers that each do separate things does not help the user IMO

True. The same applies to Docker Hub, there are currently 2065 NGINX repositories on there. If Cloud providers think Docker is important to them and want the best experience for their customers, they should invest time and contribute their own driver or assist in improving the current ones. I assume Docker (Inc) has relations to most (Cloud) Providers, so sending out a mailing that you're planning on making an official driver-repo and "now" is the time to contribute could be a start :)

[sthulb]

My purpose wasn't to try and enforce my opinion on the subject, this was my 20th idea on the subject and seemed like the best approach to me. But above all, the purpose was to make us think about what we want to commit to.

The binaries for drivers made sense to me because we're entering a space where there's potential to have 100s of providers, which could mean 100s of drivers. Shipping a machine binary that has all of that "bloat" when you're using just using 3-4 drivers.

Also as I stated earlier, we have the ability to create closed source drivers with this model in any language.

With my approach, we could end up with 20 of X driver, but that's good in a weird way. Just like there's variations of containers, the good ones will surface organically. We could offer a list in the readme/docs/wiki/* it wouldn't mean we prefer one over another, just that it exists.

[bfirsh]

+1 We should have a core set of drivers. It doesn't matter how this is implemented, but Machine should work out of the box with a bunch of popular hypervisors/providers.

These should only be included if:

1) They can be maintained. So either the maintainers of the Machine project can commit to maintaining them (e.g. virtualbox) or we have external maintainers who can commit to maintaining them (e.g. what Microsoft or VMware are currently doing) 2) We can test them over time for regressions. This means we need comprehensive integration tests and a CI system which can run the tests against a real provider.

I think it would be useful to have plugins of some kind. I don't think it's high priority compared to some of the other stuff we want to do. We should wait until lots of people are asking for it until we implement it.

In terms of whether we should have plugins with separate binaries... I don't know. There have been many discussions around how plugins should work in Docker, and I would like for Machine to get involved in discussions / follow suit there. I don't think we need to invent anything new. E.g. https://clusterhq.com/blog/docker-extensions/ https://github.com/progrium/docker-plugins

[sthulb]

I'm not sure if we should wait. There are a number of PRs and issues open that should be resolved first, but I think drivers as plugins are definitely something we should tackle soon, given there are drivers waiting to be merged now, which probably wouldn't be labelled as core given limited user bases.

[ehazlett]

thx @bfirsh. i think what we just need to do now is define the "core" set and what happens if they fall out of being maintained.

I also agree that we shouldn't build a plugin system. This is the reason we waited and pushed off the separate binary or plugin system -- we were trying to see where Docker went so we didn't have yet another system in the mix. I think the consensus is we need it so we should jump in and see if those systems would work for machine.

[ehazlett]

This should be part of the #553 refactor. We also should include a guide / policy on what driver maintainers need to do to create / maintain their drivers.

[nathanleclaire]

Let's move discussion on this one to https://github.com/docker/machine/issues/1626