search

fabric8io / fabric8

fabric8 is an open source microservices platform based on Docker, Kubernetes and Jenkins
http://fabric8.io/
1.76k stars 504 forks source link

environment builder #4641

Open rawlingsj opened 9 years ago

rawlingsj commented 9 years ago

Doing a bit of thinking...

Currently we 'Run' templates. I wonder if we could benefit from an environment builder like the 'project builder' that picked which apps should be run and wired together by fabric8.

I.e. if a developer / ops wanted to create a new env they could either import an existing yaml file (from a file or git repo) or build a new one. fabric8 could make suggestions using defaults if they wanted CD by preselecting say Jenkins and Gogs but an administrator could swap out Gogs for Gerrit. fabric8 would generate SSH keys and manage the secrets. Also request how many environments need to be created.

We could also allow users to add ip / dns entries for external services if they wanted to use Github for example, fabric8 would create the Kubernetes headless service (if this is the right way to access external services from within the cluster)

rawlingsj commented 9 years ago

Again not sure if it would be the right place but maybe it could also capture what OAuth provider should be used and also claim / setup any storage required by the environment?

jstrachan commented 9 years ago

Yeah - I've been wondering along similar lines. I guess the issue is do folks want to be able to trash and recreate an environment once it's created - or do we just create environments via the REST API / CLI / web app?

Am not totally sure how often that's the case - e.g. to maybe to switch physical kubernetes clusters? Folks might want to test the same logical environment configuration on different cloud providers or locations maybe?

Once an environment is created we could easily copy it to another environment by just copying the kubernetes resources (the specs at least for services/RCs etc) - but storing the required templates + the parameters in git might be handy?

I've wondered if we need a way to capture the templates in, say, git - along with the parameter values for the templates per environment?

e.g. something like

template1.json template1.dev.parameters template1.test.parameters template1.prod.parameters

then the parameter values to be used for template parameters could be stored per environment and its easy to grok how the differ. I guess that could be a security risk if its login/pwds; so those should go into some little secrets store or something?

Maybe those could come from something like Vault: https://vaultproject.io/ as part of the creation of the environment?

Also as you say, for persistent storage; those need to be bound to some external environment specific volumes usually; so that might be in the parameters (or some other environment specific configuration)?

The other option is to have a branch/repo per environment so that folks can merge changes and stuff.

Another thought was; rather than storing the actual template1.json inside git; if the template is generated by a build tool (e.g. fabric8 maven plugin) we'd store a link / reference (e.g. maven coordinates) so that changing from version 1.1 of template1 to 1.2 of template1 would just be a git change. Ditto folks could use git go change parameter values too?

On Saturday, August 22, 2015, James Rawlings notifications@github.com wrote:

Doing a bit of thinking...

Currently we 'Run' templates. I wonder if we could benefit from an environment builder like the 'project builder' that picked which apps should be run and wired together by fabric8.

I.e. if a developer / ops wanted to create a new env they could either import an existing yaml file (from a file or git repo) or build a new one. fabric8 could make suggestions using defaults if they wanted CD by preselecting say Jenkins and Gogs but an administrator could swap out Gogs for Gerrit. fabric8 would generate SSH keys and manage the secrets. Also request how many environments need to be created.

We could also allow users to add ip / dns entries for external services if they wanted to use Github for example, fabric8 would create the Kubernetes headless service (if this is the right way to access external services from within the cluster)

— Reply to this email directly or view it on GitHub https://github.com/fabric8io/fabric8/issues/4641.

rawlingsj commented 9 years ago

do folks want to be able to trash and recreate an environment once it's created - or do we just create environments via the REST API / CLI / web app? I think there’s a case for being able to recreate an environment and also duplicate environments for new teams / projects. For example a new microservice or PoC needs a new env set up and configured in a way that they’re familiar with or that the company governance stipulates.

Also having an environment in version control feeds into the view of infrastructure is code a bit. Using REST API / CLI / Web with fabric8 connectors to wire them up?

I've wondered if we need a way to capture the templates in, say, git - along with the parameter values for the templates per environment? yeah this is going to be needed sooner or later. some companies have teams or individuals that are responsible for config and only they are allowed to manage DB password fro example. Having all this config in git makes sense.

Maybe those could come from something like Vault: https://vaultproject.io/ as part of the creation of the environment?

Ooh yeah that looks really good! I think @jimmidyson looked at vault before too?

Another thought was; rather than storing the actual template1.json inside git; if the template is generated by a build tool (e.g. fabric8 maven plugin) we'd store a link / reference (e.g. maven coordinates) so that changing from version 1.1 of template1 to 1.2 of template1 would just be a git change. Ditto folks could use git go change parameter values too? Ah yeah thats interesting. Yeah that could quite nice.

On 22 Aug 2015, at 11:23, James Strachan notifications@github.com wrote:

Yeah - I've been wondering along similar lines. I guess the issue is do folks want to be able to trash and recreate an environment once it's created - or do we just create environments via the REST API / CLI / web app?

Am not totally sure how often that's the case - e.g. to maybe to switch physical kubernetes clusters? Folks might want to test the same logical environment configuration on different cloud providers or locations maybe?

Once an environment is created we could easily copy it to another environment by just copying the kubernetes resources (the specs at least for services/RCs etc) - but storing the required templates + the parameters in git might be handy?

I've wondered if we need a way to capture the templates in, say, git - along with the parameter values for the templates per environment?

e.g. something like

template1.json template1.dev.parameters template1.test.parameters template1.prod.parameters

then the parameter values to be used for template parameters could be stored per environment and its easy to grok how the differ. I guess that could be a security risk if its login/pwds; so those should go into some little secrets store or something?

Maybe those could come from something like Vault: https://vaultproject.io/ as part of the creation of the environment?

Also as you say, for persistent storage; those need to be bound to some external environment specific volumes usually; so that might be in the parameters (or some other environment specific configuration)?

The other option is to have a branch/repo per environment so that folks can merge changes and stuff.

Another thought was; rather than storing the actual template1.json inside git; if the template is generated by a build tool (e.g. fabric8 maven plugin) we'd store a link / reference (e.g. maven coordinates) so that changing from version 1.1 of template1 to 1.2 of template1 would just be a git change. Ditto folks could use git go change parameter values too?

On Saturday, August 22, 2015, James Rawlings notifications@github.com wrote:

Doing a bit of thinking...

Currently we 'Run' templates. I wonder if we could benefit from an environment builder like the 'project builder' that picked which apps should be run and wired together by fabric8.

I.e. if a developer / ops wanted to create a new env they could either import an existing yaml file (from a file or git repo) or build a new one. fabric8 could make suggestions using defaults if they wanted CD by preselecting say Jenkins and Gogs but an administrator could swap out Gogs for Gerrit. fabric8 would generate SSH keys and manage the secrets. Also request how many environments need to be created.

We could also allow users to add ip / dns entries for external services if they wanted to use Github for example, fabric8 would create the Kubernetes headless service (if this is the right way to access external services from within the cluster)

— Reply to this email directly or view it on GitHub https://github.com/fabric8io/fabric8/issues/4641.

— Reply to this email directly or view it on GitHub https://github.com/fabric8io/fabric8/issues/4641#issuecomment-133675518.

jstrachan commented 9 years ago

I wonder if for now a simple environment builder would just be maven and the fabric8 plugin along the same kinda lines as the installer: https://github.com/fabric8io/fabric8-installer

where an environment has a pom.xml with dependencies to be deployed and their versions (using mvn coords so its easy to roll forward/back versions in environments). https://github.com/fabric8io/fabric8-installer/blob/master/cdelivery/pom.xml#L34

The installer is using a module per feature; in the 'environment builder' use case its one maven module per environment.

I guess you could argue that the templates to be installed should be the same in all environments; its just the versions that change; so rather than copy/pasting lots of template dependencies into different maven modules, maybe a single maven module + profiles could be used. Though that seems more complex.

So maybe releases need to try generate a single template JSON in maven (so there's not loads of dependencies in each environment's pom.xml) then each environment just defines the version to use - and potentially overrides any property values in the template parameters?

So an environment builder could look like...

dev/
  pom.xml
test/
  pom.xml
stage/
  pom.xml
prod/

with the pom.xml defining the versions used of the templates along with the template parameter values?

Then to change/rebuild an environment its just

cd stage
mvn fabric8:apply
jimmidyson commented 9 years ago

I'm not struck on the idea of using maven poms in this way tbh. Wonder if some other format would be better? That way we could reuse the files with gofabric8 at some point too to provide a cmd line interface for potentially non-Java users?

rawlingsj commented 9 years ago

There was a fair bit of love for fabric8 at the last Kubernetes meetup but one of the comments was asking if it can be less Java focussed when installing and setting it up.

All all the java development tooling is still absolutely valid but I too wonder if we could use a golang client and maybe a yaml file like the fabric8.yaml to point to versioned templates in maven central? This might be over simplifying things though?

jstrachan commented 9 years ago

yeah, that'd be cool too! I just figured on the maven option as it totally works right now. But yeah would love a gofabric8 alternative for polyglot folks too.

I guess we could come up with a non maven alternative; defining:

I wonder if we use something like an environment.yml which is something like

properties:
  fabric8.version: 2.2.1
  hawtio.version: 1.2.3
templates:
  jenkins: 
    url: http://central.maven.org/maven2/io/fabric8/jube/images/fabric8/jenkins/${fabric8.version}/jenkins-${fabric8.version}-kubernetes.json
    parameters:
       FOO: bar
       SOMETHING: cheese
  nexus: 
    url: http://central.maven.org/maven2/io/fabric8/jube/images/fabric8/nexus/${fabric8.version}/nexus-${fabric8.version}-kubernetes.json
    parameters:
       OOH: nice

then we figure out a way to import secrets in there too?

rawlingsj commented 9 years ago

ah yeah ^^ thats what I was thinking :)

jstrachan commented 9 years ago

in the above I used property resolving to find URLs so that we can do a git commit on this source to easier see when we change the hawtio.version to something - rather than changing of URLs which tends to be more opaque when looking at git/gertit changes