Discussion: Expressing required and optional external dependencies & exposed interfaces

[blakelapierre]

I haven't seen this in the spec, however I believe this is very likely the level where it belongs. Currently, I'm attempting to bolt this metadata on to existing containerized applications in a high-level system description due to limitations in existing tools. I know that's the wrong place, but it's the best solution at the moment.

An App Container provides an implicit interface to the external world through it's exposed ports and the protocols used over those ports. Additionally, an App Container may wish to consume services from another App Container running somewhere else on the network.

These interfaces should be made explicit to enable tools to 1) verify that a system is complete and 2) automatically link App Containers together.

I believe it will be most useful to provide an indirection layer with a simple versioned-interface, as opposed to expressing dependencies to a particular App Container version. An App Container can then publish that it meets, requires, or can optionally use a particular interface version which can then be resolved to any App Container that meets that interface version.

---

Example:

I run cadvisor on every machine. cadvisor may optionally connect to an influxdb backend for metric aggregation. I should not have to manage cadvisor and influxdb IP addresses or ports. I should be able to drop a bunch of cadvisor instances into the same logical grouping as an influxdb and get wired up automatically (how this wiring up happens is not described here). Additionally, if I am running a bunch of cadvisor's without being logically grouped with an influxdb (or other container that implements the appropriate interface), a tool can trigger a warning or suggest that I include an influxdb.

---

Example:

I run an API server App Container that must connect to an SQL database App Container. I should not be able to launch a system where I have a logical grouping with API server App Container but with no appropriate SQL database. Additionally, I should be able to plug in and use any SQL database App Container that meets my API server App Container's requirements.

---

Draft Representation (example only):

{
  "provides": {
    "cadvisor-api": {
      "version": "0.0.1",
      "protocols": {
        "http": "${hostPort}",
        "https": "${hostPort}"
      }
    }
  },
  "requires": {},
  "optional": {
    "influxdb": {
      "version": "^0.0.1",
      "environment": {
        "use_influxdb": true,
        "storage_driver": "influxdb",
        "storage_driver_host": "${host}:${port}",
        "storage_driver_user": "cadvisor",
        "storage_driver_password": "password",
        "storage_driver_secure": true
      }
    }
  }
}

There are definitely flaws in this representation. I'm still not sure on exactly how generic the representation needs to be or which things should be parameterized or how. I don't have a working implementation of something like this in any of my systems yet. I prototyped some parts out a few months ago, but I'm only just now getting around to needing to fully implement them.

The primary idea is that containers should have as much configuration information baked into them as possible (although not necessarily hardcoded values, eg. port numbers) in a format that makes it possible for higher level tools to automatically introspect the configuration metadata and then pass in the appropriate values to make the container function properly in the context of a much larger system.

The provides section may be mostly redundant. I think there may already be enough information in the rest of the schema to be able to derive the interface automatically (except, I don't think there is a way to specify that it implements a more generic interface [which may be implemented by other App Containers]). However, I don't believe the requires or optional interfaces can currently be derived. The closest is the dependencies structure, but that appears to be reserved for pieces that need to be pulled into the running container, not an external (as in across a network port) dependency.

While I think this is the right place to put this information, I could be wrong. If this ability already exists somewhere else, or if it should go somewhere else please let me know. I'll try to update this issue as I think more about the idea, but would appreciate any feedback regarding the the possibility of bringing something like this into the spec and/or if it is something that would be useful and desired.

[blakelapierre]

Can you explain what you mean by 'service discovery'? I am specifically not talking about routing.

An App Container defines a (currently implicit and mostly hidden, black-box) interface. I propose that this interface be made explicit and exposed in a structured format. I'm sure you could shoehorn the interface description into annotations, but I'm not sure that's the best approach.

The interface that an App Container exposes is inextricably linked to the binary data contained in the images, unless your image dynamically pulls in and executes code at runtime. However, I'm not sure if that is considered best practice (it might be, I honestly don't know). It seems to me that the interface description needs to be bundled with the binary images.

[blakelapierre]

Here is a more visual approach to what I think needs to be represented. I made this several months ago, so I'm not 100% that this is still the cadvisor interface, but should still be an effective diagram.

                          ----------------------------
                          |         cadvisor         |
  (stdout) Outgoing Port--|                          |--Incoming Port (HTTP/80)
  (stderr) Outgoing Port--|                          |--Incoming Port (HTTPS/443)
                          |                          |
(influxdb) Outgoing Port--|                          |
                          ----------------------------

cadvisor: {
  in: {
    80: 'http',
    443: 'https'
  },
  out: {
    optional: ['influxdb']
  }
}

An App Container is a composable building block of a larger distributed system. In order to successfully compose App Containers, each individual Container's complete interface must be successfully inspectable. Currently, the App Container spec seems to expose information about the Incoming Ports through the Pod Manifest's ports property.

I accept that the current port information captured in the Spec may only want to deal with the realities of running an App Container on the host system. The above information does need to be captured somewhere, even if it is some additional layer the wraps around the App Container. However, my first thoughts are that there will be duplicated information (which should probably be avoided) and that it should probably really be tied directly to the binary image that is the App Container (as that is what provides the real implementation of the interface).

I'm still not sure on where the configuration should happen. For example, given we connect to an influxdb, what needs to happen to actually connect to an actual influxdb (that is, what information is exposed to the environment inside the App Container; the code running inside the App Container should likely still be responsible to using that information to establish the connection). I tried to capture some of that in the OP.

Is there work in another project that is attempting to capture this aspect of App Containers and their composition into larger systems?

[philips]

@blakelapierre I am still a bit confused on what sort of metadata you would like to see added to the specification. There are two things that I can see from your proposal:

• Configuration of an application based on the other things that are running in the environment. For example if this application knows it will have an influxdb.com container available to it it needs the environment flags CADVISOR_USE_INFLUXDB, CADVISOR_INFLUXDB_USERNAME, etc.
• Definition of the services that this container will reach out to and what are the endpoints that this thing will expose. The current schema is insufficient because you can't say things such as "this exposes MySQL version 5.0 client port".

I am intrigued about the first. But, I don't know if it saves some practical examples that save the user typing or trouble because they will need to set all of those things like username, password, and hostname anyways.

On the second I think this is an OK use case for using the current annotations. It will be extremely difficult to create a schema that will be able to accurately represent an endless amount of variety. For example I have a container that provides a postgresql client ABI compatible endpoint. But! It is actually being emulated by rethinkdb. Now do I call it rethinkdb-postgresql or postgresql? It is read-only and certain queries won't work so the application will need to know that, etc.

[blakelapierre]

Sorry for any confusion. This is something I'm still thinking about and I have yet to work out all of the details. I don't mean for this to be formal proposal, but more a discussion about some issues I've been anticipating (and have started to run into) that could lead to a proposal.

I think you are right that there are two distinct things going on: configuration and interface definition. Both need a solution and I'm not entirely sure if they can both be solved at this level, but I do think it may be possible and I think a solution could simplify some of the higher layers.

I'll give an example of the representations I'm currently working with, which will hopefully lead to the motivations behind what I'm trying to get at.

Given a system description:

{
  root: 'https://your_root/'
  configuration: {
    api: 1,
    postgresql: 1
    influxdb: 1
  },
  roles: {
    $all: ['cadvisor']
  },
  containers: {
    cadvisor: {
      container: 'google/cadvisor'
    },
    postgresql: {
      container: 'vendor/postgresql'
    },
    influxdb: {
      container: 'vendor/influxdb'
    }
  }
}

This is trivially expanded out to a 'fully-qualified' description. (See footnote.)

What I really want is to be able to take this system description and automatically produce a running system, all appropriately wired up in the way you would expect (ie. the cadvisors find their local influxdb & the api finds its local postgresql). However, to do this currently, you need additional metadata about what the various containers depend on.

There are two concerns at this point:

1. How can I validate that the above system description is complete (ie. can be successfully deployed)? [Interface Definition] That is, I (the system designer) know that api needs to connect to a postgresql, but there is nothing in the current Docker container or App Container formats that allows me to know that api depends on having a postgresql (or any other application/service). This makes automatic system composition impossible without having some additional, third-party, repository of metadata.

   The solution to this problem may be as simple as attaching names to a container:

   cadvisor: {
    provides: ['cadvisor-api'],
    optional: ['influxdb'],
   }
   
   influxdb: {
    provides: ['influxdb']
   }
   
   api: {
    requires: ['postgresql']
   }
   
   postgresql: {
    provides: ['postgresql']
   }

   The names can be URI's with a versioning scheme (most of which can be stripped out for the simplest case of: 'latest version').

   Additionally, to support the case you described, of, "it's mostly the postgresql interface, but really rethinkdb", the interface name should be abstracted away from pointing to a specific Container (although that could be allowed):

   my_rethinkdb: {
    provides: ['postgresqlABI']
   }
   
   my_container: {
    requires: ['postgresqlABI']
   }
   
   postgresql: {
    provides: ['postgresqlABI']
   }
   

   When dependency resolution occurs, either my_rethinkdb or posgresql can satisfy the requirement for my_container even though there is no container postgresqlABI.

   The primary value here is just recording the fact that one container depends on the existence of another container and exposing that fact in a way that can be automatically introspected (which is why I am bringing it to this forum).

   If the interface names are URIs then there may be additional metadata (configuration info?) available if you retrieve whatever is at the corresponding URL. I think this approach can give the application and tooling layers enormous power and flexibility with very little overhead cost in the Spec.

2. How do I wire up the containers at run-time? [Configuration] I don't know the best answer to this yet. I suppose that's what I was attempting to explore in the representation from the first post. Maybe setting environment variables is enough (although that approach alone might fail as the external environment changes). I'll have to think more about it, but encourage others to contribute any thoughts they have.

   I was talking to Redbeard several months ago and he was telling me about one of @cnelson's solutions to part of this problem, where (I believe) a container is presented a fixed IP address & port for a service and then another piece is responsible for manipulating iptables on the host to maintain a mapping to the actual service. I think this is a pretty nice solution and had considered implementing it myself (@cnelson, if you want to open up the code, I, and likely others, would be very grateful).

I'm sure you're aware that these representations are implicitly graphs. I plan to soon start work on a tool to visualize the graphs, which will possibly make it more clear what pieces may still be needed to realize the automatic composition of systems that I'm envisioning and (perhaps poorly) trying to describe.

I'm trying to get down to the very core representations necessary while trying to avoid (but still support) all of the messy details that can cloud what is really going on. I apologize if some of my representations seem to be missing something. That's because they are. I think that most things should be implied, but discoverable and overridable (which you can kind of see in the footnote below).

As I continue towards a complete, workable example, I hope some of these ideas will become more clear and succinct.

---

'Fully-qualified' expansion

{
  configuration: {
    api: 1,
    postgresql: 1
    influxdb: 1
  },
  roles: {
    api: ['cadvisor', 'api'],
    postgresql: ['cadvisor', 'postgresql'],
    influxdb: ['cadvisor', 'influxdb']
  },
  containers: {
    api: {
      container: 'https://your_root/project/api'
    },
    cadvisor: {
      container: 'https://google/cadvisor'
    },
    postgresql: {
      container: 'https://vendor1/postgresql'
    },
    influxdb: {
      container: 'https://vendor2/influxdb'
    }
  }
}

Notes:

1. These representations allow for many different variations (for example, nested roles, and nested configuration values), but I'm leaving them out for simplicity.
2. A role is generally mapped to a machine (virtual or physical).
3. There is a naming resolution scheme which can try various public repositories (Docker Hub, Quay), but here I just go to self-hosted URLs.