Comments on CLaaS architecture

[ddeboer]

@proycon Thanks for fec78b62668caa24c97aadd0b25832ef686afece. Some things that come to mind when looking at your architecture overview:

Deployed docker containers; or container registries storing the container images. (blue boxes)

In this stage, the containers haven’t yet been deployed. Do you mean built Docker containers instead? Let’s make this a bit clearer: software components (light green boxes) must push their build output as one or more Docker containers into a container registry. Merely adding a Dockerfile to software component source code is not sufficient, because CLaaS will not build the containers.

An application deployment configuration (Infrastructure as Code)

This may be stored in the same repository that holds the software component itself.

Data Store (yellow cilinder)

Do you mean a data store for managing the infrastructure or a data store that is used by the applications, such as a database?

App Interfaces

They now only run containers. Is that sufficient for all use cases, particularly HPC? Using a proper infrastructure orchestration tool, CLaaS could both deploy containers (preferred) and provision VMs (if necessary).

[proycon]

Thanks for the feedback!

In this stage, the containers haven’t yet been deployed. Do you mean built Docker containers instead? Let’s make this a bit clearer: software components (light green boxes) must push their build output as one or more Docker containers into a container registry. Merely adding a Dockerfile to software component source code is not sufficient, because CLaaS will not build the containers.

Yes, this refers to built containers indeed, as obtained from the registries. The actual Dockerfile is provided in the green layer in the bottom.

This may be stored in the same repository that holds the software component itself.

Agreed

Do you mean a data store for managing the infrastructure or a data store that is used by the applications, such as a database?

The latter, data stores such as databases, or whatever mounted data volume the containers have at their disposition.

They now only run containers. Is that sufficient for all use cases, particularly HPC? Using a proper infrastructure orchestration tool, CLaaS could both deploy containers (preferred) and provision VMs (if necessary).

I can't really judge that well. For certain High Performance Cluster use cases I guess you'd want to stick to the lower levels of running the actual software in some distributed fashion over the cluster, without unnecessary service overhead, but even then containers will provide a good solution (although there you'd rather want Singularity rather than Docker I'd say). If we can try to limit ourselves to at least the simplest common solution (containerisation using docker), then that might make things easier to adopt for developers then if we try to target too much (like what I did in LaMachine).

[proycon]

Some further comments that where raised during the meeting yesterday:

• Who is the audience for this schema?: I'd say that's primarily us rather than any end-users/scholars/researchers.
• The aim of the schema is to derive some harder high-level requirements about how we shape our infrastructure, e.g. by demanding developers provide a container and by setting up a CLARIAH-wide container registry where everybody should push their images. Moreover, some form of software metadata close to the source would be a requirement. I think we had a fair amount of consensus on this. The current schema focusses on software distribution/deployment, additional schemas could be made for other aspects:
• Such a more elaborate layered schema was suggested by @mmisworking , where the main schema would be simplified but users can dive into deeper schemas when needed, addressing different audiences.
  • I think everybody's welcome to attempt to improve this schema and/or make additional ones for other aspects, we just shouldn't forget that it's just a simple means to an end, getting some internal clarity, rather than a goal in itself.
• Working out some application interface details was suggested by @menzowindhouwer, things like authentication and authorization infrastructure would fit in there and are currently not represented in this schema at all.
• The role of data is only minimally represented in this schema. Additional thought may need to go into developing other schemas that focus more on the data-aspect rather than the software aspect.
• There was some confusion about the top layer of entry points (Ineo, mediasuite, etc), mirroring the reservations I had myself, because those too can be are services in their own right. The top layer merely represents the presentational aspect of being some kind of portal users access to find stuff.

(please add anything I forgot to mention or might have misunderstood)

[proycon]

I'm closing this now since discussion on the particular schema is a phase in the past that is less relevant now