Continous Deployment

[YannickB]

Hello everyone,

I'd like to raise an important issue regarding the next step of Clouder. Since last week we were working on integrating Gitlab in order to manage the versionning of the projects.

Fact is, I completely underestimated Gitlab. This is far more than a versionning and issue management tool now, with Gitlab CI it became a complete solution to manage the Continuous Integration and Continuous Deployment workflow, and I think it is a real opportunity to bring Clouder to the next and state of the art level.

Today we have finished the gitlab template and we already made a first work on the continous integration for the Odoo template. Of course we used the travis scripts of the OCA, there is still a lot of work to really have private continuous integration for Odoo and we will probably need to adapt some of the OCA scripts to make them work with Gitlab, but at the very least they are executed so we're already quite far in the continous integration feature.

More important is the continous deployment issue because it may fundamentally change the way Clouder work. I see three types of applications :

• The first are the applications without specific development and which are stable enough to be automatically updated. I'm thinking of the registry, backup, bind, shinken, postfix, postgres, mysql etc... Most of the subcomponents.
• The second are applications without specific developments but which contains some risks during update because it need database update. For example mautic, piwik or gitlab itself. We may want automatic update for minor version but keep manual update for major version to be sure technical team are ready to handle problem.
• The third are applications with specific developments, like Odoo, Wordpress, Drupal, and web applications in general. For them we need a real continuous deployment workflow, hence gitlab for versionning, continous integration for testing, and automatic deployment after successful test or manual validation.

Let's remind that in Clouder most of the applications, and the Odoo one in particular, use three containers :

• Data, where data related to each bases and configuration are stored, for easier backup. In Odoo case, configuration files, filestore, etc...
• Files, which contains the files of the applications and is automatically build with the last version when we build the images. If we want to update the application, we just change the Files container and update the bases.
• Exec, which contains the binary and is linked to Data and Files. It's the container which serve the data and is being exposed, it need to be regularly updated to have security patchs. This container can be duplicated for scalability, while still being connected to the same data. We can also have some other containers, like SSH to give access to the customers, but they are the three important ones. In the end, all applications (including subcomponent like bind/shinken/postgres etc...) will have a least Data and Exec in their templates.

Today Clouder need a lot of manual intervention. It's a great tool for initial deployment (and so it's perfect to integrate behind a website form) but keeping the containers updated is a complex workflow :

• You have to go in the concerned images and rebuild them
• Then you go on the container, change the image version, reinstall them
• Then trigger the update of the bases (for example the modules update in Odoo) We can easily add an update button in the container which will do all of this in one-click, and we will, but in the end it'll still require a manual intervention. I think this workflow is correct for some of the applications, the second type which need manual action for update. Note that we can make manual update for Files container and still have automatic update for Exec container.

Then we have the first type of application and Exec containers, which we can update without too much worries. For them, I think we should have a feature which will automatically update them each time there is a new version of the base image. Is there already a tool which manage this (I couldn't find one, with great surprise) or should it be a feature of Clouder?

And finally we have the third type, which need the real continuous deployment workflow. Of course here we will use Gitlab, I'll take the example of Odoo deployment to explain what I have in mind and I really need feedback here.

When we deploy a new Odoo instance in Clouder, it automatically create thanks to links a new project in Gitlab, with default .gitlab-ci.yml, .gitignore, etc... files. Clouder will also set in Gitlab secure variables the credentials needed for the deployment. The default .gitlab-ci.yml file will specify four stages :

• The build stage will, following the instruction in the the .gitlab-ci.yml file, download the specified version of Odoo/OCB, the OCA modules, and of courses the custom modules included in the repository. Then the build will be saved either as gitlab artefacts or build as image and pushed to the private docker registry Since we make already an extensive use of Docker and this build correspond exactly to the Files concept, I think it make sense to use the Docker approach (http://docs.gitlab.com/ce/ci/docker/using_docker_build.html) instead of artefacts.
• The test stage will take the build files and execute the lint/unit test on it. Of course we will not reinvent the wheel here, let's use the OCA scripts https://github.com/OCA/maintainer-quality-tools.
• The staging stage will, if tests are successful, automatically update the Files container of the staging instance if it exist, reset the bases to have the latest data from production, and update the bases. In case you don't know, you can create a staging environnement in Clouder by clicking on the Install Subservice in the container.
• The production stage will, after or not manual validation in Gitlab, deploy the new files in production.

Regarding deployment, we already said that in order to manage the first type of application and Exec containers we will need to develop in Clouder some kind of automatic update. Since our build would be pushed as docker images, we would just need to call this function to update our staging and production with the build image. So Gitlab will just need to call some kind of webhooks in Clouder to trigger the deployment and were done.

There is one last question, it's a question since the beginning of Clouder and maybe it's time to address it. Today Clouder execute each operation on your containers, updating configuration files, triggering an update, etc... by connecting directly to your containers through paramiko and, despite the use of the OCA connector which improve things a lot, Odoo is not a system tool and is not designed for this purpose.

It's the perfect place to store data and manage commercial/management workflow and that's why Clouder is based on it, but maybe we finally have the big picture now to put a real orchestrator (Ansible/Puppel/Chef/Salt) between Clouder and the containers. We could do it progressively by only managing at first the actions needed for the Gitlab deployment, and making Gitlab call the orchestrator instead of Clouder. For this, the orchestrator recipes will need to allow the same conditional and modular function than we have in the python function in the Clouder template, and will need to accept the variables from Clouder since the data are here. Do you think it's time to make the orchestrator move? And if yes, which one?

In the end, this discussion is about state of the art. From the beginning I receive many support from people which is the proof that Clouder answer a need, the need to have a tool which deploy a standard infrastructure out of the box without extensive knowledge or research. But I also receive many criticism, with good reasons, about why using Odoo for a system purpose, and having a monolythic approach instead of microservices which is the trend today especially with Docker. Odoo is the right tool for Clouder, I have absolutely no doubt about that now, and today I need your help and feedback to make sure we move in the right direction to build an awesome, secure and state of the art infrastructure around it. Gitlab opened new perspectives with big opportunities for that.

So, what are your thoughts?

[ecogit]

Thanks very much Yannick for this further integration!

[ecogit]

Either way, with or without including an orchestrator, naturally there is a high complexity of the Clouder ecosystem. Is there the option of including a GUI which visualizes the constellation and allows some basic management operations?

[YannickB]

Thank you @Agrachina, it depends what you mean Clouder is itself a GUI and designed to manage most of the operations from here.

[levkar]

Yannick, I'm glad you bring it to the point where dockerized continous deployment over a source control. I was working on it in june and decided this way to go (otherwise i would be using clouder anyway, you are doing great job). Unfortunately I won't be able to focus it for some time because of other projects.

I think you should consider kubernetes for orchestration. Kubernetes should be able to deploy from gitlab. What is left for clouder is to push proper configuration to git and rest will be handled by gitlab, kubernetes and dockerized registry. I would spare some time if you need a hand.

[YannickB]

Thank you @levkar.

I am doing extensive research this afternoon regarding orchestration and I think I am starting to have my position. I am differentiating standard orchestration tools (Ansible/Chef/Puppet/Salt) from Docker orchestration tools (Kubernetes, OpenShift, Rancher, etc...). Clouder itself is a Docker orchestrator, and even if we can use other Docker orchestrator as subcomponent it'll be complex to mix with Clouder concepts, too complex for now and without real need. Regarding the use of Docker orchestrators, my position is to stick only to tools edited by Docker Inc. (compose, swarm, etc..) so you can expect more support in Clouder for them in the future but I am really not sure for Kubernetes because I think more and more of the Kubernetes features will move in Docker itself .

But it's different for standard orchestrator, and Salt is really picking my interest now. It's standard, powerful, can replace all the ssh calls made by Clouder, and last but not least it's written in Python so we can import their function in the Clouder code itself, this is a huge advantage. I am really thinking about replacing the ssh call by Salt now, it should not be too difficult.

[dreispt]

@YannickB I'm a bit out of the water here to say something useful, but I can comment on the OCA MQT tools: while it's main use case was to run CI on TravisCI, it is desirable for MQT to evolve to a Python based, CI agnostic Odoo test tool. In this view there would be a TravisCI layer on top of the MQT core, and other CIs could be easily added. Gitlab is also popular amongst Odoo integrators.

[YannickB]

@dreispt Yes I think everyone aggree here, MQT will evolve to support all CI tools. Hope the rise of Clouder will help here.

I just commit the development for managing Odoo files building on Gitlab. Now the next step is the deployment, I think I'm gonna give Salt a go tomorrow.

[KaloyanNaumov]

Thanks a lot for the great work you are doing!

There is one important topic, which is about to become big in the following months, this is - EU General Data Privacy Regulation, and the Network and Information Security Directive (NIS), which are scheduled to take full effect by mid 2018. I'm yet to study the NISD, but with GDPR, things are already getting tight.

With this regulation, every system, that holds personal data, must be highly-secured in order to assure the customer data protection. Practically, in GDPR, there is no strict definition, as in the PCI or the ISO 27k standards, but the core security principles are becoming a must for every single merchant holding data - hardening, auditing, IDS, logging and so on.

Since the Clouder project will mostly be used by business, and they by nature store customer data, having a state of the art security standards becomes a must. Things like the CISecurity benchmarks and many of the PCI compliance points must be implemented out-of-the box. Dockers are great, as they really reduce the attack surface, but the hosts must also be in a top-secured state. Add to that, secured services are of most importance, so for each and every app/service template, if there is a hardening guide, it's best to be considered and applied.

In this regard having an orchestration system, which acts as a "configuration change management" is a must, as if done right, it clicks multiple compliance standard boxes, on top of the maintenance issues it solves.

A bit aside, but still Hope This Helps!

[YannickB]

Thank you, well the purpose of Clouder is to install in the infrastructure all the tools than you may forgot, and security tools are on top of the list. If you have some suggestions of open-source tools which should be integrated by default it will serve as reference for later.

[YannickB]

I think we can close this ticket now