Roadmap

[YannickB]

Hello everyone,

Ok so after Odoo Experience I have a bigger idea of what still need to be done in order to create an awesome service, here are the features planned in the next months, by order or priority :

• [x] Docker Swarm. We really need to have a clustering system and isolate component inside an overlay network for security. Also note that a partner want to use OpenShift instead, we're gonna make it so you can have different runner for Clouder, regular Docker Engine, Docker Swarm, Amazon ECS or OpenShift.
• [x] Third party providers : Clouder will always be able to manage internally DNS, Load balancing, Mails and other stuffs like that, but for critical infrastructures it is often best to rely on big companies like Amazon. We will integrate the libcloud library in Clouder so we can provision instances/containers/dns/other stuffs directly from Clouder. The link logic will be the same, but for example when you deploy a new base it'll not be linked to a DNS bind container but to a dummy container which on deploy will create a DNS record in Route53 or another provider.
• [ ] New instance creation features : We need to improve the frontend, with important features such as trial period, expiration, email verification, transactional email, manage invoicing based on the number of users in deployed base.
• [ ] Resource management : We should be able to specify in the container form a limit on the ressource (CPU/RAM/Disk) which can be assigned to this container. This resources shall be inherited from application.
• [ ] Improve subscription form : We need to be able to select a list of features (module to install, add more CPU etc...) defined in the application which can then be selected in the front. The price will be displayed and automatically updated according to the selected features. Also, the visitor shoud be able to specify or buy his domain from the front. And the design need to be greatly improved.
• [ ] Monitoring : Clouder currently use Shinken which is probably not the best tool today. We are considering the use of Elastic Stack for this purpose and for better log management in the Clouder infrastructures.
• [ ] Access right and CI : We need to make sure that access right are correctly configured so customer can manage their instance without having access to other customer instance. They will also have access to the gitlab repo of their instance. Salt log need to be improved (we want to see each result in clouder.job object) and the deployment log should also be visible from gitlab-ci (right now we just call a webhook)
• [x] Use Alpine as base image : We currently use the official ubuntu image as base, which weight around 200Mo. With all image and container to install, this means a complete oneclick deployment weight around 20-30Go which is very impressive for newcomers. I think it's really important to use a light base image like Alpine (recommanded by Docker), we probably can lessen the Clouder installation to around 5Go thanks to it, if not less. Remember that Dockerfile of Clouder images are here : https://github.com/clouder-images
• [ ] Download backup from Clouder interface : In order to give customer better control on their data, it'd be really good to have a download button on the save object, which will connect to the backup container and download the backup. Having an import feature would be also really interesting for migration from existing infrastructure, so we don't have to do the operation manually in the terminal. We'd just need to import the backup in the backup container and, provided the files are in the correct format, the Clouder restore feature will take care of the rest.
• [ ] Self installation of Clouder : Create a Docker Compose file which install a Clouder, capable of auto-backing himself. To give you an idea, on our main site we have an initialisation Clouder, which deploy and backup goclouder.net, which itself deploy and backup the others websites. We need to find a way for Clouder to be easier to deploy and autonomous.
• [ ] SSO/Portal : Centralize authentification in application hosted by Clouder in a central SSO platform
• [ ] Documentation : The current doc.goclouder.net is now vastly outdated (since summer 2015), it's really important to update it to easy the pain for newcomers who want to contribute.

[lasley]

While abstracting out the container management, it would be nice if we also considered Amazon ECS as a potential runner.

[YannickB]

+10 you're right for Amazon ECS

[lasley]

I haven't actually dug into the DNS controllers yet, but it would be nice to support Route53 instead of just bind. Check these out:

• infrastructure-dns/connector_dns
• preliminary Route53 connector based on connector_dns

I would be happy to make one for Bind9 - should be pretty simple.

[YannickB]

Hum.. +1, I'm in favor to use closed service for ensuring proper quality if the embedded open-source application is not enough.

According to me we shall deploy a dummy "Route53" container (which is of course not a real one) and make it so it can be selected in the place of the bind container in the base. I'll try to make a POC asap, without the code for route53.

Also, I think it's time to revive the master branch, most change which will be done starting from now will break existing infrastructure.

[lasley]

Sweet I look forward to the PoC.

Also agree on the need for an incubator branch, what if we consider it as a v10 incubator? Major version increments are the best time to make these sorts of changes IMO

[YannickB]

Yes it's going to be the incubator of 0.10.0 (remember we don't follow the Odoo version anymore.

[lasley]

Maybe the completion of the incubator is the right time for the OCA migration too? Starting with a fresh production in a fresh repo will make it even more clear that the products are incompatible.

[YannickB]

Yep, well we can migrate there anytime we want anyway. Any news on this side ?

[lasley]

We have incredible buy-in (3-4 delegates), so I'm certain it's happening.

We won't be able to move it in before we fix things like the #141 & some other OCA things like naming standards for columns and ReadMes, so it makes sense to migrate over after the incubator.

My team knows these OCA standards pretty well, so we can handle that for the most part. It's a lot of grunt work, so I'll have some juniors handle it for us most likely. I also have a set of migration regexes that help out quite a bit.

It seems a lot of devs have been working towards similar goals on their own, and all agree that what you have here wins.

Most recent message in the thread - https://odoo-community.org/groups/contributors-15/contributors-46578?mode=thread&date_begin=&date_end=

[YannickB]

Wow ok I wasn't aware that this thread was gotten so big o_o, will read it later.

I'm already working on #141, as I said I think it's not as big as it seems.

[KaloyanNaumov]

For DNS management, I would suggest you to consider the CloudFlare DNS services, as they are free and available over APIs. There is a Route53 implementation in the IT-projects-LLC's SaaS set of modules.

Also, it would be valuable to have an SSO/Identity Management, which could be Gluu. They are working on a Docker version, which can be adopted.

Security Standards Compliance and Hardening belongs in a different issue, IMO.

[lasley]

Hmmm so maybe an implementation using LibCloud, then a simple selection to determine the driver for the intended service

[YannickB]

Wow LibCloud looks like a golden mine to me, that's exactly what I hoped to find.

I think we are going to have more and more demand for using external service so here is my position :

• For each component (DNS, Load Balancer, monitoring, etc...) we should have one and only one open-source component inside Clouder templates (Bind, Postfix, Shinken etc...). This open-source component shall be chosen and elected among all as being the best available and will be deployed if you want all your infrastructure on your server (as do Clouder today).
• If you want to use external service, they should be proposed as alternative to the internal open-source component. I though we'd need to make one connector for each services, but I hoped that one project would centralize the connector and it seems that's exactly what LibCloud is doing so I am for using it. In general, external service will greatly expand the reliability of a Clouder infrastructure until all internal component become mature and secure, so I consider them as really important during the first steps.

Finally, to me SSO/Identity Management/Portal are really important and will be a core feature of Clouder. We don't have a template for them yet but know that's a component which I think about since a really long time, it will be the core of Clouder for the user, where they log into, where they will be redirected to the applications hosted in the Clouder infrastructure (not just odoo, but gitlab, owncloud, rainloop, mattermost and so on). I am aiming for the same user experience we have when we use google applications, nothing less and SSO/Portal are key for this. I am still looking for the right internal component for this part, it seems Gluu is open-source behind so it may be a choice to consider.

[lasley]

Regarding external services, I think the way to manage them is via the Connector libraries. We can be naive of the external service itself, simply providing a dummy model like you did.

That dummy model can then be inherited by a module to glue a connector into it. The best part is, it doesn't even need to be glued by service type because the connectors themselves are abstract.

For example - given the connector_dns repo, the glue module would create a polymorphic _inherits between the Clouder Domain with the Connector's DNS Record. Add a bit of logic to map the two records, then the DNS synchronization is handled via the external connector library instead. This allows us to maintain a service-agnostic approach to the whole situation - all we do is write to a dummy model.

[KaloyanNaumov]

The research I did on IDP/SSO solutions shows Gluu as the better open source SSO/IDP, compared with OpenAM and Shibboleth among others.

This seems to be good addition: Sentry There's an Odoo module too.

And was wondering what would be your choice for centralised logs collection solution: http://www.fluentd.org - https://github.com/fluent/fluentd/ https://www.elastic.co/products/logstash - https://github.com/elastic/logstash https://www.graylog.org - https://github.com/Graylog2/graylog2-server

[YannickB]

I believe we all agree, a service-agnostic approach was all I was looking for (same way we do for postfix template with external smtp providers), way to go.

@KaloyanNaumov All good names, we'll make sure to check them out when we'll work on theses features.

(On side note regarding the discussion on the OCA, I had to reply through the Odoo interface since I hadn't the mail in my mailbox, it lost the subject... grrr Odoo...)

[YannickB]

Oh, good news it seems that Salt Cloud use libcloud https://docs.saltstack.com/en/latest/topics/cloud/install/index.html, it seems we have some luck here. Since Clouder already use Stack we just have to go in that direction.

[lasley]

Damn I love it when things are convenient! Rare case right here so let us take the time to revel in it 😆 🎉 🌮

cc @t3ddftw

[lasley]

@YannickB - Might I suggest we make this an "Incubator Roadmap" (or similar) in Github projects instead? I'd like to continue conversations on some of these, but I think this ticket is going to get a bit hefty.

[YannickB]

I'm not used to Github projects, you're right it seems to be the way to go

[lasley]

They're new. Turns out they're useless though - I requested this in another repo, and it's actually not possible to create cards without write access to the repo. Pretty much nullifies that possibility for everything open source - sorry to waste your time @YannickB 😦

[YannickB]

Hum you're right... No worries.

[YannickB]

Just added a new item regarding the base image used by Clouder

[lasley]

I haven't messed around with Alpine much, will do that this week. Docker is moving their default image to that though, so it can't be all bad. IMO most anything is better than Ubuntu though.

I'm a CentOS fan personally, but that's mainly just because yum is amazing and SELinux > AppArmor. The former point is not really that important, and the latter is only relevant on the host system AFAIK.

[tedsalmon]

You can't run separate SELinux / AppArmor policies inside of Docker anyways; the host policies trickle down to the container by default 😉

[lasley]

FYI I'm in the process of ironing out a plan for the sales/billing/monitoring side of things. Will create an RFC once I have something meaningful.

[lasley]

Oh also another feature that would be worth looking into would be automatic spin-downs/ups for idle instances, similar to Heroku. Here's an implementation that explains it better http://spin-docker.readthedocs.io/en/latest/activity_monitoring.html#stopping-idle-containers-example-workflow

[YannickB]

Updated the roadmap with libcloud, download backup, monitoring and documentation items.

[YannickB]

Updated the roadmap with self installation item.

[pasgou]

in you roadmap you missed something that I had place at the first place if I were you:

• [ ] first - user documentation

the true is that if you develop every things but you do not write the doc in the same time, and it s not an option or the thing you do when you take time to, your app will be unusable.

And so, in my work to do a docker image which works to install Odoo 10 + Clouder master, I found an clouder which is so different than the doc that I can't use it.

We don't know were to begin, there is no pop up help, no doc, no explaination on how to to use which option to do anything.

I had understand how to work with clouder when it was in 8.0 version, but know, I do not understand anything.

Please, stop develop new things, make a doc for each things which exists know, in writing that, you will understand what is not really clear or understandable, take the place of the user. I really things that documentation has the same importance than the code and it's urgent to do something in this way. [edit] documentation is now written but in last position 👏🏻☹️

[pasgou]

About libcloud

Seems to make the jod needed to create a completly automatic platform Smell good 😊

[YannickB]

Hello @pasgou,

Well I couldn't agree more. Please note that the order of item are not the order or priority.

In fact at this point I should be finding a solution for the documentation, it's critical now. My own tasks were mainly swarm/alpine/libcloud and I took care of them in November. Problem is, because of that the standard installation are not working anymore.

Before I start focus my own work on the documentation, I need to :

• Make the docker-compose so beginner can have an instance right away
• Finish debug with the oneclick installation, so the standard infrastructure allow to at least deploy a working Odoo instance. The main struggle are the DNS, as discussed in #176 we need to replace bind by powerdns so we have a least the level of features we had during Odoo Experience. Alpine broke bind unfortunately, that's why we want to move on powerdns.

So to put it frankly : right now, the master/incubation branch is broken, until we fix it there is no point updating the documentation.

About my personal situation : Honestly, I could fix thoses two points in a week, but in December/January I met financial issues and since I have to concentrate on project non-relative to Clouder this is holding me back until probably the end of January. I am more confident for February though. That's why all I am doing right now is answering tickets, I just don't have the strength to throw myself into the code right now, too much projects in mind. Still I take the opportunity to find people here in Paris to help in the next month, I think we can create a strong community here and since it's difficult for me to code right now I am focussed on that.

That's all for the update. I heard your point about the documentation, it shall be done as soon as possible now, getting more people involved and ease the pain for newcomers is the priority now.

[pasgou]

I understand completly See your Gmail

Le ven. 6 janv. 2017 à 13:55, Yannick Buron notifications@github.com a écrit :

Hello @pasgou https://github.com/pasgou,

Well I couldn't agree more. Please note that the order of item are not the order or priority.

In fact at this point I should be finding a solution for the documentation, it's critical now. My own tasks were mainly swarm/alpine/libcloud and I took care of them in November. Problem is, because of that the standard installation are not working anymore.

Before I start focus my own work on the documentation, I need to :

• Make the docker-compose so beginner can have an instance right away

• Finish debug with the oneclick installation, so the standard infrastructure allow to at least deploy a working Odoo instance. The main struggle are the DNS, as discussed in #176 https://github.com/clouder-community/clouder/issues/176 we need to replace bind by powerdns so we have a least the level of features we had during Odoo Experience. Alpine broke bind unfortunately, that's why we want to move on powerdns.

So to put it frankly : right now, the master/incubation branch is broken, until we fix it there is no point updating the documentation.

About my personal situation : Honestly, I could fix thoses two points in a week, but in December/January I met financial issues and since I have to concentrate on project non-relative to Clouder this is holding me back until probably the end of January. I am more confident for February though.

That's why all I am doing right now is answering tickets, I just don't have the strength to throw myself into the code right now, too much projects in mind. Still I take the opportunity to find people here in Paris to help in the next month, I think we can create a strong community here and since it's difficult for me to code right now I am focussed on that.

That's all for the update. I heard your point about the documentation, it shall be done as soon as possible now, getting more people involved and ease the pain for newcomers is the priority now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/clouder-community/clouder/issues/139#issuecomment-270898462, or mute the thread https://github.com/notifications/unsubscribe-auth/AIue8fPm44nQqf2hOk0SXysx4GEYY1yjks5rPjnLgaJpZM4KW7O_ .

-- Cordialement

Pascal GOUHIER

COPAGEX Société d'expertise comptable Membre de la Communauté des Experts en Gestion 8 Boulevard Albert Einstein Parc de la Rivière 44300 NANTES

T : 02 40 58 56 49 F : 02 40 95 11 81

http://www.copagex.fr

[pasgou]

Concernant docker compose i made the Job with master and it is almost ready for branch 0.9.0

See my github

Le ven. 6 janv. 2017 à 23:06, Pascal GOUHIER pascal.gouhier@copagex.fr a écrit :

I understand completly See your Gmail

Le ven. 6 janv. 2017 à 13:55, Yannick Buron notifications@github.com a écrit :

Hello @pasgou https://github.com/pasgou,

Well I couldn't agree more. Please note that the order of item are not the order or priority.

In fact at this point I should be finding a solution for the documentation, it's critical now. My own tasks were mainly swarm/alpine/libcloud and I took care of them in November. Problem is, because of that the standard installation are not working anymore.

Before I start focus my own work on the documentation, I need to :

• Make the docker-compose so beginner can have an instance right away

• Finish debug with the oneclick installation, so the standard infrastructure allow to at least deploy a working Odoo instance. The main struggle are the DNS, as discussed in #176 https://github.com/clouder-community/clouder/issues/176 we need to replace bind by powerdns so we have a least the level of features we had during Odoo Experience. Alpine broke bind unfortunately, that's why we want to move on powerdns.

So to put it frankly : right now, the master/incubation branch is broken, until we fix it there is no point updating the documentation.

About my personal situation : Honestly, I could fix thoses two points in a week, but in December/January I met financial issues and since I have to concentrate on project non-relative to Clouder this is holding me back until probably the end of January. I am more confident for February though.

That's why all I am doing right now is answering tickets, I just don't have the strength to throw myself into the code right now, too much projects in mind. Still I take the opportunity to find people here in Paris to help in the next month, I think we can create a strong community here and since it's difficult for me to code right now I am focussed on that.

That's all for the update. I heard your point about the documentation, it shall be done as soon as possible now, getting more people involved and ease the pain for newcomers is the priority now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/clouder-community/clouder/issues/139#issuecomment-270898462, or mute the thread https://github.com/notifications/unsubscribe-auth/AIue8fPm44nQqf2hOk0SXysx4GEYY1yjks5rPjnLgaJpZM4KW7O_ .

-- Cordialement

Pascal GOUHIER

COPAGEX Société d'expertise comptable Membre de la Communauté des Experts en Gestion 8 Boulevard Albert Einstein Parc de la Rivière 44300 NANTES

T : 02 40 58 56 49 F : 02 40 95 11 81

http://www.copagex.fr

-- Cordialement

Pascal GOUHIER

COPAGEX Société d'expertise comptable Membre de la Communauté des Experts en Gestion 8 Boulevard Albert Einstein Parc de la Rivière 44300 NANTES

T : 02 40 58 56 49 F : 02 40 95 11 81

http://www.copagex.fr