PROPOSAL: Improve Contributor Tooling

blaggacao commented 9 years ago

This evolves from and closes #145.

I claim, there is a benefit on incentive to converge contributor tooling, as odoo community, for the nature of the product, is characterized by many half-programmers. For the sake of efficient discussion, if you want to challenge this claim, please mark this specifically in your comment.

A generally adoptable tooling should be:

platform independent
easy to setup / automated
solve as many topics implicitly as possible
It comprises the following subsystems:
editor Subsystem / IDE
git subsystem
dev testing subsystem
Optimization candiates per Subsystem:

Editor / IDE:
snippets and templating
formatting
file layout
git subsystem:
feature workflow
correct origin/upstream setup
sorting/isolating of this messy (sorry for opnioning) multypurpose repositories, which I still not know how to do this...
dev testing subsystem:
platform independent
isolated (for the efimeral developer)
easy setup
autoreload on file change features
local or remote/proxy for individual and teams

blaggacao commented 9 years ago

For the dev testing subsystem, I'm working on a comprehensive docker tooling. I'm just waiting for the new volume driver https://github.com/docker/docker/pull/14242 to be released so to be able to handle file sync to the docker host elegantly AND completely platform independent.

dreispt commented 9 years ago

Most of this if subject to personal preference. It's nice to have wiki pages sharing best practices for each tool, for the people that use it.

Docker is a matter of preference. Editors/IDEs are a matter of preference. Git workflow is not, but it's not specific to OCA or Odoo either.

blaggacao commented 9 years ago

You're absolutely right, it is very important to follow the maxime "default but swapable". The long term benfit of this kind of tooling/best practice flourishes slowly with the ones that do not have a personal preference, yet. Preparing a wiki stub page, right now.

rvalyi commented 9 years ago

guys, about that, at Akretion we are using Voodoo a tool for that: https://github.com/akretion/voodoo feel free to contribute to the project.

We were in a kind of a big rush during 2015, so we didn't communicate about the project yet but we are using it since 6 months in many projects, so far so good. So if your are interested you can test and contribute. But please don't make public statements about it yet as we plan to adjust a few things and then make some big presentation of it.

Voodoo is opinionated and built around the Anybox buildout recipe.

Regards.

blaggacao commented 9 years ago

@rvalyi that's very interesting, I wasn't aware of this. I'll have a deeper look, for now:

Be aware, that docker-compose doesn't ship windows binaries yet. Are you open to a discussion of pro and cons about docker vs anybox? If yes, I would file some concerns here soon. At first sight, it also seems a heaviweight, probably what we would need is something called "voodoo light"? I want to avoid to block productivity by extensive or complex tooling...

blaggacao commented 9 years ago

Update: It's VERY interesting! :smile: What's your solution for syncing local files to the docker volumes? Mind the new release of docker volume api docker/docker#14242, this would solve the windows syncing case, however I'm not sure, what would happen to pynotify functionality, when syncing over NFS or SMB or similar..

rvalyi commented 9 years ago

so if for instance some project will create a bash script doing the OCA lint tests (like Travis), then this should be a standalone script and Voodoo will use it (it shouldn't be done inside Voodoo). Also because t extends Docker Compose, Voodoo is very easy to extend with plugins (like Eclipse dev mode plugin etc...).

@blaggacao IMHO the light mode is just to take a Voodoo repo and run it as a standard buildout project with virtualenv and without Docker. Now, once you want to avoid cluttering the dev machine with the python and native deps, it means virtualization and here I believe the default Voodoo Docker usage is the lightest thing you can imagine,

Now as I said voodoo could ship a more generic lint script. It can embedded the Acsone Eclipse debug buildout recipe... So we see that kind of modularization.

About windows: either Docker compose get mature for Windows (Microsoft is investing on that right and eventually we clean the Voodoo python launcher a bit), either we ship a Vagrant of the thing with a degraded Vagrand Launcher but that would still do 80 of what user need (the smart is inside the portable Dockerized Linux image anyway).

Voodoo is also a way to tell our customers: look; you can easly get a local replica of your Cloud instance: so no stress we are really doing open source...

rvalyi commented 9 years ago

@blaggacao @dreispt we also plan to fit Voodoo with OCA specific commands like: add an OCA repo X to my project, like a single command to fork and OCA repo or checkout a fresh pull request etc...

blaggacao commented 9 years ago

In spite of vagrant: docker-machine. It's maturing, also on windows. Linting: consider PyYapf, there is no better way :smile: Let's get rid of this topic once and for all... Just to make sure, as you didn't pick it up in your response: Did you get my point about docker volume? It is meant to sync the local source files of your IDE/editor to the docker in realtime. I would prefer a simple DOCKERFILE over any specific recipt, that in dubio, tends to be more complicated without adding extra benefit.

moylop260 commented 9 years ago

FYI we are working with a runbot based on docker https://github.com/Vauxoo/runbot-addons/tree/9.0/runbot_travis2docker module. The main base of this module is the package travis2docker https://github.com/vauxoo/travis2docker And the base of travis2docker is the .travis.yml that use maintainter-quality-tools script. Then I can replied easiest the environment of travis in local with travis2docker and we can re-use the script in runbot too.

Feel free of use and comment.

2015-10-16 11:25 GMT-05:00 David Arnold notifications@github.com:

In spite of vagrant: docker-machine. It's maturing, also on windows. Linting: consider PyYapf, there is no better way [image: :smile:] Let's get rid of this topic once and for all... Just to make sure, as you didn't pick it up in your response: Did you get my point about docker volume? It is meant to sync the local source files of your IDE/editor to the docker in realtime. I would prefer a simple DOCKERFILE over any specific recipt, that in dubio, tends to be more complicated without adding extra benefit.

— Reply to this email directly or view it on GitHub https://github.com/OCA/maintainer-tools/issues/146#issuecomment-148760154 .

Moisés López Calderón Vauxoo - OpenERP's Gold Partner Mobile: (+521) 477-752-22-30 Office: (+52) 477-773-33-46 web: http://www.vauxoo.com twitter: @vauxoo @moylop260 hangout: moylop260@vauxoo.com

blaggacao commented 9 years ago

@moylop260 :+1: Thanks!

@rvalyi @moylop260 Can you elaborate shortly the main characteristics, so we can move forward the decision making on which approche is best for the goal "make it dead simple for newcomers to start".

I'm very in favor of docker in general, because of it's portability. For real platform independence, we need a virtualization layer for it to work. To be as agile as possible, I'm in favor of bindmounting the source files directly into the docker via the new docker volume command (which implements smb, nfs etc.) this way a simple docker restart spins up a staging with the actual code, locally ore remotely. This way, there is no need for any build steps, as far as we do not add external dependencies. But this can be a very simple adaption to the Dockerfile itself as well (a apt-get install followed by a simple and generic docker commit).

What's your opinion about this strategy, coming from the background of your respective tools?

rvalyi commented 9 years ago

@blaggacao yes we do sync local files with what the Docker server uses, it's inherited from Devstep (Voodoo is based on Docker Compose, Devstep and Anybox recipe basically).

@moylop260 great! all that will converge, eager to test it and kudos for all the runbot work.

blaggacao commented 9 years ago

@rvalyi For understanding: Do you sync with the docker host, or the docker directly?

rvalyi commented 9 years ago

@blaggacao Voodoo and Devstep makes it the simplest you can imagine; you just fucking cd to the fucking host git project. Do voodoo run and then ak run and it's running. If you change a file with your IDE, then just do CTRL+C and ak run again (the Odoo dev mode will do it too). the project directory is a shared volume (-v option) shared with /workspace in your Docker container (read about Devstep to understand http://fgrehm.viewdocs.io/devstep/ )

Note: the guy behind Devstep isn't exactly a beginner, it's him who did the Vagrant LXC plugin for years for instance...

blaggacao commented 9 years ago

OK nice, and I assume it's platform independent?

rvalyi commented 9 years ago

@blaggacao I believe Devstep only works on Ubuntu. As for Voodoo, provided we do some Vagrant fall back launcher we can easily have a degraded mode on Windows and MacOS.

blaggacao commented 9 years ago

@moylop260 @moylop260 I'd like to discuss the pros and cons semi-publickly in a chat room, as it seems by now, we are doomed to IRC. Can I join there without tooling? Are you there at some time of today's wonderful day?

bealdav commented 9 years ago

I confirm voodoo is a really efficient dev environnement. All akretion team use it and now some of our clients to contribute to the dev. Thanks for that to the main contributors @rvalyi and @sebastienbeau and also to Anybox for their recipe embedded in voodoo. It miss a good and updated documentation.

blaggacao commented 9 years ago

I yesterday had the chance to test @moylop260 solution. I only can list the pros, as I percive them, those are:

Relies on travis.yml for the environment generation. Travis is already maintained in all OCA repositories by the "real experts". It's watertight and already present.
It uses otherwise just standard docker tooling, there is no additional plumbing there. You work directly with the docker commands.

What it does is, it transforms the defintions of the travis.yaml into a buildable dockerfile and adds a build and run script which does ease the job for the unexperienced. It fetches all contents from remote or local (!) repos, so it is proxied by the git subsystem. This ensures an aligned workflow. Finally, it syncs source files via sftp or similar protocol, but this is likely to change with the advent of docker volume and the docker itself can delegate this syncing part to the host. Even better...

Anyone, can give insight about the pros and technical details of voodoo to get this discussion to the next level? Thx.

blaggacao commented 9 years ago

Another question, how do we get compiled binaries for this tool?

yajo commented 9 years ago

Just my 2 cents:

Ship an editorconfig
You can review yajo/odoo docker image. No custom vodoo, just docker-compose.

blaggacao commented 9 years ago

Thank you for your input! I think editorconfig is quite a good idea, too. It seems they enlarged support for editors by now. The problem to spin up instance does unfortunatley not stop with a single docker image. Many repositories need a special environment either OS or python, which needs to be handled. But the bigger problem right now is that it doesn't work smoothly on windows because of the way the code is mounted to the docker in windows (via a hypervisor).

Your idea to include wdb is VERY intersting, I suppose the reason for it is to avoide the interactive shell problem, that might occure across guest-host connections?

bealdav commented 9 years ago

1 cent: http://komodoide.com/blog/editorconfig-your-komodo/

blaggacao commented 9 years ago

I take this as a vote for .editorconfig :+1: :smile: I'll update the Wiki Page. How should I construct a PR in order to implement this across OCA? Wen can vote then on this PR... But as there are many repos, I'm not sure how to do this... What's the standard way? One by one? Are there populating scripts in place? Can anyone help me with this? Remember please: let's vote on the concrete PR... :smile:

yajo commented 9 years ago

You need 1 PR per repo, adding a .editorconfig file to the root of each one.

I recommend you to do the first PR to https://github.com/OCA/maintainer-tools/ and let OCA review the file, and when merged just copy + paste + PR the same file to each other repo.

About the Docker volumes + Windows problem, just drop Windows :wink:. Another much harder option would be to share a folder from windows and add samba to the server, mounting sources from there in the VM. Even better, you can add the samba stuff in another image and link it to the stack.

And about wdb, that's a problem that it solves, but it's also about wdb being the most awesome debugger in history.

blaggacao commented 9 years ago

@Yajo Thank's for your inputs! That's the spirit, the problem about dropping windows is, that it is not sustainable because with their new strategy, they are the only "deviceless os" on the market. I bet the gain market share. Obviously programmers love linux, but odoo is not so hard to understand for non programmers, so that we would need to exclude them... Actually, I think samba with the new docker volume driver is the adequate solution... For now, there are rough edges around the virtualization layer (hyper-v, virtualbox), not so much the sharing, this is quite feasible... If there is an option for nfs on windows, we could even reuse an existing volume driver https://github.com/SvenDowideit/docker-volumes-nfs But this guy is also very into samba, and for sure will come up with something during the next months...

blaggacao commented 9 years ago

Let's see: https://github.com/SvenDowideit/docker-volumes-nfs/issues/5

blaggacao commented 9 years ago

@Yajo do you know go programming?

blaggacao commented 9 years ago

Needed Tooling Design: https://docs.google.com/document/d/11D4gAk05uE7LM5xftaiJexZN1Xbnh3iCcAi8rL7sA_8/edit#heading=h.8zzxhxwqd8d8

yajo commented 9 years ago

I never used Go, sorry.

If we want an easy, stable and reproducible environment for everyone, Docker is probably the best chance. Although obviously there is no native version for Windows yet, it is officially supported through VirtualBox.

Also this might be interesting for Windows users: https://github.com/docker/docker/issues/7052#issuecomment-49166340

Now opinions:

Voodoo adds little to docker-compose except for some syntactic sugar. docker-compose now supports inheritance of .yml files, so an environment with some of these preconfigured to launch different scripts in the image and just switching it. Examples:

voodoo server_test = docker-compose -f server-test.yml up
voodoo code_test = docker-compose -f code-test.yml up
voodoo console = docker exec -it odooserver_app_1 bash

So mostly voodoo adds alias, which is something quite personal to each developer IMHO; and anyway if you want to standarize those, just add alias.sh and source it. It will be much smaller and easier to maintain. Or make it even better with environment variables (footnote).

I'd prefer to stick with docker-compose since it's a common tool to all Docker community and it eases the learning curve for those knowing it (or at least reuses it for those coming directly from Odoo). Also it's compatible with voodoo for those wanting it as an alternative manager.

In any case, I don't think implementing Voodoo in Go is a good idea. docker-compose itself is made in Python, and Odoo too, so all OCA developers are more comfortable with that language. I see no need for rewriting.

Footnote

IMHO it's much better to do that with variables. Imagine this docker-compose.yml:

app:
 env:
  CODE_TEST: 1
  BUILD: 1
  DEBUG: 1
  DUMP: 1
  [...]

The developer comments out what he does not want, runs docker-compose up, and the docker image does the rest.

Another idea is to do it by scripts, as you saw I have in my stack, but maybe this is easier for sharing with a community.

blaggacao commented 9 years ago

I'm basically very much on the edge of 7052, so this is atm my main concern, but I have same clues... :dancer:

I'm dreaming of a vodoo get as a wrapper around the heavyilifting of setting up a git repo (with tracking branches, upstream, origin, etc.) it also could take care of OCA folder standards, so that those are no more bother of personal choice... (yes it is sometimes difficult to figure out how to best organize your code)

vodoo up would be marely syntactic sugar arround docker or docker-compose, for sure... So, maybe it should have less prio. Well both thinks, actually, because the first thing is to define a good set of base containers... any additional tooling is then nice to have... So, I agree with you!

yajo commented 9 years ago

CC @rafaelbn

moylop260 commented 8 years ago

Fixed with the following script https://github.com/OCA/maintainer-quality-tools/blob/master/git/pre-commit

OCA / maintainer-tools