max3903 commented 9 years ago

Write a script to be triggered during a merge:

[ ] check if version has changed in the manifest of all the modules of the branch (better detection could be implemented later)
[x] generate a setup.py and an egg file for each modified module, using the dependencies and extra-dependencies from the module manifest
[ ] generate a deb package (some doc: https://pypi.python.org/pypi/stdeb#bdist-deb-distutils-command)
[ ] generate a rpm package (https://docs.python.org/2/distutils/builtdist.html)
[x] publish the wheel on Pypi (with setuptools)
[ ] publish the deb on deb.odoo-community.org
[ ] publish the rpm on rpm.odoo-community.org
Tests
[ ] use pip, yum or apt-get to deploy an instance

max3903 commented 9 years ago

ping @dreispt @rvalyi

moylop260 commented 9 years ago

Hello @max3903

check if the versions has changed in the manifest of all the modules (better detection could be implemented later)

Can you help to understand this point? Do you want to check all oca projects to identify if the version value is agree with guideline? If is yes, then we can add in MQT lint checks. Example MQT/234

generate a setup.py and an egg file for each modified module, using the dependencies and extra-dependencies from the module manifest

rvalyi commented 9 years ago

Hello @max3903

I'm not sure I understand the philosophy of what you want to achieve: so it seems you want a new egg for each git commit right? But in my opinion, the solution is rather to have the package manager to download from the git repo directly than generating fake release at every commit with no meaning at all. If the commit is the module version (why not), then the git branch should be used directly. This is achieved a bit with Buildout, but probably the biggest drains are the download of the dependencies recursively and from git too eventually. This is what I mean when I say we are lagging far behind stuff like Bundler. For me, having the buildout recipe use pip or a superior package manager seems like smarter thing worth spending efforts.

For me publishing a python module at every commit is not something worth dedicating resources. It could be at each git tag eventually but at each commit I think it's better to use the git repo directly. At every commit we just add an extra caching and level of indirection for the module, adding risk of errors in the chain with no benefit I can see.

Could you elaborate on you idea please? Is that for the module discovery by the end user? If yes I'm not convinced either as Pypi is not really much more user friendly than just a search on Github.

rvalyi commented 9 years ago

@max3903 ok so you sent a mail too. I m reading your email then...

rvalyi commented 9 years ago

@max3903 OK. So correct me if I m right: you don t want to publish the module at every commit, you only want to publish it if version changed based on what is written in openerp.py ? Then it makes sense indeed. I would be surprised if there is no such tools already to help doing that. Then we would just adapt them for the Odoo module meta information. So I suggest 1st looking for existing tools to automate the publishing of python modules on pypi.

rvalyi commented 9 years ago

Generally speaking, when I say we lack behind Bundler and this block from having 1 module = 1 repo, I'm mainly talking about:

being able to specify released module version (pypi) or development version (git). You will typically want peace of mind for stable features of your deployment and choose released modules while you will want the most direct workflow possible for features you or the OCA are actively developping for the project, and then you will want to set the sources to git. So today Buildout can use the two but then yes publishing released modules as eggs would help closing the gap.
being able to do this recursively for the dependencies of the modules without having to resolve everything manually. Here I think we have nothing. By getting stable modules published as egg, we could have the dependencies resolution for the stable modules, is this your idea? As for having this to work with development versions, may be having pip to fetch modules in Buildout could get this to work?

max3903 commented 9 years ago

@moylop260 I want to check if the version of a module is different between the source and destination branches. If it is then we generate a new package, otherwise nothing happen.

Yes it is similar. I understood that @dreispt has already done that part.

@rvalyi The script will be triggered by a merge, not a commit. I updated the description accordingly.

Regarding the Bundler comparison,

let's package released versions first. Development ones can come later.
pip, apt and yum can manage dependencies recursively. I want to be able to do:

# apt-get install odoo8-hr-commission

and it would download, install, configure and start PostgreSQL, Python, Python libraries for Odoo8, Odoo8, Python libraries for my modules and its dependencies, Odoo modules that hr-commission depends on and hr-commission module.

nhomar commented 9 years ago

2015-09-02 16:47 GMT-05:00 max3903 notifications@github.com:

apt-get install odoo8-hr-commission

in favor of multi platform environments

pip install odoo8-hr-commission

Saludos Cordiales

CEO at Vauxoo https://www.vauxoo.com Odoo's Gold Partner.

[image: --] Nhomar Hernandez [image: http://]about.me/nhomar http://about.me/nhomar?promo=email_sig

rvalyi commented 9 years ago

Hello, yes, I think like @nhomar, we should focus on pip packaging, not Debian. Distro packaging will be assuming too many debatable choice and won't be usefull for many of us. Once we have pip packaging it's really easy to have a full blown distro install if we need to.

@max3903 I know that pip install stuff recursively of course. My point is that at the moment when we specify a branch to use in Buildout, these dependencies are not managed recursively and this has to be done manually. Having pip modules will help somewhat in the sense that we will then be able to specify the wanted leaf modules as eggs to install in the buidout.cfg.

Now what you can do with a Gemfile and that would be really nice to have is to specify released packaged to download the dependency tree of modules but still be able to pin a determined module to a dev version with a git repo and get the dependency resolution automatically and pinnable. This is really something very usefull in Ruby projects that I'm lacking a lot in Odoo.

At some point it will be interesting to investigate this Bundler clone in Python https://github.com/Deepwalker/pundler But not to be taken lightly as a bunch of very smart people spent a lot of efforts to bring Bundler where it is today...

max3903 commented 9 years ago

@nhomar bdist supports egg, rpm, deb and even msi and wininst. https://docs.python.org/2/distutils/builtdist.html

so you can use what you want.

moylop260 commented 9 years ago

@rvalyi: download of the dependencies recursively

+1 I commented a current issue that we have by use N platforms: https://github.com/OCA/runbot-addons/issues/38#issuecomment-133185723

If our have centralize the installation of packages recursively this should be a good approach to create instance in all paltforms: runbot, docker, buildout, documentation, travis, manifest external_dependencies, manifest depends, erp_peek with a simple: * install oca-odoo80-hr-commission

@max3903: ... and it would download, install, configure and start PostgreSQL, Python, Python libraries for Odoo8, Odoo8, Python libraries for my modules and its dependencies, Odoo modules that hr-commission depends on and hr-commission module.

+1 A little comment: database environment should be splitted because exists environment with an (or multiple) odoo server and a (or multiple) postgresql server.

dreispt commented 9 years ago

I'm adapting my previous work on pipo to fit this design. I'm assuming that new packages will only be built on a version bump, and I can fit in version number not yet adapted to the OCA guideline format.

I think we can use odoo-hr-commission instead of odoo8-hr-commission. Full version numbers will include the Odoo version, so if you want v7 you will require < 8.0.

lmignon commented 9 years ago

@max3903 Nice initiative! For sure, packaging and releasing our addons as simple eggs with the dependency properly defined will ease the deployment. @rvalyi I understand and share your concern on how to deal with module we want to get the dev version. The problem mainly comes from that we don't have 1 repo = 1 module. With a 1 repo = 1 module it would be possible to easily get the dev version with pip

pip install -e git+https://github.com/acsone/anybox.recipe.odoo@master-add-release-recipe#egg=anybox.recipe.odoo

A lot of extension exists to specify in buildout that you want some 'dev' version of specific eggs in place of the official one. gp.vcsdevelop, mr.developer, ... Here it is my buildout, when I want to use the dev version of pycoda with l10n-belgium

[buildout]
parts = openerp
find-links = http://download.gna.org/pychart/
versions = versions
socket-timeout = 60
extensions = gp.vcsdevelop
vcs-extend-develop = git+https://github.com/acsone/pycoda@master#egg=pycoda
develop-dir=src
vcs-update = False

[openerp]
recipe = anybox.recipe.odoo:server
version =  git https://github.com/odoo/odoo.git openerp 8.0
addons =   git https://github.com/OCA/l10n-belgium.git addons-l10n-belgium 8.0
           git https://github.com/acsone/l10n-belgium.git addons-l10n-belgium 8.0-add-account_bank_statement_import_coda
           git https://github.com/OCA/bank-statement-import bank-statement-import 8.0
           git  https://github.com/OCA/account-financial-reporting account-financial-reporting 8.0

eggs = xlrd
       pyPdf
       pycoda

A solution (not tested) could be to create one egg-module-name repository by addon in github only used to publish the addons as a 1 module = 1 repo. The structure could be structured as

egg_hr_timesheet_task
  setup.py
  setup.cfg
  hr-timesheet #Git submodule -> https://github.com/OCA/hr-timesheet
  hr-timesheet/hr_timesheet_task
  hr-timesheet/..
  ....
  hr_timesheet_task # symlink -> hr-timesheet/hr_timesheet_task

The goal is to automatically provide a repository structure compliant with setuptools/pip that could be used to get the 'dev' version of an addon.

generate a setup.py and an egg file for each modified module, using the dependencies and extra-dependencies from the module manifest

A mapping file (or ???) will be needed since extra-dependencies doesn't always refer to the name of a python package but the name of an 'importable' python module. For exemple, [account_bank_statement_import_coda](git https://github.com/acsone/l10n-belgium.git addons-l10n-belgium 8.0-add-account_bank_statement_import_coda) depends of pycoda but the extra-dependencies refers 'coda' https://github.com/acsone/l10n-belgium/blob/8.0-add-account_bank_statement_import_coda/account_bank_statement_import_coda/__openerp__.py#L41

My 2 cents...

lmi

sbidoul commented 9 years ago

@lmignon I read it is possible to install a subdirectory of a git repo with pip although I never tried myself: pip install -e git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path # install a python package from a repo subdirectory

If that works, no need for 1 module one repo

sbidoul commented 9 years ago

@max3903 @dreispt how about generating setup.py from __openerp__.py in the addon directory with a github webhook on each push? Then having a bot that publishes the eggs on pypi for those where the version has changed on the main branch. I see more flexibility if we decouple the generation of setup.py from the publishing on pypi.

Further, I'm wondering if it is possible to have the setup.py entirely autogenerated in all cases. So having some mechanism to let authors hand-tune their setup.py would be even more interesting and could cover cases such as requiring specific addons versions in dependencies or specific python dependencies that cannot be expressed correctly in __openerp__.py (cfr the last § of https://github.com/OCA/maintainer-tools/issues/116#issuecomment-137377336) .

lmignon commented 9 years ago

@sbidoul A new thing learned. This kind of functionality if available makes the 1 module one repo 'obsolete'. The only things we would need are the setup files alongside openerp.py

dreispt commented 9 years ago

@sbidoul Setup.py and actual building are implemented as separate functions.

lmignon commented 9 years ago

@sbidoul @max3903 I did some tests to see if the subdirectory option works with pip. With a setup.py file alongside openerp.py, (see https://github.com/lmignon/l10n-belgium/tree/8.0-add-account_bank_statement_import_coda/account_bank_statement_import_coda), I'm able to run the following command.

./bin/pip install -e 'git+https://github.com/lmignon/l10n-belgium@8.0-add-account_bank_statement_import_coda#egg=openerp-account-bank-statement-import-coda&subdirectory=account_bank_statement_import_coda'

Nevertheless, at this stage one issue remains. I'm not able to import the package in my python interpreter.

max3903 commented 9 years ago

@dreispt Do you have a branch we can follow and contribute ?

@all What do you think of using the packaging to differentiate the maturity of a module ?

For example with Pypi, we can set the development status to '4 - Beta' by default and if the module is not modified for one month, then we update the development status to '5 - Production/Stable'.

With deb package, we could push to the testing repository by default and after one month, push to the stable one.

Some of you wanted to create an incubator, this could be a great solution. I proposed one month, but this is open to discussion.

dreispt commented 9 years ago

I'm not sure where the code should end up in the end, but in the meanwhile I'm working here: https://github.com/dreispt/pipo/tree/odoo2egg

max3903 commented 9 years ago

@dreispt in maintainer-quality-tools. The repo is cloned in travis configuration, so we can trigger the script during a merge.

lmignon commented 9 years ago

@dreispt If we pack our addons as eggs we must declare these eggs as namespace package under 'openerp.addons' In your script I don't find any trace of such declaration and when I 'pip' install the generated egg, the addon is not available in the python path with from openerp.addons. .... Do I miss something?

dreispt commented 9 years ago

Maybe that's because my scrip is using "odoo.addons". I guess that's a bad idea, and it should be "openerp.addons"... The setup.py is defined here.

lmignon commented 9 years ago

@dreispt Same result with 'openerp.addons'. But at this stage it seems that if we want to support addons packaged as eggs in odoo we need to monkey_patch Odoo itself to search for addons in the python path in addition to the addons_path option. (https://github.com/odoo/odoo/blob/8.0/openerp/modules/module.py#L84)

moylop260 commented 9 years ago

FYI sys.path with addons_path param: https://gist.github.com/moylop260/c18883487be31724f62a

2015-09-14 7:34 GMT-05:00 Laurent Mignon (ACSONE) notifications@github.com :

@dreispt https://github.com/dreispt Same result with 'openerp.addons'. But at this stage it seems that if we want to support addons packaged as eggs in odoo we need to monkey_patch Odoo itself to search for addons in the python path in addition to the addons_path option. ( https://github.com/odoo/odoo/blob/8.0/openerp/modules/module.py#L84)

— Reply to this email directly or view it on GitHub https://github.com/OCA/maintainer-tools/issues/116#issuecomment-140055766 .

Moisés López Calderón Vauxoo - OpenERP's Gold Partner Mobile: (+521) 477-752-22-30 Office: (+52) 477-773-33-46 web: http://www.vauxoo.com twitter: @vauxoo @moylop260 hangout: moylop260@vauxoo.com

sbidoul commented 9 years ago

Hi,

[updated to use the setuptools-odoo package]

I made my own try at packaging Odoo modules with setuptools. The general idea is to have a lean setup.py in each addon, next to __openerp_py, and do the rest (release, packaging and distribution) from that foundation.

Here is my prototype: https://gist.github.com/sbidoul/fb3d75de4ac0d44c04ed#file-setup-py

To test:

make sure you have https://github.com/odoo/odoo/pull/8758
create and activate a virtualenv
install Odoo: from the odoo main directory, run pip install -r requirements.txt && python setup.py develop
install setuptools-odoo with pip install git+https://github.com/acsone/setuptools-odoo.git --upgrade
copy the setup.py from the link above to your addon directory, next to __openerp__.py, and run python setup.py develop
start odoo.py as usual: your addon should be available without fiddling with addons-path

The other setup.py commands should work as expected (install, sdist, bdist...).

I'm looking forward to additional tests and feedback on this approach.

lmignon commented 9 years ago

@sbidoul I've successfully tested your script and it's really promising. For the first time I'm able to install Odoo with an OCA addons (for my test I've used https://github.com/OCA/web/tree/8.0/help_online) by using only standard python command. Thank you for this great contribution that opens a lot of new opportunities.

how to handle dependencies (and in particular dependencies to other addons that are not packaged with setup.py such as standard addons)?

If you package the prepare() method, you can include the list of names of addons provided by odoo. For these addons, the setup.py should only declare a dependency on 'odoo'.

how to handle python dependencies (wrong names in openerp.py)

A solution could be to rely on a requirement.txt file put next to the __openerp__.py file.

max3903 commented 9 years ago

@all do we really want to maintain additional files (requirements.txt, setup.py, spec file for rpm, control file for deb) when we could generate those files on the fly from the information in __openerp__.py?

sbidoul commented 9 years ago

@max3903 I'm not sure we can generate the setup.py from __openerp__.py in all cases. That's why I propose an approach where the canonical setup.py is trivial but can be manually hand-tuned if needed.

It would look like this https://github.com/acsone/setuptools-odoo#setuptools-odoo

Moreover, having setup.py in the addon directory enables classical python development workflow (eg deploying with python setup.py develop).

sbidoul commented 9 years ago

I made good progress with my packaging attempt.

Together with @lmignon we reached the conclusion that making openerp.addons a true namespace package is not possible without a relatively heavy patch which consists in removing all code from openerp/__init__.py. This cannot be done by monkey patching. The good news is that https://github.com/odoo/odoo/pull/8758 has been assigned so Odoo SA is looking into it. It will probably never make it in 8.0 nor 9.0, though.

However, following a suggestion from @lmignon I explored another approach which is giving very encouraging results and can be a very acceptable interim solution. In short it consists in packaging addons in another namespace package named odoo_addons, which is never imported but only used to autodiscover addons. A simple monkey patch in the Odoo startup script is sufficient to make this work at runtime, so it is completely viable. It should work with 9 and probably 7 too with minor changes.

Here is how to test it [update 2015-10-05: updated to use setup.py develop instead of install for setuptools-odoo]:

create and activate a virtualenv
install Odoo: from the odoo source directory, run pip install -r requirements.txt && python setup.py develop
install setuptools-odoo with git clone https://github.com/acsone/setuptools-odoo.git && cd setuptools-odoo && python setup.py develop
create your setup.py in your addon directory with the following command: wget https://gist.github.com/sbidoul/b5049a5e12a824318091/raw/a875a7073cd1f5250d20b9356199777f9faabd8f/setup-odoo_addons.py -O setup.py
run python setup.py develop in your addon directory
start Odoo with odoo-server-autodiscover and notice the addon is automatically discovered without using --addons-path

You can develop normally in the source directories of Odoo and your addon.

Then binary packaging and distribution can be done using the usual setup.py commands.

[update 2015-10-05] While this is already good for testing, note it is still work in progress. In particular, setuptools_odoo.prepare() does not yet generate install_requires nor a complete list of the static folder content in package_data.

@dreispt I had a look at pipo2eggs and started to re-use some good ideas from yours in setuptools-odoo (eg looking at several manifest names __odoo__.py, __openerp__.py, __terp__.py). I plan to continue with package_data and dependencies.

Testers and feedback welcome!

dreispt commented 9 years ago

@sbidoul Please feel free to take everything you like and leave behind everything you dislike. I'm sorry I didn't find the time to work more on this, but I believe it is in very good hands now.

sbidoul commented 9 years ago

@dreispt thanks. I hope you can find some time to test and provide ideas, because you have a lot of valuable experience in this matter.

max3903 commented 9 years ago

@sbidoul I get an error message at step 3:

$ pip install git+https://github.com/acsone/setuptools-odoo.git --upgrade
Downloading/unpacking git+https://github.com/acsone/setuptools-odoo.git
  Cloning https://github.com/acsone/setuptools-odoo.git to /tmp/pip-SUHrCu-build
  Running setup.py (path:/tmp/pip-SUHrCu-build/setup.py) egg_info for package from git+https://github.com/acsone/setuptools-odoo.git

Could not find any downloads that satisfy the requirement odoo>=8 in /home/max/odoo/8.0/git/odoo (from setuptools-odoo==0.2.0)
Downloading/unpacking odoo>=8 (from setuptools-odoo==0.2.0)
Cleaning up...
No distributions at all found for odoo>=8 in /home/max/odoo/8.0/git/odoo (from setuptools-odoo==0.2.0)

sbidoul commented 9 years ago

@max3903 thanks for testing. I think the error comes from the fact that odoo is installed in develop mode and setuptools-odoo in install mode. I updated the instructions to install setuptools-odoo in develop mode too.

lmignon commented 9 years ago

@sbidoul I've successfully tested your new proposal and it works like a charm... Although I am a little disappointed not to be able to package the modules in their real namespace, your solution seems to be the best compromise to move towards more standard in packaging and the distribution and of Odoo addons.

max3903 commented 9 years ago

@sbidoul I have made it through all the steps. Works like a charm :+1:

sbidoul commented 8 years ago

Thanks for the test @max3903. I'm making slow but steady progress. I wrote an update there: https://github.com/OCA/maintainer-tools/pull/156#issuecomment-157046354.

JordiBForgeFlow commented 6 years ago

@sbidoul We are in need to package the modules using DEB, instead of pip. At the stage of maturity we're in now with pip, do you think it is feasible to convert to deb packages using? https://pypi.python.org/pypi/stdeb#bdist-deb-distutils-command, and perhaps publish to deb.odoo-community.org?

sbidoul commented 6 years ago

@jbeficent I'm curious about the use case...

in theory it should work, as the setup.py we have are spec compliant. I've never attempted that tough.

odoo-autodiscover might be tricky to get right in a .deb. It's not necessary in 11 tough. And it may work without in 10 in some situations.

Anyway, it's an interesting rabbit hole to explore.

JordiBForgeFlow commented 6 years ago

Well, in debian environments deb packages are the proper way to go install applications. We have this requirement.

On Dec 29, 2017 5:15 PM, "Stéphane Bidoul (ACSONE)" < notifications@github.com> wrote:

@jbeficent https://github.com/jbeficent I'm curious about the use case...

in theory it should work, as the setup.py we have are spec compliant. I've never attempted that tough.

odoo-autodiscover might be tricky to get right in a .deb. It's not necessary in 11 tough. And it may work without in 10 in some situations.

Anyway, it's an interesting rabbit hole to explore.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OCA/maintainer-tools/issues/116#issuecomment-354465839, or mute the thread https://github.com/notifications/unsubscribe-auth/AHU_VkJmBmLpxD5U8Gqj_JWa3h79QBy4ks5tFRAWgaJpZM4F2w10 .

OCA / maintainer-tools

Packaging #116

Tests

apt-get install odoo8-hr-commission

pip install odoo8-hr-commission