search

plone / plone.app.testing

Testing tools for Plone-the-application
http://pypi.python.org/pypi/plone.app.testing
2 stars 7 forks source link

How to debug which zcml's are loaded when running bin/instance and which ones with bin/test? #27

Open idgserpro opened 8 years ago

idgserpro commented 8 years ago

From plone.app.testing docs:

Create a configuration context with the disable-autoinclude feature set. This has the effect of stopping Plone from automatically loading the configuration of any installed package that uses the z3c.autoinclude.plugin:plone entry point via z3c.autoinclude. (This is to avoid accidentally polluting the test fixture - custom layers should load packages’ ZCML configuration explicitly if required).

The problem with this approach is that we have a perfectly functional website, but when trying to test it, a bunch of the tests simply do not work because the instance from bin/test is different from bin/instance.

We've already tried to ask about more convention over configuration in the test framework to ease the pain of users in Plone, with the approach we think is the best cost benefit from an user perspective, without success, but we comprehend the performance problem and the conceptual definition that the test should be the minimal possible (we were trying a hybrid, pragmatic approach).

Fact is: we've been bitten by these differences in Plone itself and it's test layers and we don't know how to solve the problem: we don't know which packages should be explicity added since the dependency tree is really huge, so is pratically impossible to do it.

We know we should be creating packages that don't rely in z3c.autoinclude, but the real world isn't yet perfect specially when talking about third-party packages.

The question is: is there at least a way to compare what's been automatically included when running bin/instance and bin/test? This can be a log, a snippet of code I need to run, a view, anything that can ease the pain when testing our infrastructure.

Another question: is there anything I need to compare as well that isn't automatic loaded in plone.app.testing? We're not that confident anymore with our tests since they're not running on same code/runtime as prodution.

Somewhat related: https://community.plone.org/t/what-is-best-practice-for-meta-zcml-files-and-test-layers-in-plone-app-testing/442

@tomgross @mauritsvanrees @tisto @jensens @hannosch We're adding you to the discussion from previous conversations and knowledge we believe you have about this issue, feel free to remove yourselves if not related. Thanks for your time.

gforcada commented 8 years ago

I may be totally wrong and I'm half asleep, but ... if I remember correctly the problem here is using the <includeDependencies package="." /> in your configure.zcml.

So if you replace that line for a list of <include package="five.pt" /> then on your testing layers you only need to load your own ZCML and that's it.

To compile the list of packages to include on your configure.zcml you should look at which dependencies you have on your setup.py (and double check that all of them are still relevant and that you are not missing any - z3c.dependencychecker can help here).

I'm I right?

mauritsvanrees commented 8 years ago

Correct.

idgserpro commented 8 years ago

@hvelarde This is somewhat related to your question in https://stackoverflow.com/q/35799092/644075: subtle differences between bin/instance and bin/test and what can we do to debug it.

idgserpro commented 8 years ago

@hvelarde and this issue https://github.com/collective/collective.cover/issues/586 as well.

hvelarde commented 8 years ago

@idgserpro I don't think so: I already removed <includeDependencies package="." /> in that package.

I'll take a look on collective.cover next week if you can't work on that today.

idgserpro commented 8 years ago

@hvelarde we still have differences when running bin/test and bin/instance, the meaning of this issue. If even removing includeDependencies we still have problems, we need a way to compare both instances and see what is difference to know what it needs to be added (or removed) in our testing.py.

idgserpro commented 8 years ago

After all, I must or not use z3c.autoinclude?

hvelarde commented 8 years ago

@idgserpro z3c.autoinclude is a different beast and it saves you the need of adding your eggs to zcml also; that's why your packages will be discovered by Zope by using only:

[buildout]
...
eggs = my.package

instead of:

[buildout]
...
eggs = my.package
zcml = my.package

I think this is safe enough but @gotcha had a different view some time ago:

http://bubblenet.be/blog/lets-get-rid-of-z3c.autoinclude

idgserpro commented 8 years ago

So, what you're saying is that it's safe to use z3c.autoinclude to avoid using the zcml part in our buildout, but we should avoid using <includeDependencies package="." /> in our configure.zcml? If so, we agree on that.

idgserpro commented 8 years ago

@pbauer we think this should be removed from bob templates. https://github.com/plone/bobtemplates.plone/blob/247585a1c777cc625e4e154cdffd2f7fe72862c7/bobtemplates/plone_addon/src/%2Bpackage.namespace%2B/%2Bpackage.name%2B/configure.zcml.bob#L12

Thoughts?