search

juju-solutions / bundletester

A juju charm and bundle test runner
http://juju-solutions.github.io/bundletester
15 stars 20 forks source link

OSError: Couldn't find executable for command 'charm-proof' #109

Open dannf opened 6 years ago

dannf commented 6 years ago

I have the following bundletester and charm snaps installed:

$ snap list
Name          Version    Rev   Developer  Notes
bundletester  0.11.0     58    lutostag   classic
charm         2.2.3      114   charms     -
core          16-2.29.3  3440  canonical  core
juju          2.2.6      2739  canonical  classic

When I try to run bundletester, it crashes:

$ bundletester -t cs:~ivoks/elasticsearch
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.jujucharms.com
Traceback (most recent call last):
  File "/snap/bundletester/58/bin/bundletester", line 11, in <module>
    load_entry_point('bundletester==0.12.2', 'console_scripts', 'bundletester')()
  File "/snap/bundletester/58/lib/python2.7/site-packages/bundletester/tester.py", line 166, in entrypoint
    status = main()
  File "/snap/bundletester/58/lib/python2.7/site-packages/bundletester/tester.py", line 140, in main
    suite = spec.SuiteFactory(options, options.testdir)
  File "/snap/bundletester/58/lib/python2.7/site-packages/bundletester/spec.py", line 395, in SuiteFactory
    suite.find_suite()
  File "/snap/bundletester/58/lib/python2.7/site-packages/bundletester/spec.py", line 233, in find_suite
    self.find_implicit_tests()
  File "/snap/bundletester/58/lib/python2.7/site-packages/bundletester/spec.py", line 293, in find_implicit_tests
    dirname=self.model['directory'], suite=self)
  File "/snap/bundletester/58/lib/python2.7/site-packages/bundletester/spec.py", line 97, in spec
    self.append(Spec(testfile, self.config, **kwargs))
  File "/snap/bundletester/58/lib/python2.7/site-packages/bundletester/spec.py", line 25, in Spec
    "Couldn't find executable for command '%s'" % cmd[0])
OSError: Couldn't find executable for command 'charm-proof'

Perhaps it should be calling "charm proof" instead?

ktsakalozos commented 6 years ago

Hi @dannf ,

In your setup make sure you have apt installed charm-tools. The recommended way of deploying charm tools is as a snap but for the needs of bundletester charm proof should be installed the old way. Have a look at the full set of dependencies we deploy on systems we need to run our tests on:
https://github.com/juju-solutions/kubernetes-jenkins/blob/master/integration-tests/install-deps.sh#L8

However, note that the use of bundletester is not recommended. You should instead write your tests using libjuju https://github.com/juju/python-libjuju

Thanks

dannf commented 6 years ago

On Tue, Dec 05, 2017 at 05:51:53PM +0000, Konstantinos Tsakalozos wrote:

Hi @dannf ,

In your setup make sure you have apt installed charm-tools. The recommended way of deploying charm tools is as a snap but for the needs of bundletester charm proof should be installed the old way. Have a look at the full set of dependencies we deploy on systems we need to run our tests on:
https://github.com/juju-solutions/kubernetes-jenkins/blob/master/integration-tests/install-deps.sh#L8

However, note that the use of bundletester is not recommended. You should instead write your tests using libjuju https://github.com/juju/python-libjuju

Konstantinos, Thanks for the reply! Yeah, I do use libjuju for my tests, but I'm confused about how that relates to bundletester, which I understand to just be a runner for tests.

Is there another test runner I should be using instead? Are in-charm tests no longer supported?

-dann

ktsakalozos commented 6 years ago

I am not sure what you mean by runner. Bundletester will make sure the juju environment is setup before running the tests, that includes creating models installing dependencies. I am suggesting you do this kind of operations without the need of bundletester. Both bundletester and amulet are not actively developed. The focus is on libjuju. libjuju paired with pytest is the combination we are using as you might have guessed from the install-debs.sh script.

What kind of support would you expect for in-charm tests? You can still keep your tests inside the charm and you can trigger them through whatever means you want.

dannf commented 6 years ago

On Wed, Dec 06, 2017 at 11:00:51AM +0000, Konstantinos Tsakalozos wrote:

I am not sure what you mean by runner.

bundletester is defined as "A juju-deployer based test runner for bundles and charms": https://github.com/juju-solutions/bundletester

I interpret that to meant that, like the "juju test" command before it, it would be able to run the tests from an arbitrary charm's test directory against an in-focus model and report the results.

Bundletester will make sure the juju environment is setup before running the tests, that includes creating models installing dependencies. I am suggesting you do this kind of operations without the need of bundletester. Both bundletester and amulet are not actively developed. The focus is on libjuju. libjuju paired with pytest is the combination we are using as you might have guessed from the install-debs.sh script.

+1 libjuju - we make heavy use of it in our project: https://jujucharms.com/u/ce-hyperscale/scalebot-jenkins

More on pytest below...

What kind of support would you expect for in-charm tests? You can still keep your tests inside the charm and you can trigger them through whatever means you want.

Good question - let me explain my goals here.

One of my team's responsibilities is to test workloads on specific hardware platforms. We do this using Juju charms and MAAS (using the above "scalebot" project). Generally these aren't charms that we write/maintain, we are just leveraging charms from the charm store and making sure they work well.

Now, we could of course write our own snowflake charm tests for the charms we care about - but, I've been pushing our team to instead try to integrate tests upstream in the charms following the specification here:

https://jujucharms.com/docs/devel/developer-testing

There it describes using bundletester to run each of the tests included in a charm, in a charm agnostic way. By contributing our tests in that format, everyone else running charm tests could make use of our work, and vice-versa.

However, if bundle-tester is deprecated (and I'd suggest updating the docs if so), it isn't clear that there is a standard way of running arbitrary tests from any compliant charm. If that's the case, then there's not a lot of justification to push for the extra effort of upstreaming our tests.

If the charm community adopts a new standard based on pytest, I'd be happy to use that instead - but I don't see any evidence that this has occurred.

-dann