zaza should retry bundle deploy due to failure if charmstore drops the connection

This is annoying as it can make a large test fail simply because the charmstore is a bit overloaded. We can either do one of the following:

a) Detect the charmstore error (see below) in the deploy and just try redeploying the bundle. b) Download (and retry) all the charms first (like mojo does) and then deploy locally. Obviously, this would require bundle re-writing which may become hairy.

Example error is:

2020-12-23 14:34:55 [INFO] Deploying bundle '/var/lib/jenkins/checkout/0/rabbitmq-server/tests/bundles/bionic-ussuri.yaml' on to 'zaza-69976805f31c' model
2020-12-23 14:34:55 [INFO] Rendered template '<Template memory:7fa59ddc3d30>' to file '/tmp/tmptvo0qi10/local-charm-overlay.yaml'
2020-12-23 14:34:55 [INFO] Deploying overlay '/tmp/tmptvo0qi10/local-charm-overlay.yaml' on to 'zaza-69976805f31c' model
2020-12-23 14:34:56 [INFO] Resolving charm: cs:~openstack-charmers-next/cinder
2020-12-23 14:34:56 [INFO] Resolving charm: cs:~openstack-charmers-next/keystone
2020-12-23 14:34:56 [INFO] Resolving charm: cs:nagios
2020-12-23 14:34:56 [INFO] Resolving charm: cs:nrpe
2020-12-23 14:34:57 [INFO] Resolving charm: cs:~openstack-charmers-next/percona-cluster
2020-12-23 14:34:57 [INFO] Executing changes:
2020-12-23 14:34:57 [INFO] - upload charm cs:~openstack-charmers-next/cinder-426 for series bionic
2020-12-23 14:35:07 [INFO] ERROR cannot deploy bundle: cannot add charm "cs:~openstack-charmers-next/cinder-426": cannot retrieve charm "cs:~openstack-charmers-next/cinder-426": cannot get archive: Get "https://api.jujucharms.com/charmstore/v5/~openstack-charmers-next/cinder-426/archive?channel=stable": net/http: TLS handshake timeout
2020-12-23 14:35:07 [ERROR] {'default_alias': 'zaza-69976805f31c'}
2020-12-23 14:35:07 [ERROR] Model default_alias (zaza-69976805f31c)
2020-12-23 14:35:07 [WARNING] unknown delta type: id
Traceback (most recent call last):
  File "/tmp/tmp.nrqUT819yj/func/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/tmp/tmp.nrqUT819yj/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 278, in main
    test_directory=args.test_directory)
  File "/tmp/tmp.nrqUT819yj/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 213, in func_test_runner
    force=force, test_directory=test_directory)
  File "/tmp/tmp.nrqUT819yj/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 128, in run_env_deployment
    test_directory=test_directory)
  File "/tmp/tmp.nrqUT819yj/func/lib/python3.5/site-packages/zaza/charm_lifecycle/deploy.py", line 368, in deploy
    deploy_bundle(bundle, model, model_ctxt=model_ctxt, force=force)
  File "/tmp/tmp.nrqUT819yj/func/lib/python3.5/site-packages/zaza/charm_lifecycle/deploy.py", line 345, in deploy_bundle
    utils.check_output_logging(cmd)
  File "/tmp/tmp.nrqUT819yj/func/lib/python3.5/site-packages/zaza/charm_lifecycle/utils.py", line 493, in check_output_logging
    raise subprocess.CalledProcessError(popen.returncode, cmd)
subprocess.CalledProcessError: Command '['juju', 'deploy', '-m', 'zaza-69976805f31c', '/var/lib/jenkins/checkout/0/rabbitmq-server/tests/bundles/bionic-ussuri.yaml', '--overlay', '/tmp/tmptvo0qi10/local-charm-overlay.yaml']' returned non-zero exit status 1
ERROR: InvocationError for command /tmp/tmp.nrqUT819yj/func/bin/functest-run-suite --keep-model (exited with code 1)

openstack-charmers / zaza

zaza should retry bundle deploy due to failure if charmstore drops the connection #401