Optimize CI for concurrent builds

[jrgriffiniii]

By default, CircleCI only permits a low threshold for concurrent builds (please see https://circleci.com/docs/2.0/parallelism-faster-jobs/ and https://circleci.com/pricing/). These test suites could be organized to run tests for more than one Role for optimizing the limited number of concurrent builds we have available.

[jrgriffiniii]

Using shell scripting doesn't appear to work with this approach (most of the features needed are only available in the BASH - CircleCI does not offer this):

#!/bin/sh -eo pipefail
PULIBRARY_ROLES=$( `find . -type d -regex '.*/roles/pulibrary.*/molecule/.*' | awk 'BEGIN { FS="/"; } { print $3 }' | uniq` )
CURRENT_PULIBRARY_ROLE=1
CURRENT_PULIBRARY_ROLES=${PULIBRARY_ROLES[@]:$CURRENT_PULIBRARY_ROLE:2}
/bin/sh: pulibrary.pas_code: not found
Exited with code 127

(which was built from https://github.com/pulibrary/princeton_ansible/commit/10c85700f1c06e120d448fc6f05186f6cafa651e)

[jrgriffiniii]

Trying to restructure the parallelization of test executions using a Python script is proving to be more problematic, and I'm not quite certain how to proceed, as the following error is triggered with an experimental approach:

#!/bin/sh -eo pipefail
python ./test.py
Executing the tests for roles/pulibrary.nginxplus
Traceback (most recent call last):
  File "./test.py", line 29, in <module>
    test_process = subprocess.run(['molecule', 'test'], stdout=subprocess.STDOUT)
  File "/usr/local/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 9] Bad file descriptor
Exited with code 1

https://github.com/pulibrary/princeton_ansible/compare/master...jrgriffiniii:issues-627-jrgriffiniii-circleci-concurrent-builds

[jrgriffiniii]

Looking at the documentation for GitLab CI, I'm not certain that this approach can be undertaken without addressing the following:

Tip: Parallelize tests suites across parallel jobs. Different languages have different tools to facilitate this.

(https://docs.gitlab.com/ee/ci/yaml/#parallel)

https://pypi.org/project/pytest-xdist/ provides parallelization for Python test suites, and https://github.com/pycontribs/pytest-molecule/blob/master/pytest_molecule/__init__.py#L157 might provide a way to integrate Molecule test suites for Roles with pytest

[jrgriffiniii]

MOLECULE_INVENTORY_FILE does not seem to be set when these tests are run using pytest:

E   KeyError: 'MOLECULE_INVENTORY_FILE'
________ ERROR collecting roles/pulibrary.tomcat8/molecule/default/tests/test_tomcat8.py _________
roles/pulibrary.tomcat8/molecule/default/tests/test_tomcat8.py:7: in <module>
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')
../../../../.local/share/virtualenvs/princeton_ansible-gphETEdW/lib/python3.7/os.py:678: in __getitem__
    raise KeyError(key) from None
E   KeyError: 'MOLECULE_INVENTORY_FILE'

[jrgriffiniii]

It appears to be the case that the testinfra integration for Ansible on https://github.com/pulibrary/princeton_ansible/blob/master/roles/pulibrary.ruby/molecule/default/tests/test_ruby.py#L6 must remain in the global scope. I was able to get the Docker environment provisioned successfully, but the following pattern simply will not work:

import os
import testinfra.utils.ansible_runner
import unittest

class Apache2TestCase(unittest.TestCase):
    def setUp(self):
        self.testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
                os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')

    def test_is_apache_installed(self, host):
        pkg = host.package('apache2')

        assert pkg.is_installed

    def test_apache_listening_http(self, host):
        socket = host.socket('tcp://0.0.0.0:80')

        assert socket.is_listening

[jrgriffiniii]

I'm very sorry, I'm now becoming skeptical that pytest can at all be used to integrate with Molecule using the approach which is currently undertaken. During executing, the following Ansible inventory is generated in a role-specific directory: ~/.cache/molecule/pulibrary.apache2/default/inventory/ansible_inventory.yml:

# Molecule managed

---
all:
  hosts:
    instance: &id001
      ansible_connection: docker
  vars:
    molecule_ephemeral_directory: '{{ lookup(''env'', ''MOLECULE_EPHEMERAL_DIRECTORY'')
      }}'
    molecule_file: '{{ lookup(''env'', ''MOLECULE_FILE'') }}'
    molecule_instance_config: '{{ lookup(''env'', ''MOLECULE_INSTANCE_CONFIG'') }}'
    molecule_no_log: '{{ lookup(''env'', ''MOLECULE_NO_LOG'') or not molecule_yml.provisioner.log|default(False)
      | bool }}'
    molecule_scenario_directory: '{{ lookup(''env'', ''MOLECULE_SCENARIO_DIRECTORY'')
      }}'
    molecule_yml: '{{ lookup(''file'', molecule_file) | molecule_from_yaml }}'
ungrouped:
  hosts:
    instance: *id001
  vars: {}

Running pytest alone does not seem to regenerate these, so I'm concerned that avoiding the usage of molecule test without abandoning testinfra might not be possible.

[jrgriffiniii]

https://github.com/renderedtext/test-boosters is provided as an example for parallelizing builds using GitLabCI. From what I'm currently seeing, these jobs are issued using shell invocations: https://docs.gitlab.com/ee/ci/yaml/#parallel

The CliParser for this Gem is what parses the job arguments: https://github.com/renderedtext/test-boosters/blob/master/lib/test_boosters/cli_parser.rb#L30

These, in turn, are used to select the subset of files which are actually run on each machine using:

• https://github.com/renderedtext/test-boosters/blob/master/lib/test_boosters/boosters/base.rb#L21
• https://github.com/renderedtext/test-boosters/blob/master/lib/test_boosters/boosters/rspec.rb#L24
• https://github.com/renderedtext/test-boosters/blob/master/lib/test_boosters/job.rb#L36
• https://github.com/renderedtext/test-boosters/blob/master/lib/test_boosters/shell.rb#L6

[jrgriffiniii]

https://github.com/pulibrary/princeton_ansible/compare/gitlab-test-jrgriffiniii introduces an attempt to work with invoke in order to leverage the GitLab CI parallelism for distributing Role tests between build workers. https://github.com/tox-dev/tox-venv/issues/29 currently prevents me from integrating this into tox for invocation.

[jrgriffiniii]

pipenv shell requires an interactive terminal; these are not offered for GitLab CI build worker environments (perhaps due to Docker constraints).

[jrgriffiniii]

The failures on https://gitlab.com/pulibrary/princeton_ansible/-/jobs/291891969 indicated that installing Python dependencies fail due to the absence of the gcc compiler in the build worker environment. I'm now attempting to ensure that these tools are available for each worker, but I hope that this does not become an issue with increasing the time for each parallel build.

[jrgriffiniii]

The build runner environment (Alpine Linux) required the following packages be installed:

• linux-headers
• musl-dev
• gcc
• libffi-dev
• libressl-dev
• make

[jrgriffiniii]

https://gitlab.com/pulibrary/princeton_ansible/pipelines/81374897/builds ensures that the branch runs a random subset of roles to be tested on the GitLab CI build runner. Unfortunately, the Docker daemon is not running for each test suite (this is likely a configuration error), and the test results are (falsely) reported as passing in these cases.

[jrgriffiniii]

It should also be noted that this branch should be able to be reconfigured to work with CircleCI: https://circleci.com/docs/2.0/parallelism-faster-jobs/#using-environment-variables-to-split-tests

[jrgriffiniii]

https://github.com/pulibrary/princeton_ansible/commit/c343d95c747691eac3f7d90085768b75b7788e55 successfully runs the test suites, but it appears that failures are still not propagated (perhaps the destroy exit code is consistently 0?)

[jrgriffiniii]

https://gitlab.com/pulibrary/princeton_ansible/pipelines/81418923 gets test suites for Roles distributed between GitHub build runners.

[acozine]

We moved to GitHub Actions in part due to these concerns. Obsolete.