Make Travis-CI script smarter about dependencies

[msmith-techempower]

Update our Travis-CI scripts to be more intelligent about situations in which dependencies are updated.

For example, if toolset/setup/linux/languages/dotnetcore.sh is updated, only collect tests which have dotnetcore in the fw_depends line.

[NateBrady23]

Here's how I see the scope of this work, and what I'd like this to look like:

Travis-CI Diffing Tool

Should be moved to the travis folder

Right now this is called run-ci.py which runs the diffing tool then runs run-tests.py with the gathered tests. This runs every time, for each matrix entry. Let's first move this to the travis folder, call it something like travis-diff.py and it shouldn't need any of the tfb suite dependencies. See below.

Run it once, before anything, and save results to an environment

Instead of running this for each matrix entry, we can have a global $TRAVIS_DIFF variable (or cached file). If it's empty we run the diffing tool. The tool will then determine which tests need to be run and save it there. For each matrix run, see if the test exists in $TRAVIS_DIFF (or something like run-all) and proceed to the installation part, or exit 0. Doing this as the first step means we can skip installing any dependencies if the test doesn't need to be run, and cut 30 seconds or more off skipped tests.

Diffing Outline

1. Test for toolset changes. Any toolset changes, like to bash_functions would kick off an entire run. Skip the rest.
2. Because of the way the toolset is organized now, .sh files can exist at any level within toolset/setup/linux. However, we are keeping to the structure toolset/setup/linux/databases/mobgodb/<mongo files>. So we know if there are any changes to toolset/setup/linux/database/monogodb/* find all frameworks that have an ^fw_depends *mongodb*$ (pseudo regex 🤷‍♂️ ) Any changes to any other *.sh files will look for any framework directory that includes an fw_depends of it anywhere.
3. As before, any files that changed within any framework directory will kick off a run for that framework directory. Push those on to $TRAVIS_DIFF.

If we can accomplish this, it should also speed up our full runs, as well as the ones where a lot of frameworks are skipped, but more importantly as @msmith-techempower pointed out, wouldn't kick off a full run when not needed.

[msmith-techempower]

Right now this is called run-ci.py which runs the diffing tool then runs run-tests.py with the gathered tests. This runs every time, for each matrix entry. Let's first move this to the travis folder, call it something like travis-diff.py and it shouldn't need any of the tfb suite dependencies.

Love it.

Instead of running this for each matrix entry, we can have a global $TRAVIS_DIFF variable (or cached file). If it's empty we run the diffing tool.

This is only cached for the tests in the current job, right? If so, this sound correct.

The tool will then determine which tests need to be run and save it there. For each matrix run, see if the test exists in $TRAVIS_DIFF (or something like run-all) and proceed to the installation part, or exit 0. Doing this as the first step means we can skip installing any dependencies if the test doesn't need to be run, and cut 30 seconds or more off skipped tests.

About 10,000,000% this. In fact, you could do this in the before_script stage, probably, and avoid lots of cruft.

1. Test for toolset changes. Any toolset changes, like to bash_functions would kick off an entire run. Skip the rest.

Yes, but we may (in a future change) need something 🤷‍ to manually kick off a forced entire run. Adding a space to bash_functions feels hacky, but it'll work for now, I guess.

2. Because of the way the toolset is organized now, .sh files can exist at any level within toolset/setup/linux. However, we are keeping to the structure toolset/setup/linux/databases/mobgodb/. So we know if there are any changes to toolset/setup/linux/database/monogodb/ find all frameworks that have an ^fw_depends mongodb$ (pseudo regex 🤷‍♂️ ) Any changes to any other .sh files will look for any framework directory that includes an fw_depends of it anywhere.

I think this needs to be more general. There are, and should be, dependencies which depend on other dependencies but are not strictly linked. An example of this is openresty, which depends on lua, but lua can be depended upon by itself.

So, one problem you're going to face is walking the dependency chain. Example: you update lua.sh; you need to check everything that has fw_depends lua, and that includes openresty.sh which is a dependency... so, you will need to also check everything that fw_depends openresty, and so on and so on. I don't think this will get too difficult, but it's something to consider.

3. As before, any files that changed within any framework directory will kick off a run for that framework directory. Push those on to $TRAVIS_DIFF.

Yis

If we can accomplish this, it should also speed up our full runs, as well as the ones where a lot of frameworks are skipped, but more importantly as @msmith-techempower pointed out, wouldn't kick off a full run when not needed.

Love it

[NateBrady23]

Travis has a command to automatically skip doing any travis runs: [ci-skip] in the commit message. So to do a full run on all matrix tests we can do something similar like [ci-run] and check for that commit message in step 1.