Weekly test should be able to dispatch workflow with arguments

[rgildein]

There are some repos which will benefit from dispatching workflow with arguments.

For example, the charm-prometheus-juju-exporter could benefit from it and the weekly test could run multiple times for each juju-channel. This was discussed internally, and we decide to drop juju-channel 3.1, 3.2 and 3.3 from #52 due to stability (lot of times one will fail) and time expense of such run.

Example:

         - repo: charm-prometheus-juju-exporter
            workflow_file_name: check.yaml
            branch: main
            arguments: 
              juju-channel: '3.1/stable'
         - repo: charm-prometheus-juju-exporter
            workflow_file_name: check.yaml
            branch: main
            arguments: 
              juju-channel: '3.2/stable'
...

[samuelallan72]

What's the benefit for this exactly?

The check.yaml in charm-prometheus-juju-exporter has the matrix config for juju channels, so it will run tests on all supported juju channels automatically.

Ideally, weekly tests should run all the tests that the test workflows normally run; if we skip some, then we could miss being alerted if the skipped ones are failing. If it's too time expensive to run all of them, then that's a problem with the workflow, not with the weekly tests (especially as the workflow only runs once a week in the weekly tests workflow, but it could run many times as someone iterates on a PR).

[rgildein]

The check.yaml in charm-prometheus-juju-exporter has the matrix config for juju channels, so it will run tests on all supported juju channels automatically.

Yes, but there is no retry and if tests are not stable, we got a lot of false positives.

Ideally, weekly tests should run all the tests that the test workflows normally run; if we skip some, then we could miss being alerted if the skipped ones are failing. If it's too time expensive to run all of them, then that's a problem with the workflow, not with the weekly tests (especially as the workflow only runs once a week in the weekly tests workflow, but it could run many times as someone iterates on a PR).

I agree that it could be dangerous that we missed something or need to update in two places.

This issue was opened for feature discussion, and you are maybe right and proposed solution is not correct, but I believe that we are getting the lot of failing results and a bunch of them are due to unstable tests in charm or snap. In an ideal world, we should make those tests more stable. Maybe we can propose some retry policy for failed jobs? WDYT?

[samuelallan72]

In an ideal world, we should make those tests more stable. Maybe we can propose some retry policy for failed jobs? WDYT?

I absolutely agree :+1: This is the direction I'd prefer to go rather than dropping tests.

[samuelallan72]

I'm closing this issue:

• We don't currently have an immediate use for this feature.
• This should be a means, not an end. If/when we need to do something that requires passing arguments to workflows, we can implement it as part of that task.