Add the ability to select checks run via a CI/CD trigger using Command Line via tags

[tdeekens]

💡 For general support requests and bug reports, please go to checklyhq.com/support

Is your feature request related to a problem? Please describe.

We often trigger checks using a CI/CD trigger on a group. To do so we integrate them using a Command Line Trigger. This is needed as we use CircleCI.

When triggering checks using a Command Line Trigger we often do not need to run all checks within a group as we know what checks would test affected functionality.

Our groups are structured per environment with API and browser checks in each. Given we deploy API A we'd like to run browser check A but not B.

Allowing us to limit the checks run by a Command Line Trigger would not only reduce run time but also always keep us under the 30 second mark of maximum aggregated runtime for a Command Line Trigger.

Describe the solution you'd like

A Command Line Trigger already allows passing variables such as runLocation or deployment. Being able to pass an additional variable called tags would allow us to select a certain set of checks within the group in question to trigger.

Describe alternatives you've considered

Alternatives are splitting up our groups and running checks on a smaller group but comes at the complication of environment variable inheritance not working as nicely as with one group per environment.

Another alternative would be to allow sub groups instead of tags while each sub group has its own Command Line trigger.

Additional context

So far none.

[tnolet]

@tdeekens thanks for the extensive write up. We are soon going to have a look at a more unified, simpler and flexible way of running checks through our API. The CMD line triggers are a bit old and have some downsides we are very aware of. I think filtering based on tags makes a lot of sense as it is a pattern we use in many other places in the platform.

[tnolet]

@tdeekens we are moving ahead with a new "trigger" endpoint for running checks in a CI/CD or testing scenario. Part of this are two specific things:

1. make the API async to dodge any 30 second or long lived HTTP connections.
2. allow triggering checks by tag.

is the use case you described in this ticket still relevant for you and your team?

[tdeekens]

Hi @tnolet,

yes I think that would be relevant. The async nature would change a bit how we integrate using our CLI on CI but should be manageable. The tag filtering would be super nice!

[tnolet]

@tdeekens we are building a brand new CLI that should do the heavy lifting for you. I love you built your own CLI, but hopefully you will not have to maintain that one anymore. A lot more news on this will come soon!

[tdeekens]

@tnolet even better. Happy to drop code on our side 😄.

[tnolet]

@tdeekens we have a proposal for this for our new CLI https://github.com/checkly/checkly-cli and it's pretty much what we talked about. This is from our internal project doc:

Two additions to the CLI are needed:

1. Running checkly test, a user can add the --record flag to the CLI test command to record the test session. This means persisting the test runs, their metrics and assets and linking them to a Test Session entity we can display.

2. Running checkly account trigger --tags=tag1,"tag with spaces" --record has the same effect for recording a test session on arbitrary checks not necessarily part of the codebase. Note that

   1. This introduces the account scope.
   2. We need to allow tags with white spaces by using “some tag with space”
   3. The --record flag should work the same as checkly test

[tdeekens]

Thanks and sorry for the late reply. A bit too busy times.

I finally managed to port some of our checks to the CLI and I can't give anything but praise or positive feedback @tnolet - it's a joy to use!

I noticed that the --tags option isn't documented here is there a reason for that? I also can't find it in the source here. Is it really a feature?

[tnolet]

@tdeekens --tags is not a feature (yet?) on the test and deploy commands. We are seeing two scenarios here:

1. users that code, test and deploy their checks as code.
2. users that are not quite there yet, but want a way to trigger checks already in the same way as the test commands works.

• scenario 1 does not have filter by tags option.
• scenario 2 is in the works right now, shipping soon.

[tdeekens]

Thanks for the response. Can you, if you have time, elaborate why you distinguish between the two scenarios? To me (likely out of naivety) I would argue that it does not matter if a user gets to her checks by either checkly deploy or by authoring them by hand. In either case being able to checkly test --tags gcp-us,feature-a would help (at least us) to trigger checks after and event for a given environment for a specific feature or product line.

Right now the --grep option is file based hence can not be used as granular (or imposes file splitting restrictions) upon users. When for instance using the constructs one easily ends up with a practical abstraction which often can be for env of envs to create multiple checks across environments in one file.

Hope this makes sense.

[tnolet]

@tdeekens the split is pretty simple:

1. the checkly test command requires you to code your checks and have them inside a Project inside your code base. These checks might not be deployed yet, or might be a different version than you have in your account as you are merging a PR and only want to hit deploy after you are sure they are all good.

2. the upcoming checkly trigger command enables you to just trigger any check you have in your account. No coding or deploying needed. This is mostly a way to start using the CLI and move away from the older Command Line Trigger, which we will deprecate as it comes with the 30sec time limit. The trigger will have no hard time limit.

In any case, solution 1 is by far the most powerful as you get the "as-code" aspect, but we wanted to enable the other solution to for current users.

[tdeekens]

Got ya, I wasn't aware you where planning a trigger command also and was assuming you were distinguishing among the test and deploy command only for the scenarios.

[tnolet]

@tdeekens this feature is now available in our last CLI update, now living under npm install checkly. See our updated docs. https://www.checklyhq.com/docs/cli/command-line-reference#npx-checkly-trigger

[tdeekens]

Thanks a ton. Good timing. I was just using a relaxing evening to explore the CLI some more and setup a CI integration with it :)