Run behat tests when secret is available

[szepeviktor]

I've noticed that behat tests run on PR-s even when the token is not available.

[iamdharmesh]

Hi @szepeviktor, Thanks for adding this, it looks good. However, it marks the action check as successful, which may lead to confusion about whether the tests are run successfully. Do you know the way to mark it as skipped if the secret is not available? any idea?

Thanks for your contribution here.

[szepeviktor]

The steps are skipped

Running CI on PR-s is not possible now because of the unavailability of tokens. I think this needs a decision.

[szepeviktor]

@iamdharmesh

The workflow is not valid. .github/workflows/testing.yml (Line: 23, Col: 9): Unrecognized named-value: 'env'. Located at position 1 within expression: env.VULN_API_TOKEN

https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability

For job-level if-s env context is not available :(

[szepeviktor]

I'd like to try reusable workflows!

[jeffpaul]

@szepeviktor @iamdharmesh checking in here to see if there's still work/decisions needed here or if its ready for review/merge?

[szepeviktor]

I've done all I could. Please merge/change/close it.

[iamdharmesh]

@jeffpaul It seems it is not possible to mark the job as skipped if the secret is not available, as of now. We can skip job steps as mentioned here but skipping steps shows a check as passed at the PR level. So, it shows the test passed without running the tests.

We tried some ways for this but as mentioned here, we have some limitations here.

I would suggest closing the PR as of now. Infuture, if GH action limitations get removed or we have better support for handling this then we may revisit this.

Thanks.

[szepeviktor]

There is a work-around, an indirect way.

if: ${{ github.event_name == 'pull_request' && ! github.event.pull_request.head.repo.fork }}

Please see https://github.com/szepeviktor/byte-level-care/actions/runs/6510918356/job/17685613640 I let you decide and implement it.

[iamdharmesh]

@szepeviktor Thanks for suggesting the new workaround. However, it seems that the "if" condition is running ahead of the matrix before it gets finalized. As a result, we can skip tests for all API providers but not for any specific ones yet (see the action title in the screenshot below).

I would prefer to put this on hold and close this for now until we have a better approach. I really appreciate your efforts to improve this.

Please feel free to reopen this if you think I misunderstood anything here. Thanks again for your contribution. cc: @jeffpaul