Why do new commits to a branch result in 2 shippable jobs in different "contexts"?

[jli]

Problem

My team is using Shippable for CI on our Github repo. Dashboard: https://app.shippable.com/github/gro-intelligence/api-client/dashboard/history

I noticed that when I push changes to a feature branch as part of a pull request, Shippable launches 2 jobs: one with "context" of my feature branch, and the other with the context of "development" (our main branch, which the PR was merging into).

The job with the feature branch "context" succeeded (job 3937 on branch jli-poetry2). But the job with the development branch context failed (job 3938), due to missing an encrypted/secure environment variable that I added to shippable.yml in my feature branch.

Questions

• Why there are 2 jobs when I push new commits to my feature branch? I would have expected just 1 job to run, in the context of my feature branch.
• Why did the 2nd job fail? It seemed to ignore my changes to shippable.yml which added the secure environment variables.
• What exactly are "contexts" in Shippable? (I didn't see reference to this term in the docs.)

More details

• the relevant pull request: https://github.com/gro-intelligence/api-client/pull/277
• change to shippable.yml where i added the new environment variable: diff on github
• the variable is missing on the job status console page for job 3938, and it's present for job 3937 despite them both being the same commit. The only different is the "context":

[jli]

I believe this comment answers the question about why there are 2 jobs when I push new commits to a PR: https://github.com/Shippable/support/issues/3938#issuecomment-342244497

However, I am still a bit confused why the ENV for the "development" context job doesn't include the new environment variables I added to shippable.yml in my branch.

[a-murphy]

The pull request build for the pull request to the base branch would normally run with a merged shippable.yml file if shippable.yml was updated in pull request. However, there are a few cases where it can't be determined that the file was updated. Did your pull request previously contain a few hundred more files? Too many files for the GitHub API to return is the most common reason. And if there are multiple YML files with different names, such as shippable.yml and .shippable.yml, it can be that the file expected wasn't the one used.

[jli]

@a-murphy Thanks for the response! In this case, the pull request only changed 11 files, and there's only one file that matches *shippable* in my repository.

What seems to be happening

The PR-merged-to-base-branch builds don't have the secure environment available, as a security precaution. From http://docs.shippable.com/ci/env-vars/#security-considerations

Due to the security risk of exposing your secure variables, we do not decrypt secure variables for pull request from the forks of public projects. Secure variable decryption is limited to the pull request triggered from the branches on the same repository.

This makes sense. However, it's even happening for PRs based on branches from the same repository. Seems like there may be a bug in the logic for avoiding decrypting secure variables?

Example

• This PR https://github.com/gro-intelligence/api-client/pull/281 tries to make use of a secure environment variable.
• It triggered 2 shippable runs:
  • context: PR branch https://app.shippable.com/github/gro-intelligence/api-client/runs/3984/1/console
  • context: base branch https://app.shippable.com/github/gro-intelligence/api-client/runs/3986/1/console

The 2nd job fails due to trying to use the secure environment variable. You can see in the Shippable job header that the 2nd job is missing "#SHIPPABLE_SECURE_VARIABLE" from the "ENV" section, which is present in the first job.

This PR is from a branch on the repository, not a branch on a fork, so I'd expect it to work.

Thanks!