Hit error: "Waiting for build on master" #290

Closed cgbl-90 closed 1 month ago

Describe the bug

When we try to run VTA, we it error Waiting for build on master

I suspect the root cause is that builds published by our CI (GitLab) have branch name master and builds published by Visual Tests plugin have branch name fu--------team:master. I have also digged a little bit into your source code and I suspect that different slug is returned locally and on GitLab CI. Anyway I don't understand why as setup is the same.

I have also tried to publish a build from my local machine (yarn chromatic --project-token XXX) and it's tagged with branch name fu--------team:master instead of just master

We digged a bit into your source code and figured out that if we change this line (https://github.com/chromaui/addon-visual-tests/blob/7d6d6c6575c4f7a07be3f38c0ddbe5fc6386b4e7/src/screens/VisualTests/BuildContext.tsx#L42) to ...(gitInfo.slug ? {} : {}) than everything starts working.

We are using an original repository, not a fork.

To reproduce

storybook link in the chat and Linear: AP-4531 reached out to support > chat 27254951045

Environment

OS X 10.15.7 Chrome 123.0.0.0 Storybook 8
Visual Test Addon 1.3.2

Chromatic considers any branch name containing a : to be prefixed with the repository owner name. For example: chromaui:main would be the main branch for a repository owned by chromaui. This owner name is then cross-referenced with the repository that's linked to the Chromatic project. If there is a mismatch between the branch owner and the linked repository owner, we consider the build to originate from a fork.

In this case it appears branch names are used that follow this format for another purpose. Unfortunately there is no way to configure this behavior, so I recommend changing the branch prefix separator to one of the safe characters. Typically a / is used for prefixing branch names. Alternatively (but not recommended), it's possible to override the branch name used by Chromatic using the --branch-name flag or CHROMATIC_BRANCH environment variable.

@cgbl-90 asked me to post our investigation directly in here so here it is:

Issue which we are facing isn't related to branch naming itself but to how visual testing addon compares local branch with branch on selected build (last build).

When addon request query AddonVisualTestsBuild to get build, it send branch name without prefixing owner, and pass owner as repositoryOwnerName. Original code. Here are variables of that query:

{
  "branch": "feat/FBI-169",
  "gitUserEmailHash": "9788c12d52ae5ec2f915877b18a9d6ad",
  "hasSelectedBuildId": true,
  "projectId": "Project:6412fd7f8d2c0cc2cc733d99",
  "repositoryOwnerName": "fuzebox-team",
  "selectedBuildId": "Build:661920ddf95b78bbfe1b559b",
  "storyId": "beta-button--default",
  "testStatuses": [
    "PENDING",
    "FAILED",
    "DENIED",
    "BROKEN"
  ]
}

and we got response, with following data in selectedBuild:

{
  "__typename": "CompletedBuild",
  "id": "Build:661920ddf95b78bbfe1b559b",
  "number": 1047,
  "branch": "fuzebox-team:feat/FBI-169",
  "commit": "468ba8003d85f43a51db7359c077620d4cce9bd6",
  "committedAt": "2024-04-11T08:25:21.000Z",
  "uncommittedHash": "e0875b0258ebc0cab93e7ebdb6ab909d9ecf006a",
  "status": "PENDING",
  "startedAt": "2024-04-12T11:54:24.818Z",
  "testsForStory": {
    "nodes": [
      {
        "id": "Test:661920f0eeb9242d80aefc5d",
        "status": "PASSED",
        "result": "EQUAL",
        "webUrl": "https://www.chromatic.com/test?appId=6412fd7f8d2c0cc2cc733d99&id=661920f0eeb9242d80aefc5d",
        "comparisons": [
          {
            "id": "TestComparison:661920f0eeb9242d80aefb1c",
            "result": "EQUAL",
            "browser": {
              "id": "Y2FwdHVyZS12Ni1jaHJvbWU=",
              "key": "CHROME",
              "name": "Chrome",
              "version": "121"
            },
            "captureDiff": {
              "diffImage": null,
              "focusImage": null
            },
            "headCapture": {
              "captureImage": {
                "backgroundColor": "rgb(252, 252, 255)",
                "imageUrl": "https://snapshots.chromatic.com/snapshots/6412fd7f8d2c0cc2cc733d99-661920f0eeb9242d80aefb1c/capture-a40ed72a.png?assetToken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJyZXNvdXJjZUtleSI6IjY0MTJmZDdmOGQyYzBjYzJjYzczM2Q5OS02NjE5MjBmMGVlYjkyNDJkODBhZWZiMWMiLCJleHAiOjE3MTUzNDA1ODV9.bVwhmXviNYcnAuCInLwuKk5jmQqcDyWb1L6jX5NU1tEuv2t05U9rCMEyVtdmgLyHVBq4q6e6NKgIwdHWrznUGkGooa4VREomdteTv_gt_-6FnHeHokMSn8A1TiZUU-6OcMFBuOhZuOEDAglzM3MUCI_FdMoKshNB3Zy_w4AmUOc24nGfUQnwQouAKatNwIsS1WXcE_X8CdKF_NcCw55-1ErnA_qpdJGsNQN1NPgt39Wqkv8qfMyeMYRwzivKNvkOekxctzS_xGBToItTWezpYia051BzDbxy73UqsT9wxoYewqfQQD7BHN8uiHJ8SzAnMZCtsVJr6r1IsA5MKz70pnXtx5C_I_hHMPRFslQFxyIV0XgmRruCUqdTf42s8q2AtaPaH6Xp-CXxF09-h1y-w8cZpBC0xkWX1XkzOHj8UuLMW9SaojfoJRTQ1ckU2fWWY3Oei9Ka-bKFXqLTNUA7cNAFjElSA1AnlHZrNSeRITxkkSzNJ-TEcxysIUXt4cZNoT5SAn3Zsu6pS154EFZDeJH7L0ZK6LCm8dLbNkv1uFHXZanK4WuxQgGfNNfqUv0iYQOrzMG2nd91dYEU8qC3ReK1B-PbZQysE3yubLmXGIuDVt0kNUq0JLtwtPgG0rczceph1WIrUX-eg4ED_v1OnNYAPtALy3kBxK6AnjYExws",
                "imageWidth": 77,
                "thumbnailUrl": "https://snapshots.chromatic.com/snapshots/6412fd7f8d2c0cc2cc733d99-661920f0eeb9242d80aefb1c/thumb/capture-a40ed72a.png?assetToken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJyZXNvdXJjZUtleSI6IjY0MTJmZDdmOGQyYzBjYzJjYzczM2Q5OS02NjE5MjBmMGVlYjkyNDJkODBhZWZiMWMiLCJleHAiOjE3MTUzNDA1ODV9.bVwhmXviNYcnAuCInLwuKk5jmQqcDyWb1L6jX5NU1tEuv2t05U9rCMEyVtdmgLyHVBq4q6e6NKgIwdHWrznUGkGooa4VREomdteTv_gt_-6FnHeHokMSn8A1TiZUU-6OcMFBuOhZuOEDAglzM3MUCI_FdMoKshNB3Zy_w4AmUOc24nGfUQnwQouAKatNwIsS1WXcE_X8CdKF_NcCw55-1ErnA_qpdJGsNQN1NPgt39Wqkv8qfMyeMYRwzivKNvkOekxctzS_xGBToItTWezpYia051BzDbxy73UqsT9wxoYewqfQQD7BHN8uiHJ8SzAnMZCtsVJr6r1IsA5MKz70pnXtx5C_I_hHMPRFslQFxyIV0XgmRruCUqdTf42s8q2AtaPaH6Xp-CXxF09-h1y-w8cZpBC0xkWX1XkzOHj8UuLMW9SaojfoJRTQ1ckU2fWWY3Oei9Ka-bKFXqLTNUA7cNAFjElSA1AnlHZrNSeRITxkkSzNJ-TEcxysIUXt4cZNoT5SAn3Zsu6pS154EFZDeJH7L0ZK6LCm8dLbNkv1uFHXZanK4WuxQgGfNNfqUv0iYQOrzMG2nd91dYEU8qC3ReK1B-PbZQysE3yubLmXGIuDVt0kNUq0JLtwtPgG0rczceph1WIrUX-eg4ED_v1OnNYAPtALy3kBxK6AnjYExws"
              },
              "captureError": null
            },
            "baseCapture": {
              "captureImage": {
                "imageUrl": "https://snapshots.chromatic.com/snapshots/6412fd7f8d2c0cc2cc733d99-66179b7786af2afa9eb530e0/capture-a40ed72a.png?assetToken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJyZXNvdXJjZUtleSI6IjY0MTJmZDdmOGQyYzBjYzJjYzczM2Q5OS02NjE3OWI3Nzg2YWYyYWZhOWViNTMwZTAiLCJleHAiOjE3MTUzNDA1ODV9.VuDd2qCg0D6pTOTthsRDMNfhFhwRDkuXmBVRwyZInjffhG7djaEiE-Nc3-dgOMwMFCcQY0UfHiU9Jfxcnt6wj6YJP7fmLW9Qds7mHp33DKdRMjiU6AM29pURKiaUUv5tJVUsOPvQlWgI7PTTzxhXZt2NW0jkQZRtmtOrnyTSmVnmkKjZT2Y1P8nvheXouCSIHSDELIKNbRwBqfDD5EYgxSJxXn6_h6Gb0o314IQjmMkyB8N-rZHqw5oHRmtz3buCMbKvGwoB4qPEFA6EtmlhwyuCx4Y148ClphbDTcW8D5j9Both0x98Wx1MrtPVJP16_GxLYG8Co6YxKbxGkVpbfV2ukznLZUsGz3Yb-NRRdb1SDGi6-iJtrcT8Y1tjePCA8-OwphYfqRZamenu5qJjnTIsElUTs9Ilk08fECzD_x3ACiBfxCmkYtug-6XhBsExctRQt-ZjOtnyC5U7QjJAfrXJz2U-_p6c8b5-w1waJTwybUYMhN5pqF-ybDEJj1i4xVE8BRRnIQEOIc5XenK_KKDr6_tFL9Q3YQIYQctyuDRbK5UJ3opgFmqixa2UCm36uJQYstk2sMlKFRZau2I6BzColIRdqtnEA_ClyfY9uALaodoYXN0lR4tkkoDYEd_WdcUgBjTTRA14zh1yRp5_eL0Xw92z7n5yG04iVoxgcIA",
                "imageWidth": 77
              }
            }
          }
        ],
        "mode": {
          "name": "1200px",
          "globals": {}
        },
        "story": {
          "storyId": "beta-button--default",
          "name": "Default",
          "component": {
            "name": "Button"
          }
        }
      }
    ]
  }
}

As you can see branch in response contain owner name (fuzebox-team:feat/FBI-169). So now when addon compare branch from build and branch from local context is mismatched because local context doesn't have prefixed branch name. Original code.

Fix for this would be to: a) remove owner name from branch name returned by query b) add owner name to local context branch name when comparing with selectedBuild.branch

@tomaboro Thanks for getting to the bottom of it and spelling it out. I totally misunderstood the problem, as I thought y'all we're using some strange branch names. As you explained it, it is clearly a bug. I'll address it ASAP.

@tomaboro I've put up #299 to address this issue. Looking at that build you shared (1047), it was marked as originating from a fork. This happens when the repository owner of the locally linked Git repository ("fuzebox-team") does not match that of the Git repository linked to the Chromatic project (which includes an organization prefix in your case).

Interestingly, other local builds (e.g. 1046) didn't have this issue. Both appear to originate from you. Did you change any local Git config between these builds?

While #299 should fix the VTA for forked repositories, I suspect you may still run into the same problem as your local environment may specify a mismatching owner name and thus only find builds from that same owner name rather than your regular (CI) builds. If this happens, please reach out to us on Intercom and provide the return value of running git config --get remote.origin.url in your local Git repository (as you may not want to share it here publicly).

You may try @chromatic-com/storybook@1.3.5--canary.5c97177.0 (from #299) to see if it resolves the problem.

Interestingly, other local builds (e.g. 1046) didn't have this issue. Both appear to originate from you. Did you change any local Git config between these builds?

We have been experimenting with setting local variables and CLI flags so it's probably one of experimental builds. I wouldn't pay too much attention to it.

While https://github.com/chromaui/addon-visual-tests/pull/299 should fix the VTA for forked repositories, I suspect you may still run into the same problem as your local environment may specify a mismatching owner name and thus only find builds from that same owner name rather than your regular (CI) builds. If this happens, please reach out to us on Intercom and provide the return value of running git config --get remote.origin.url in your local Git repository (as you may not want to share it here publicly).
You may try @chromatic-com/storybook@1.3.5--canary.5c97177.0 (from https://github.com/chromaui/addon-visual-tests/pull/299) to see if it resolves the problem.

I've tried linked build and everything works perfectly. We're eagerly waiting for the release now - thank you for you help 🙏

P.S. Big kudos to everyone involved into support process - It was very smooth and responsive from the start to the end ❤️

chromaui / addon-visual-tests