Chromatic visual testing not failing Azure pipeline on UI changes

[MAlli-RSAC]

Description

We've integrated Chromatic for visual testing in our Azure pipeline, but it's not failing the build when UI changes are detected. The pipeline continues to pass even when there are visible changes in the components. We're not sure if this is a bug or configuration issue.

What We've Tried

• [x] Added --exit-zero-on-changes flag to Chromatic command
• [x] Implemented a separate step to check Chromatic results
• [x] Attempted to fail the pipeline using exit 1 when changes are detected
• [x] Verified that UI changes are visible in Chromatic dashboard
• [x] Checked Azure pipeline logs for any error messages or warnings

Current Behavior

• Chromatic runs successfully in the pipeline
• UI changes are visible in the Chromatic dashboard
• Azure pipeline passes despite these changes

Expected Behavior

• Chromatic detects UI changes
• Azure pipeline fails when Chromatic detects changes
• Requires manual review and approval before the PR can be merged

Steps to Reproduce

1. Make a visible UI change in a component (e.g., change background color, add padding)
2. Commit and push changes to a feature branch
3. Create a Pull Request
4. Observe that the Azure pipeline passes despite UI changes

Code Snippets

Current Configuration

trigger:
  - main
variables:
  - group: project-variables
pr:
  - '*'
pool:
  vmImage: 'ubuntu-latest'
steps:
  - checkout: self
    fetchDepth: 0
  - script: sed -i 's/\${PROJECT_XX_XXXXX}/$(PROJECT_XX_XXXXX)/' .npmrc
    displayName: 'Inject PROJECT_XX_XXXXX into .npmrc'
  - script: |
      echo "##vso[task.setvariable variable=nodeVersion]$(cat .nvmrc)"
    displayName: 'Read Node.js version from .nvmrc'
  - task: NodeTool@0
    inputs:
      versionSpec: $(nodeVersion)
    displayName: 'Install Node.js'
  - script: npm ci
    displayName: 'Install dependencies'
  - script: npm run storybook:build
    displayName: 'Build Storybook'
  # Publish to Chromatic
  - script: npm run chromatic:publish -- --project-token=$(CHROMATIC_PROJECT_TOKEN)
    displayName: 'Publish to Chromatic'
    env:
      CHROMATIC_PROJECT_TOKEN: $(CHROMATIC_PROJECT_TOKEN)
  # Run Chromatic for visual testing on pull requests
  - task: CmdLine@2
    displayName: 'Run Chromatic for visual testing'
    inputs:
      script: npx chromatic ##--exit-zero-on-changes
    env:
      CHROMATIC_PROJECT_TOKEN: $(CHROMATIC_PROJECT_TOKEN)
    condition: eq(variables['Build.Reason'], 'PullRequest')
    # This step runs Chromatic and continues even if visual changes are detected
  # Require approval for visual changes
  - task: CmdLine@2
    displayName: 'Require approval for visual changes'
    inputs:
      script: |
        if [ -f chromatic-report.json ]; then
          changes=$(jq '.changes' chromatic-report.json)
          if [ "$changes" -gt 0 ]; then
            echo "Visual changes detected. Please review and approve."
            exit 1
          fi
        fi
    condition: eq(variables['Build.Reason'], 'PullRequest')
    # This step checks the Chromatic report and requires approval if changes are detected

Additional Information:

• Chromatic version: 11.5.6
• Azure Pipelines version: 1.0.1
• Node.js version: v20.12.2

Questions:

1. Is there a specific environment variable or output from Chromatic that we should be checking?
2. Are there any known issues with integrating Chromatic and Azure Pipelines?
3. Is there a recommended way to force pipeline failure when Chromatic detects changes?

Any help you can provide to help us get this working would be greatly appreciated.

[andresilva-yld]

I would like to understand this as well. Very surprised to see the GH action not failing with unreviewed changes. Aside from that, it's also problematic that the GH action doesn't return any build data when you re-run a workflow.