Option to return `exit code 1` on failure for CI integration

leanrob commented 4 years ago

Expected Outcome

An option to force a failure in the @zeplin/cli 'zeplin connect' command to return exit code 1 to indicate to CI services that the zeplin connection process has failed.

Current Outcomes

Currently, when running the zeplin connect command using the cli-connect-storybook-plugin a CI tool, the @zeplin/CLI returns an exit code 0 in both error and success cases. Resulting in the CI task reporting as Successful even when there is a failure in the CLI zeplin connect command.

This return seems to be expected behaviour based on this console log https://github.com/zeplin/cli-connect-storybook-plugin/blob/c556e18a2a7333193f00e5e7c3e08f2314d777af/src/storybook/stories.js#L90

However, this is leading to silent failures in CI.

Relevant Example

The following is an example of code from a config.yml file for CircleCI:

zeplin:
    working_directory: ~/client

    docker:
      - image: cypress/base:12.14.0
        environment:
          TERM: xterm

    steps:
      - attach_workspace:
          at: ~/
      - run:
          name: Install Zeplin CLI & Plugins
          command: npm install -g @zeplin/cli @zeplin/cli-connect-react-plugin @zeplin/cli-connect-storybook-plugin
      - run:
          name: Deploy connected components to Zeplin & connect storybook
          command: zeplin connect —verbose

This script works, but when a deliberate error is introduced to test the failure state, the output from CircleCI reads:

zeplin connect 

Zeplin CLI - v1.0.2

Detected Storybook at [REDACTED]/iframe.html

The following problems were reported from your storybook:
Warnings:
=========================
Warning: React.createFactory() is deprecated and will be removed in a future major release. Consider using JSX or use React.createElement() directly instead.
=========================
This may lead to some stories not working right or getting detected by Chromatic
We suggest you fix the errors, but we will continue anyway..
=========================
Loaded 282 stories from Storybook.
Connecting components to Zeplin components failed.
    Error occurred while processing src/js/components/common/Button/Button.jsx with @zeplin/cli-connect-react-plugin:
    /usr/local/lib/node_modules/@zeplin/cli-connect-react-plugin/dist/template/base.pug:18
        16| 
        17| mixin propType(name, propType)
      > 18|     if propType.name === "union"
        19|         |   #{name}={union[
        20|         +propertyValues(propType.value)
        21|         | }]

    Cannot read property 'name' of undefined
Please check ~/.zeplin/cli.log for details.

CircleCI received exit code 0

Which return exit code 0 indicating to CircleCI a successful step, leading to silent failures of Zeplin connected components.

An option to return exit code 1 on Storybook connection errors would allow CI tools to monitor the success/failure of automated Zeplin-Storybooks connected components.

yuqu commented 4 years ago

Hi Robert,

CLI should have return exit code 1 on errors. I've realized a simple trick I put on winston logger (to ensure all debug logs are written to the file before exit) somehow causes the process to return with exit code 0. I'm investigating to fix it.

On a side note, Storybook errors you mentioned was actually happening inside the Storybook instance. The way we consume is just to start it up and load it into JSDOM, to retrieve information about the stories inside. That's why the errors are ignored since those are in scope of Storybook (even their own chromatic-cli handle it this way). However, I think that having an option to count it as an error to return exit code 1 would be helpful on some cases, so I've added failFastOnErrors option https://github.com/zeplin/cli-connect-storybook-plugin/pull/22 and it is available in v0.2.0. Still, the problem in @zeplin/cli should be fixed before you can use it.

yuqu commented 4 years ago

v1.0.3 fixes exit code issues.

As I mentioned previously. You can also use the following configuration to terminate the process with exit code 1 on CI builds if any errors or warning are reported from Storybook frontend.

{
    "plugins": [
        {
            "name": "@zeplin/cli-connect-storybook-plugin",
            "config": {
                "url": "http://localhost:9009",
                "startScript": "storybook",
                "failFastOnErrors": true
            }
        }
    ]
}

zeplin / cli