Percy snapshot command returns exit code 0 when build fails

IanVS commented 1 year ago

The problem

When my percy snapshots fail due to running out of credits or some other reason, my CI does not alert me, because the exit code returned is 0.

For example:

▶ npx percy snapshot dist/blog
[percy] Percy has started!
[percy] Snapshot taken: /internationalizing-docs-pages-with-astro/index.html
[percy] Failed to create build
[percy] Error: Missing Percy token
[percy] Build not created

▶ echo $?
0

You can see that the build failed (in this case because it couldn't find a token), but the process returned with an exit code of 0, which is supposed to mean everything worked correctly.

Environment

Node version: 16.16.0
@percy/cli version: 1.18.0
Version of Percy SDK you’re using:
If needed, a build or snapshot ID:
OS version:
Type of shell command-line [interface]: zsh

Details

A similar issue is discussed in https://github.com/percy/cli/discussions/539, but the fix there only involved Cypress.

Debug logs

I tried not to trim this, but GitHub gave me an error (There was an error creating your Issue: body is too long (maximum is 65536 characters).), so I trimmed out some of the middle.

Debug logs

[percy:config] Found config file: .percy.yml (0ms) [percy:config] Using config: { version: 2, upload: { files: '**/*.{png,jpg,jpeg,avif,webp}', ignore: '' }, snapshot: { widths: [ 375, 1280 ], minHeight: 1024, percyCSS: '' }, discovery: { networkIdleTimeout: 750, disableCache: true } } (19ms) [percy:core:browser] Launching browser (25ms) [percy:core:browser] Browser connected [23778]: HeadlessChrome/96.0.4664.0 (390ms) [percy:core] Percy has started! (1ms) [percy:core:snapshot] --------- (78ms) [percy:core:snapshot] Received snapshot: /index.html (0ms) [percy:core:snapshot] - url: http://localhost:62602/index.html (1ms) [percy:core:snapshot] - widths: 375px, 1280px (0ms) [percy:core:snapshot] - minHeight: 1024px (0ms) [percy:core:snapshot] - discovery.allowedHostnames: localhost (0ms) [percy:core:snapshot] - discovery.disableCache: true (0ms) [percy:core:snapshot] --------- (1ms) [percy:core:discovery] - Skipping remote resource (0ms) [percy:core:discovery] Processing resource: https://stats.g.doubleclick.net/j/collect?t=dc&aip=1&_r=3&v=1&_v=j99&tid=UA-179012409-1&cid=89141066.1675434506&jid=1787100395&gjid=822920426&_gid=1002200174.1675434506&_u=YEBAAUAAAAAAACAAIC~&z=535542665 (57ms) [percy:core:discovery] - Skipping remote resource (1ms) [percy:core:discovery] Handling request: https://www.google.com/ads/ga-audiences?t=sr&aip=1&_r=4&slf_rd=1&v=1&_v=j99&tid=UA-179012409-1&cid=89141066.1675434506&jid=1787100395&_u=YEBAAUAAAAAAACAAIC~&z=1918529209 (0ms) [percy:core:discovery] Processing resource: https://www.google.com/ads/ga-audiences?t=sr&aip=1&_r=4&slf_rd=1&v=1&_v=j99&tid=UA-179012409-1&cid=89141066.1675434506&jid=1787100395&_u=YEBAAUAAAAAAACAAIC~&z=1918529209 (112ms) [percy:core:discovery] - Skipping remote resource (0ms) [percy:core:page] Inject @percy/dom (504ms) [percy:core:page] Serialize DOM (5ms) [percy:core:page] Inject @percy/dom (4ms) [percy:core:page] Serialize DOM (4ms) [percy:core:discovery] Wait for 750ms idle (2ms) [percy:core:page] Inject @percy/dom (1ms) [percy:core:page] Inject @percy/dom (0ms) [percy:core:discovery] Wait for 750ms idle (3ms) [percy:core:page] Serialize DOM (0ms) [percy:core:page] Serialize DOM (0ms) [percy:core:page] Inject @percy/dom (1ms) [percy:core:page] Serialize DOM (2ms) [percy:core:discovery] Wait for 750ms idle (4ms) [percy:core:discovery] Wait for 750ms idle (2ms) [percy:core:discovery] Wait for 750ms idle (2ms) [percy:core:page] Inject @percy/dom (48ms) [percy:core:page] Inject @percy/dom (0ms) [percy:core:page] Inject @percy/dom (0ms) [percy:core:page] Inject @percy/dom (0ms) [percy:core:page] Serialize DOM (3ms) [percy:core:page] Serialize DOM (0ms) [percy:core:page] Serialize DOM (0ms) [percy:core:page] Serialize DOM (1ms) [percy:core:discovery] Wait for 750ms idle (6ms) [percy:core:discovery] Wait for 750ms idle (1ms) [percy:core:discovery] Wait for 750ms idle (1ms) [percy:core:discovery] Wait for 750ms idle (2ms) [percy:core:page] Inject @percy/dom (156ms) [percy:core:page] Serialize DOM (2ms) [percy:core:discovery] Wait for 750ms idle (7ms) [percy:core:page] Resize page to 1280x1024 @1x (508ms) [percy:core:page] Resize page to 1280x1024 @1x (4ms) [percy:core:page] Taking snapshot: /introducing-oidc-single-sign-on-for-admins/index.html @1280px (2ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/defined-og_kkrsT.avif (1ms) [percy:core:page] Taking snapshot: /introduction-to-defined-networking/index.html @1280px (0ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:page] Resize page to 1280x1024 @1x (4ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/defined-og_kkrsT.avif (0ms) [percy:core:discovery] - Skipping disallowed status [404] (1ms) [percy:core:page] Taking snapshot: /blocklisting/index.html @1280px (1ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:page] Resize page to 1280x1024 @1x (1ms) [percy:core:page] Resize page to 1280x1024 @1x (2ms) [percy:core:page] Taking snapshot: /index.html @1280px (0ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/defined-og_kkrsT.avif (9ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/road-gate_ZsW3x0.avif (1ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/library_1Jzvth.avif (6ms) [percy:core:page] Taking snapshot: /internationalizing-docs-pages-with-astro/index.html @1280px (0ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/defined-og_kkrsT.avif (2ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/road-gate_ZsW3x0.avif (0ms) [percy:core:discovery] - Skipping disallowed status [404] (1ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/library_1Jzvth.avif (3ms) [percy:core:discovery] - Skipping disallowed status [404] (1ms) [percy:core:discovery] - Skipping disallowed status [404] (2ms) [percy:core:page] Resize page to 1280x1024 @1x (32ms) [percy:core:page] Resize page to 1280x1024 @1x (2ms) [percy:core:page] Taking snapshot: /announcing-relay-support-in-nebula/index.html @1280px (0ms) [percy:core:discovery] Wait for 750ms idle (1ms) [percy:core:page] Resize page to 1280x1024 @1x (0ms) [percy:core:page] Resize page to 1280x1024 @1x (1ms) [percy:core:page] Taking snapshot: /newsletter-advanced-firewall-and-audit-logs/index.html @1280px (0ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:page] Taking snapshot: /newsletter-admin-api-cert-rotation-multiple-lighthouses/index.html @1280px (1ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:page] Taking snapshot: /relays-are-in-dn-today/index.html @1280px (1ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/radio-dishes_gicLf.avif (35ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/radio-dish-hero_Z1NeKLk.avif (2ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/auditlogs_29ls4A.avif (1ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/lighthouse_1JKmcn.avif (5ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/radio-dish-hero_Z1NeKLk.avif (6ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/radio-dishes_gicLf.avif (1ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/auditlogs_29ls4A.avif (1ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/lighthouse_1JKmcn.avif (1ms) [percy:core:discovery] - Skipping disallowed status [404] (0ms) [percy:core:discovery] - Skipping disallowed status [404] (1ms) [percy:core:discovery] - Skipping disallowed status [404] (0ms) [percy:core:discovery] - Skipping disallowed status [404] (0ms) [percy:core:page] Resize page to 1280x1024 @1x (110ms) [percy:core:page] Taking snapshot: /open-for-business/index.html @1280px (3ms) [percy:core:discovery] Wait for 750ms idle (0ms) [percy:core:discovery] Handling request: http://localhost:62602/assets/open_jieyQ.avif (27ms) [percy:core:discovery] Processing resource: http://localhost:62602/assets/open_jieyQ.avif (17ms) [percy:core:discovery] - Skipping disallowed status [404] (1ms) [percy:core:page] Serialize DOM (467ms) [percy:core:page] Page closed (9ms) [percy:core:page] Serialize DOM (1ms) [percy:core:page] Serialize DOM (0ms) [percy:core:page] Serialize DOM (1ms) [percy:core:page] Page closed (9ms) [percy:core:page] Page closed (1ms) [percy:core:page] Page closed (2ms) [percy:core:page] Serialize DOM (14ms) [percy:core:page] Page closed (12ms) [percy:core:page] Serialize DOM (21ms) [percy:core:page] Serialize DOM (1ms) [percy:core:page] Serialize DOM (1ms) [percy:core:page] Serialize DOM (0ms) [percy:core:page] Page closed (5ms) [percy:core:page] Page closed (4ms) [percy:core:page] Page closed (2ms) [percy:core:page] Page closed (4ms) [percy:core:page] Serialize DOM (152ms) [percy:core:page] Page closed (7ms) [percy:core:browser] Closing browser (2ms) [percy:core:browser] Browser closed (40ms) [percy:core] Build not created (0ms)

Code to reproduce issue

N/A

itsjwala commented 1 year ago

Hi @IanVS 👋

I could see the build was not created, and thus build "fails" won't be correct.
- If a build was created then you can use build:wait ref, it outputs useful information which you may process further.
Currently, it's by design that exit codes are managed this way, we'll evaluate the need for this in the future.
Though as a workaround, for this issue you may parse this log message and handle exits on your end.

IanVS commented 1 year ago

Hm, but the point of the command is to perform a build and snapshot it, so how is failing to perform a build a successful outcome? :-/

The use case I really want to enable is to fail my CI when my snapshots run out or there's another problem. How would you suggest handling it? Does Build not created get logged out in all situations, and I should fail my CI based on that?

itsjwala commented 1 year ago

@IanVS

In current terminologies, the build is generally not created when PERCY_TOKEN is not passed, or the API has failed to acknowledge the request.
- so Build not created log line can be used for this scenario.
for other scenarios where the build was created and for any reason didn't succeed can be evaluated usingbuild:wait ref.

percy / cli