willsuto
commented
3 weeks ago Thanks so much for opening this issue! I think you've hit on two salient points here:
1) Flake Guard currently only works with test suites using the Jest framework. In the Swell repo, the unit tests that you were able to run successfully with flake-guard use the Jest framework. The other tests that weren't working use other test-running frameworks, so they are not compatible with flake-guard currently. This is something we would like to address going forward by expanding the existing FlakeGuard logic to accommodate other testing frameworks, particularly popular E2E testing tools such as Playwright and Cypress, especially since the types of tests run in these frameworks tend to be the ones most prone to test flake.
2) I think we can strive for better documentation and error handling in the NPM package to make the type of issue you encountered easier to understand.
Thank you so much for using FlakeGuard and providing feedback! We truly appreciate it and will always welcome any other feedback or contributions.
Description
When attempting to run FlakeGuard on integration tests for the Swell Electron application, I consistently receive a "SyntaxError: Unexpected end of JSON input" error. This occurs for all integration tests, while unit tests run successfully with FlakeGuard.
Thanks in advance for your help (as I know Y'all are NOT Flakes!)
Reproduction
https://github.com/open-source-labs/Swell
System information
System: OS: macOS 14.5 CPU: (8) arm64 Apple M3 Memory: 46.84 MB / 16.00 GB Shell: 5.9 - /bin/zsh Binaries: Node: 16.15.0 - ~/.nvm/versions/node/v16.15.0/bin/node npm: 8.19.4 - ~/Dev/Codesmith/GitHub/Others-Repos/Swell_15_04/node_modules/.bin/npm Browsers: Chrome: 127.0.6533.120
Additional information
Things Ive tried:
I have tried using nvm 16, 18, 19, 22
Running integration tests directly with Mocha also fails, which might suggests that the issue might be related to how the integration tests are set up or how they're interacting with the Electron environment, rather than a simple configuration issue with FlakeGuard.:
➜ test git:(master) ✗ npx flake-guard "npx mocha ./test/IntegrationTests/httpIntegrationTests.js" Number of runs: 10 Error in run number 1: SyntaxError: Unexpected end of JSON input Error in run number 2: SyntaxError: Unexpected end of JSON input Error in run number 3: SyntaxError: Unexpected end of JSON input Error in run number 4: SyntaxError: Unexpected end of JSON input Error in run number 5: SyntaxError: Unexpected end of JSON input Error in run number 6: SyntaxError: Unexpected end of JSON input Error in run number 7: SyntaxError: Unexpected end of JSON input Error in run number 8: SyntaxError: Unexpected end of JSON input Error in run number 9: SyntaxError: Unexpected end of JSON input Error in run number 10: SyntaxError: Unexpected end of JSON input Total FlakeGuard runtime: 3.666 seconds Results Summary: {} Results successfully sent to FlakeGuard server Press Enter to open results in the dashboard, or Ctrl+C to exit...
Link to integration tests: https://github.com/open-source-labs/Swell/tree/f1d1f71c4bffc46cb8963d71b010e256e6ff2c43/test
Pic of terminal with successful and failing tests:
package.json (from beginning til not pertinent): { "name": "swell", "version": "1.18.0", "description": "Swell", "main": "main.js", "repository": "https://github.com/open-source-labs/Swell", "scripts": { "report": "istanbul report --dir ./test/coverage/total-coverage html", "test": "jest && webpack --mode=production --config ./webpack.production.js && cross-env process.env.NODE_ENV=test nyc --report-dir ./test/coverage/mocha-coverage --reporter json --reporter text --reporter html mocha --timeout 15000 --exit", "server-sse": "node ./test/SSE_HTTP1_server.js", "server-gql": "node ./test/graphqlServer.mjs", "server-grpc": "node ./test/grpcServer.js", "server-http": "node ./test/httpServer.js", "server-http2": "node ./test/HTTP2_server.js", "server-trpc": "ts-node ./test/tRPC_Test_Servers/Express_Test_Server/server/api.ts", "server-websocket": "node ./test/websocketServer.js", "server-webrtc": "node ./test/webrtcWSServer.js", "test-jest": "jest", "test-mocha": "webpack --mode=production --config ./webpack.production.js && cross-env process.env.NODE_ENV=test nyc --report-dir ./test/coverage/mocha-coverage --reporter json --reporter text --reporter html mocha --timeout 15000 --exit", "test-mocha-zero": "webpack --mode=production --config ./webpack.production.js && cross-env process.env.NODE_ENV=test mocha --timeout 0 --exit", "format": "prettier --write \"*/.+(js|jsx| tsx| json|css|md)\"", "lint": "eslint .", "lint:fix": "eslint --fix . ", "dev": "concurrently --success first \"webpack-dev-server --mode=development --config ./webpack.development.js\" \"nodemon --legacy-watch ./src/server/server.js\"", "build": "concurrently --success first \"webpack --mode=production --config ./webpack.production.js\"", "prod": "concurrently --success first \"webpack --mode=production --config ./webpack.production.js && electron .\"", "package-mac": "webpack --mode=production --config ./webpack.production.js && electron-builder build --x64 --mac", "package-all": "webpack --mode=production --config ./webpack.production.js && electron-builder build -mwl", "package-win": "webpack --mode=production --config ./webpack.production.js && electron-builder build --x64 --win", "package-linux": "webpack --mode=production --config ./webpack.production.js && electron-builder build --linux", "gh-publish-mac": "electron-builder build --x64 --mac -p always", "gh-publish": "electron-builder build -mwl -p always", "check-types": "tsc", "postinstall": "patch-package" }, "build": { "npmRebuild": false, "productName": "Swell", "appId": "com.getswell.Swell.app", "extends": null, "files": [ "dist/", "menu/mainMenu.js", "package.json", "main.js", "main_process/", "preload.js", "src/server/*" ], "nodeVersion": "16.15.0", "nsis": { "createDesktopShortcut": "always" },
👨👧👦 Contributing