search

EnricoMi / publish-unit-test-result-action

GitHub Action to publish unit test results on GitHub
Apache License 2.0
625 stars 183 forks source link

Unhandled error that crashes GitHub Actions - .NET with coverlet.cobertura #552

Closed kamil-oberaj closed 10 months ago

kamil-oberaj commented 11 months ago

Please note that all information belongs to private repo and I cannot share anymore private information (I'll do whatever I can to help resolving this issue)

Affected versions (the one in use): 5.1.24 & 5.2.0

First occurrence: Around last week

Action steps:

  1. Run action on self-hosted-ubuntu-latest
  2. Checkout repository with actions/checkout@3
  3. Setup .NET 6 with temp directory /tmp/dotnet-install
  4. Restore dependencies using dotnet restore
  5. Build solution with --configuration Release --no-restore
  6. (additionally, but works with and without) Publish solution with -c Release
  7. Use dotnet test to run tests using --filter 'FullyQualifiedName!~XXX.Specs' --settings coverage.settings --results-directory "TestResults" --collect:"XPlat Code Coverage" --logger "trx;LogFilePrefix=TestResults.trx" --no-build --configuration Release
  8. Generate test report using danielpalme/ReportGenerator-GitHub-Action@5.1.24 and save it to TestResults directory
  9. Write down the report to action summary
  10. Upload the test results using actions/upload-artifact@v3
  11. Publish the results using EnricoMi/publish-unit-test-result-action@v2 with trx_files: "TestResults/**/*.trx"

The result of broken action is:

The self-hosted runner: xxx-runner-ubuntu-2204-xxxxx-xxxx lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Logs

2024-01-02T08:12:33.7179043Z Artifact TestResults has been successfully uploaded!
2024-01-02T08:12:33.7418913Z ##[group]Run EnricoMi/publish-unit-test-result-action@v2
2024-01-02T08:12:33.7419515Z with:
2024-01-02T08:12:33.7419826Z   trx_files: TestResults/**/*.trx
2024-01-02T08:12:33.7420461Z   github_token: ***
2024-01-02T08:12:33.7420828Z   github_token_actor: github-actions
2024-01-02T08:12:33.7421274Z   github_retries: 10
2024-01-02T08:12:33.7421627Z   check_name: Test Results
2024-01-02T08:12:33.7422003Z   comment_mode: always
2024-01-02T08:12:33.7422360Z   fail_on: test failures
2024-01-02T08:12:33.7422721Z   action_fail: false
2024-01-02T08:12:33.7423083Z   action_fail_on_inconclusive: false
2024-01-02T08:12:33.7423526Z   time_unit: seconds
2024-01-02T08:12:33.7423879Z   report_suite_logs: none
2024-01-02T08:12:33.7424248Z   ignore_runs: false
2024-01-02T08:12:33.7424581Z   check_run: true
2024-01-02T08:12:33.7424914Z   job_summary: true
2024-01-02T08:12:33.7425265Z   compare_to_earlier_commit: true
2024-01-02T08:12:33.7425694Z   pull_request_build: merge
2024-01-02T08:12:33.7426198Z   check_run_annotations: all tests, skipped tests
2024-01-02T08:12:33.7426695Z   seconds_between_github_reads: 0.25
2024-01-02T08:12:33.7427127Z   seconds_between_github_writes: 2.0
2024-01-02T08:12:33.7427591Z   json_thousands_separator:  
2024-01-02T08:12:33.7427988Z   json_suite_details: false
2024-01-02T08:12:33.7428364Z   json_test_case_results: false
2024-01-02T08:12:33.7428758Z   search_pull_requests: false
2024-01-02T08:12:33.7429122Z env:
2024-01-02T08:12:33.7429429Z   DOTNET_INSTALL_DIR: /tmp/dotnet-install
2024-01-02T08:12:33.7429888Z   DOTNET_ROOT: /tmp/dotnet-install
2024-01-02T08:12:33.7430284Z ##[endgroup]
2024-01-02T08:12:33.7739589Z ##[command]/usr/local/bin/docker run --name ghcrioenricomipublishunittestresultactionv2120_9172df --label 285772 --workdir /github/workspace --rm -e "DOTNET_INSTALL_DIR" -e "DOTNET_ROOT" -e "INPUT_TRX_FILES" -e "INPUT_GITHUB_TOKEN" -e "INPUT_GITHUB_TOKEN_ACTOR" -e "INPUT_GITHUB_RETRIES" -e "INPUT_COMMIT" -e "INPUT_CHECK_NAME" -e "INPUT_COMMENT_TITLE" -e "INPUT_COMMENT_MODE" -e "INPUT_FAIL_ON" -e "INPUT_ACTION_FAIL" -e "INPUT_ACTION_FAIL_ON_INCONCLUSIVE" -e "INPUT_FILES" -e "INPUT_JUNIT_FILES" -e "INPUT_NUNIT_FILES" -e "INPUT_XUNIT_FILES" -e "INPUT_TIME_UNIT" -e "INPUT_TEST_FILE_PREFIX" -e "INPUT_REPORT_INDIVIDUAL_RUNS" -e "INPUT_REPORT_SUITE_LOGS" -e "INPUT_DEDUPLICATE_CLASSES_BY_FILE_NAME" -e "INPUT_LARGE_FILES" -e "INPUT_IGNORE_RUNS" -e "INPUT_CHECK_RUN" -e "INPUT_JOB_SUMMARY" -e "INPUT_COMPARE_TO_EARLIER_COMMIT" -e "INPUT_PULL_REQUEST_BUILD" -e "INPUT_EVENT_FILE" -e "INPUT_EVENT_NAME" -e "INPUT_TEST_CHANGES_LIMIT" -e "INPUT_CHECK_RUN_ANNOTATIONS" -e "INPUT_CHECK_RUN_ANNOTATIONS_BRANCH" -e "INPUT_SECONDS_BETWEEN_GITHUB_READS" -e "INPUT_SECONDS_BETWEEN_GITHUB_WRITES" -e "INPUT_SECONDARY_RATE_LIMIT_WAIT_SECONDS" -e "INPUT_JSON_FILE" -e "INPUT_JSON_THOUSANDS_SEPARATOR" -e "INPUT_JSON_SUITE_DETAILS" -e "INPUT_JSON_TEST_CASE_RESULTS" -e "INPUT_SEARCH_PULL_REQUESTS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_EVENT_PATH" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "GITHUB_ACTION" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/runner/_work/_temp/_github_home":"/github/home" -v "/runner/_work/_temp/_github_workflow":"/github/workflow" -v "/runner/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/runner/_work/XXX-XXX-XXX-XXX/XXX-XXX-XXX-XXX":"/github/workspace" ghcr.io/enricomi/publish-unit-test-result-action:v2.12.0
2024-01-02T08:12:35.2114486Z 2024-01-02 08:12:35 +0000 - publish -  INFO - Available memory to read files: 23.1 GiB
2024-01-02T08:12:35.2122313Z 2024-01-02 08:12:35 +0000 - publish -  INFO - Reading TRX files TestResults/**/*.trx (7 files, 1.7 MiB)
2024-01-02T08:12:35.2950239Z 2024-01-02 08:12:35 +0000 - publish -  INFO - Finished reading 7 files in 0.08 seconds
2024-01-02T08:12:35.6011761Z Traceback (most recent call last):
2024-01-02T08:12:35.6012543Z   File "/action/publish_test_results.py", line 546, in <module>
2024-01-02T08:12:35.6013213Z     main(settings, gha)
2024-01-02T08:12:35.6013721Z   File "/action/publish_test_results.py", line 269, in main
2024-01-02T08:12:35.6014575Z     Publisher(settings, gh, gha).publish(stats, results.case_results, conclusion)
2024-01-02T08:12:35.6016746Z   File "/action/publish/publisher.py", line 189, in __init__
2024-01-02T08:12:35.6017785Z     self._repo = gh.get_repo(self._settings.repo)
2024-01-02T08:12:35.6019113Z   File "/usr/local/lib/python3.8/site-packages/github/MainClass.py", line 380, in get_repo
2024-01-02T08:12:35.6020315Z     headers, data = self.__requester.requestJsonAndCheck("GET", url)
2024-01-02T08:12:35.6021637Z   File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 494, in requestJsonAndCheck
2024-01-02T08:12:35.6022965Z     return self.__check(*self.requestJson(verb, url, parameters, headers, input, self.__customConnection(url)))
2024-01-02T08:12:35.6024288Z   File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 525, in __check
2024-01-02T08:12:35.6025222Z     raise self.createException(status, responseHeaders, data)
2024-01-02T08:12:35.6027582Z github.GithubException.BadCredentialsException: 401 {"message": "Bad credentials", "documentation_url": "https://docs.github.com/rest"}
2024-01-02T08:12:35.8975087Z Post job cleanup.
EnricoMi commented 11 months ago

You say this first occurred last week, did this ever run before? Can you provide the logs of the last time this worked?

You say this is a private repo, is this hosted at github.com or in an enterprise environment? In the latter case, can you check that environment variable GITHUB_API_URL is set to your enterprise URL?

The error indicates that the token given to input github_token is not authenticated for your repo. This is unrelated to the setup that creates the test result files (.Net).

EnricoMi commented 11 months ago

Also strange that the failing action / job kills your self-hosted runner.

kamil-oberaj commented 11 months ago

Okay, so I've checked and it occurred 2 months ago and it is not fixed till this day.

  1. It is enterprise env
  2. GITHUB_TOKEN is 100% valid as my org is taking care of it
  3. GITHUB_API_URL does not exist in our global/private variables (it did not exist, even when the workflow worked)

Valid run logs:

2023-11-19T00:23:54.8411367Z Artifact TestResults has been successfully uploaded!
2023-11-19T00:23:54.8647559Z ##[group]Run EnricoMi/publish-unit-test-result-action@v2
2023-11-19T00:23:54.8647989Z with:
2023-11-19T00:23:54.8648216Z   trx_files: TestResults/**/*.trx
2023-11-19T00:23:54.8648799Z   github_token: ***
2023-11-19T00:23:54.8649063Z   github_token_actor: github-actions
2023-11-19T00:23:54.8649369Z   github_retries: 10
2023-11-19T00:23:54.8649617Z   check_name: Test Results
2023-11-19T00:23:54.8649881Z   comment_mode: always
2023-11-19T00:23:54.8650126Z   fail_on: test failures
2023-11-19T00:23:54.8650374Z   action_fail: false
2023-11-19T00:23:54.8650632Z   action_fail_on_inconclusive: false
2023-11-19T00:23:54.8650924Z   time_unit: seconds
2023-11-19T00:23:54.8651161Z   report_suite_logs: none
2023-11-19T00:23:54.8651414Z   ignore_runs: false
2023-11-19T00:23:54.8651643Z   job_summary: true
2023-11-19T00:23:54.8651889Z   compare_to_earlier_commit: true
2023-11-19T00:23:54.8652181Z   pull_request_build: merge
2023-11-19T00:23:54.8652536Z   check_run_annotations: all tests, skipped tests
2023-11-19T00:23:54.8652896Z   seconds_between_github_reads: 0.25
2023-11-19T00:23:54.8653204Z   seconds_between_github_writes: 2.0
2023-11-19T00:23:54.8653541Z   json_thousands_separator:  
2023-11-19T00:23:54.8653825Z   json_suite_details: false
2023-11-19T00:23:54.8654097Z   json_test_case_results: false
2023-11-19T00:23:54.8654386Z   search_pull_requests: false
2023-11-19T00:23:54.8654642Z env:
2023-11-19T00:23:54.8654870Z   DOTNET_INSTALL_DIR: /tmp/dotnet-install
2023-11-19T00:23:54.8655190Z   DOTNET_ROOT: /tmp/dotnet-install
2023-11-19T00:23:54.8655638Z ##[endgroup]
2023-11-19T00:23:54.8879634Z ##[command]/usr/local/bin/docker run --name ghcrioenricomipublishunittestresultactionv2110_2520cc --label ba54c3 --workdir /github/workspace --rm -e "DOTNET_INSTALL_DIR" -e "DOTNET_ROOT" -e "INPUT_TRX_FILES" -e "INPUT_GITHUB_TOKEN" -e "INPUT_GITHUB_TOKEN_ACTOR" -e "INPUT_GITHUB_RETRIES" -e "INPUT_COMMIT" -e "INPUT_CHECK_NAME" -e "INPUT_COMMENT_TITLE" -e "INPUT_COMMENT_MODE" -e "INPUT_FAIL_ON" -e "INPUT_ACTION_FAIL" -e "INPUT_ACTION_FAIL_ON_INCONCLUSIVE" -e "INPUT_FILES" -e "INPUT_JUNIT_FILES" -e "INPUT_NUNIT_FILES" -e "INPUT_XUNIT_FILES" -e "INPUT_TIME_UNIT" -e "INPUT_TEST_FILE_PREFIX" -e "INPUT_REPORT_INDIVIDUAL_RUNS" -e "INPUT_REPORT_SUITE_LOGS" -e "INPUT_DEDUPLICATE_CLASSES_BY_FILE_NAME" -e "INPUT_LARGE_FILES" -e "INPUT_IGNORE_RUNS" -e "INPUT_JOB_SUMMARY" -e "INPUT_COMPARE_TO_EARLIER_COMMIT" -e "INPUT_PULL_REQUEST_BUILD" -e "INPUT_EVENT_FILE" -e "INPUT_EVENT_NAME" -e "INPUT_TEST_CHANGES_LIMIT" -e "INPUT_CHECK_RUN_ANNOTATIONS" -e "INPUT_CHECK_RUN_ANNOTATIONS_BRANCH" -e "INPUT_SECONDS_BETWEEN_GITHUB_READS" -e "INPUT_SECONDS_BETWEEN_GITHUB_WRITES" -e "INPUT_SECONDARY_RATE_LIMIT_WAIT_SECONDS" -e "INPUT_JSON_FILE" -e "INPUT_JSON_THOUSANDS_SEPARATOR" -e "INPUT_JSON_SUITE_DETAILS" -e "INPUT_JSON_TEST_CASE_RESULTS" -e "INPUT_SEARCH_PULL_REQUESTS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/runner/_work/_temp/_github_home":"/github/home" -v "/runner/_work/_temp/_github_workflow":"/github/workflow" -v "/runner/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/runner/_work/XXX-XXX-XXX-XXX/XXX-XXX-XXX-XXX":"/github/workspace" ghcr.io/enricomi/publish-unit-test-result-action:v2.11.0
2023-11-19T00:23:55.9016711Z 2023-11-19 00:23:55 +0000 - publish -  INFO - Available memory to read files: 28.1 GiB
2023-11-19T00:23:55.9026181Z 2023-11-19 00:23:55 +0000 - publish -  INFO - Reading TRX files TestResults/**/*.trx (10 files, 2.0 MiB)
2023-11-19T00:23:55.9665142Z 2023-11-19 00:23:55 +0000 - publish -  INFO - Finished reading 10 files in 0.06 seconds
2023-11-19T00:23:56.6588529Z 2023-11-19 00:23:56 +0000 - publish -  INFO - Publishing success results for commit c91e616a8bddf643afe2082aa55d084f17578429
2023-11-19T00:23:57.6523162Z 2023-11-19 00:23:57 +0000 - publish -  INFO - Created check https://github.com/XXX/XXX-XXX-XXX-XXX/runs/18818040753
2023-11-19T00:23:57.6541535Z 2023-11-19 00:23:57 +0000 - publish -  INFO - Created job summary
2023-11-19T00:23:58.9032001Z 2023-11-19 00:23:58 +0000 - publish -  INFO - There is no pull request for commit c91e616a8bddf643afe2082aa55d084f17578429
2023-11-19T00:23:59.1327647Z Post job cleanup.
EnricoMi commented 11 months ago

The env var GITHUB_API_URL is set by the GITHUB runner, try to add this step to see the environment variable:

steps:
- name: env
  run: echo $GITHUB_API_URL
  shell: bash
EnricoMi commented 11 months ago

Your second log says Created check https://github.com/XXX/XXX-XXX-XXX-XXX/runs/18818040753, so that was a public / private repo on public github.com and not an enterprise installation. Did you move to an enterprise environment in within the last to months?

kamil-oberaj commented 10 months ago

Hey, sorry for the late response - I was away on vacation. I'll provide the answers later today

EnricoMi commented 10 months ago

I saw Github runners crash when disk went full.

kamil-oberaj commented 10 months ago

Okay, so the steps You've provided shown me this (screen): image

So we're not using the GH runners, we use self-hosted runners with quite a large memory available. I am not sure if this is the .NET test runner fault, because the workflow crashes on test process, but the funny thing is that the tests are actually done and we've the tests summary generated

It might be out of memory, I'll ask devops group to confirm that

kamil-oberaj commented 10 months ago

So the runner crashed on kernel not on OOM error. The OOM error produces other response, not the one presented in this thread.

Edit: I've switched to GitHub Runners and the workflow passed. Interesting

EnricoMi commented 10 months ago
  1. Your self-hosted github runner has to be configured so that the GITHUB_API_URL points to your enterprise URL, not the public github.com.
  2. Job summary does not depend on the GITHUB_API_URL being set correctly, which is why that bit is working.
  3. The log indicates that the publish action finishes and the runner fails after the action completed, so I reckon this is not related to this action.

Please run the publish action with the following option to get more insights into whether the action finishes:

with:
  log_level: DEBUG
kamil-oberaj commented 10 months ago

Okay, thank You so much! I'll ask devops team to fix that. Hope that will work. Anyway now, that we're using Github runners and the issue is solved, I can close the issue.