search

actions / runner-images

GitHub Actions runner images
MIT License
10.15k stars 3.06k forks source link

ubuntu-latest causes Node.js process to hang indefinitely #6698

Closed fostimus closed 1 year ago

fostimus commented 1 year ago

Description

We have been using ubuntu-latest for our build process for about a year now. Last week, it started to hang indefinitely and timeout. After going through GitHub support, they recommended to open an issue here.

The last successful run used: Image: ubuntu-20.04 Version: 20221027.1

And now neither ubuntu-20.04 and ubuntu-latest work for us. We are building a Gatsby site via GitHub Actions.

Platforms affected

Runner images affected

Image version and build link

Version: 20221127.1

https://github.com/anvilco/marketing-site/actions/runs/3623252436/jobs/6108931180

We have many other build links that have failed as well.

Is it regression?

20221027.1

Expected behavior

No timeout occurs. Build completes and deploys site to Netlify, and the workflow is successful.

Actual behavior

Worklow never completed. GitHub actions is forced to cancel the workflow after 6 hours (I have updated this to be 40 minutes recently)

Repro steps

Kick off netlify-pr-deploy.yml

mikhailkoliada commented 1 year ago

Hello! We do not have access to the yaml file you attached

fostimus commented 1 year ago

Try this: logs_1979.zip

fostimus commented 1 year ago

Is there a way I can pin my workflow to use verison 20221027.1?

erik-bershel commented 1 year ago

Hello @fostimus There is no way to pin outdated releases, sorry. We'll take a look on your issue. It would be nice if you help us to reach the root of the problem with additional information. Can you share logs from one of previous runs, successful one?

fostimus commented 1 year ago

@erik-bershel bummer about not being able to pin an older release.

Here are the logs from the last successful job for this workflow: last_sucessful_job.zip.

fostimus commented 1 year ago

@erik-bershel any news? We've definitely ruled out any weird caching issues. I went as far as cloning they entire repo to do a fresh build, even against the latest ubuntu-20.04, and it still hangs

OKiMaureen commented 1 year ago

Hello @erik-bershel, the customer shared the following workflow files with me. Let me know if you need any other information:

workflow files ``` pull_request: branches: - master workflow_dispatch: inputs: app_url: description: "User defined URL that points to the application: useful for testing against a deploy preview for the app" default: "https://staging.useanvil.com" env: SLACK_CHANNEL: alerts-marketing-site SLACK_USERNAME: "GitHub Actions" SLACK_COLOR: "#ff3300" SLACK_ICON: https://github.com/anvilco.png?size=48 SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }} jobs: lint: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 with: ref: ${{ github.event.pull_request.head.sha }} - name: Set up app uses: ./.github/actions/setup-app with: github_token: ${{ secrets.CI_GITHUB_TOKEN }} - name: Run Linter run: yarn lint:quiet - name: Slack Notification uses: rtCamp/action-slack-notify@master if: failure() env: SLACK_MESSAGE: "Linting failed, bummer :thumbsdown:" SLACK_TITLE: Linting failed deploy: runs-on: ubuntu-latest needs: lint timeout-minutes: 40 outputs: NETLIFY_URL: ${{ steps.deploy-netlify.outputs.NETLIFY_URL }} steps: - name: Get/set value for APP_URL id: app-url-step run: | USER_INPUT=${{ github.event.inputs.app_url }} echo "::set-output name=APP_URL::${USER_INPUT:-"https://staging.useanvil.com"}" # use $NAME in your action, value will be always provided shell: bash - name: Slack Notification uses: rtCamp/action-slack-notify@master env: SLACK_MESSAGE: "Deploy preview: starting an app build & deploy for ${{ github.event.pull_request.head.sha }}" SLACK_TITLE: Starting deploy preview SLACK_COLOR: "#00cc00" - name: Checkout code uses: actions/checkout@v2 with: ref: ${{ github.event.pull_request.head.sha }} - name: Set up app uses: ./.github/actions/setup-app with: github_token: ${{ secrets.CI_GITHUB_TOKEN }} ``` ``` - name: Build & Deploy Gatsby id: deploy-netlify uses: ./.github/actions/build-gatsby with: NODE_ENV: development APP_URL: ${{ steps.app-url-step.outputs.APP_URL }} NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }} NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }} SEGMENT_WRITE: ${{ secrets.SEGMENT_WRITE }} - name: Slack Notification uses: rtCamp/action-slack-notify@master if: failure() env: SLACK_MESSAGE: "Build & deploy to Netlify failed :grimacing:" SLACK_TITLE: Build & deploy failed - name: Netlify Preview URL uses: unsplash/comment-on-pr@master env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: msg: "This pull request has been deployed to Netlify.\n\n🔍 [Inspect Deploy Logs](${{ steps.deploy-netlify.outputs.NETLIFY_LOGS_URL }}) 🔍\n✅ [Check out the Deploy Preview](${{ steps.deploy-netlify.outputs.NETLIFY_URL }}) ✅ " delete_prev_regex_msg: This pull request has been deployed to Netlify.* # duplicate_msg_pattern: This pull request has been deployed to Netlify.* lighthouse: runs-on: ubuntu-latest needs: deploy steps: - name: Sanity check run: echo "${{ needs.deploy.outputs.NETLIFY_URL }}" - uses: actions/checkout@master - name: Lighthouse uses: foo-software/lighthouse-check-action@master with: urls: ${{ needs.deploy.outputs.NETLIFY_URL }} slackWebhookUrl: ${{ secrets.SLACK_WEBHOOK }} env: SLACK_CHANNEL: alerts-marketing-site SLACK_USERNAME: "GitHub Actions" SLACK_COLOR: "#ff3300" SLACK_ICON: https://github.com/anvilco.png?size=48 SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }} ``` ``` ## https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions name: "Build site" description: "This will set up Node and dependencies on a Github Actions runner" inputs: github_token: description: The Github token to be used for the private repo stuff required: true runs: using: "composite" steps: - name: Use Node version from .nvmrc file id: setup-node uses: actions/setup-node@v2 with: node-version-file: ".nvmrc" registry-url: https://npm.pkg.github.com/ scope: "@anvilco" env: NODE_AUTH_TOKEN: ${{ inputs.github_token }} - uses: actions/cache@v2 id: yarn-cache with: path: node_modules key: v1-${{ hashFiles('**/yarn.lock') }} restore-keys: | v1- - name: Install dependencies with yarn if not cached shell: bash run: | if [ "${{ steps.yarn-cache.outputs.cache-hit }}" == "true" ]; then echo "Cache hit detected! Skipping yarn install..." exit fi yarn install name: "Build & Deploy Gatsby site" description: "Builds the Gatsby site in GH Actions, and triggers a deploy to Netlify" inputs: NODE_ENV: description: The environment to trigger a deploy to. Defaults to 'development' required: false default: development APP_URL: description: The URL to use for the Anvil app. For NODE_ENV == 'production', it should never change from 'https://app.useanvil.com' required: true SEGMENT_WRITE: required: false NETLIFY_AUTH_TOKEN: description: "Auth token of the user to initiate Netlify deploy" required: true NETLIFY_SITE_ID: description: "API ID of the Netlify site you are deploying" required: true commit_sha: description: "The commit SHA that triggered this build & deploy" required: true clear_cache: description: "Boolean to clear existing cache" required: false outputs: NETLIFY_LOGS_URL: value: ${{ steps.deploy-netlify-preview.outputs.NETLIFY_LOGS_URL }} NETLIFY_URL: value: ${{ steps.deploy-netlify-preview.outputs.NETLIFY_URL }} runs: using: "composite" steps: - name: Caching Gatsby id: gatsby-cache-build uses: actions/cache@v2 with: path: | public .cache key: ${{ runner.os }}-gatsby-build-${{ github.run_id }} restore-keys: | ${{ runner.os }}-gatsby-build- - name: Validate prod URL run: | if [ "$APP_URL" != "https://app.useanvil.com" ]; then exit 1 fi shell: bash if: inputs.NODE_ENV == 'production' env: APP_URL: ${{ inputs.APP_URL }} # gatsby is lame, we need to create an .env.* file instead of # actual env vars - name: Make envfile - prod uses: SpicyPizza/create-envfile@v1.3 if: inputs.NODE_ENV == 'production' with: envkey_SITE_URL: https://www.useanvil.com envkey_APP_URL: ${{ inputs.APP_URL }} envkey_SEGMENT_WRITE: ${{ inputs.SEGMENT_WRITE }} file_name: .env.production fail_on_empty: true env: NODE_OPTIONS: --max_old_space_size=8192 SEGMENT_WRITE: ${{ inputs.SEGMENT_WRITE }} - name: Make envfile - not prod if: inputs.NODE_ENV != 'production' run: | touch .env.production echo SITE_URL=https://deploy-preview-${{ github.event.number }}--anvil-marketing.netlify.app >> .env.production echo APP_URL=${{ inputs.APP_URL }} >> .env.production echo SEGMENT_WRITE=${{ inputs.SEGMENT_WRITE }} >> .env.production cat .env.production shell: bash env: NODE_OPTIONS: --max_old_space_size=8192 SEGMENT_WRITE: ${{ inputs.SEGMENT_WRITE }} - name: Clear cache shell: bash run: | if [ "${{ inputs.clear_cache }}" == "true" ]; then echo "Cleaing Gatsby cache...." yarn clean fi - run: NODE_OPTIONS=--max_old_space_size=8192 yarn build shell: bash - name: Deploy to Netlify Preview id: deploy-netlify-preview # uses: netlify/actions/cli@master run: | JSON_OUTPUT=$(npx netlify-cli deploy --dir=public --alias=deploy-preview-${{ github.event.number }} --message="$PREVIEW_MESSAGE" --json) echo "NETLIFY_URL=$(echo $JSON_OUTPUT | python3 -c "import sys, json; print(json.load(sys.stdin)['deploy_url'])")" >> $GITHUB_OUTPUT echo "NETLIFY_LOGS_URL=$(echo $JSON_OUTPUT | python3 -c "import sys, json; print(json.load(sys.stdin)['logs'])")" >> $GITHUB_OUTPUT shell: bash if: inputs.NODE_ENV != 'production' env: NETLIFY_AUTH_TOKEN: ${{ inputs.NETLIFY_AUTH_TOKEN }} NETLIFY_SITE_ID: ${{ inputs.NETLIFY_SITE_ID }} PREVIEW_MESSAGE: "Deploy Preview: ${{ github.event.pull_request.title }}" # with: # args: 'deploy --dir=public --alias=deploy-preview-${{ github.event.number }} --message="$PREVIEW_MESSAGE"' - name: Deploy to Netlify # uses: netlify/actions/cli@master run: | npx netlify-cli deploy --dir=public --prod --message='Production Deploy Successful: ${{ inputs.commit_sha }}' shell: bash if: inputs.NODE_ENV == 'production' env: NETLIFY_AUTH_TOKEN: ${{ inputs.NETLIFY_AUTH_TOKEN }} NETLIFY_SITE_ID: ${{ inputs.NETLIFY_SITE_ID }} # with: # args: "deploy --dir=public --prod --message='Production Deploy Successful: ${{ inputs.commit_sha }}'" ```
dsame commented 1 year ago

Hello @erik-bershel

Is it possible to get content of this directory /.github/actions/build-gatsby or, even better /.github/actions ?

The action above makes actual build and it is absolutely no way to understand the problem without seeing its source code.

erik-bershel commented 1 year ago

Hello everyone here! @fostimus can you share source code which @dsame asking about? If there is such an opportunity, but you would not like to do it publicly, then you can connect our colleague to the question.

cc: @OKiMaureen

dsame commented 1 year ago

@erik-bershel also it make sense to create an issue for Gatsby support (i suppose the client has some kind of it) with the question about the hanging build and attached deploy/8_Build & Deploy Gatsby.txt files from the .zip's of this comment and this.

The hanging during the Gatsby build is not such a rare case and we do not have enough expertise to discover the reason in acceptable timeframe while the Gatsby can be able to solve the issue immediately

fostimus commented 1 year ago

@erik-bershel @dsame It looks like the actions were copied into this issue wrong, as build-gatsby was included in @OKiMaureen's comment. It starts with: 

name: "Build & Deploy Gatsby site"
description: "Builds the Gatsby site in GH Actions, and triggers a deploy to Netlify"

Here it is again:

name: "Build & Deploy Gatsby site"
description: "Builds the Gatsby site in GH Actions, and triggers a deploy to Netlify"

inputs:
  NODE_ENV:
    description: The environment to trigger a deploy to. Defaults to 'development'
    required: false
    default: development
  APP_URL:
    description: The URL to use for the Anvil app. For NODE_ENV == 'production', it should never change from 'https://app.useanvil.com'
    required: true
  SEGMENT_WRITE:
    required: false
  NETLIFY_AUTH_TOKEN:
    description: "Auth token of the user to initiate Netlify deploy"
    required: true
  NETLIFY_SITE_ID:
    description: "API ID of the Netlify site you are deploying"
    required: true
  commit_sha:
    description: "The commit SHA that triggered this build & deploy"
    required: true
  clear_cache:
    description: "Boolean to clear existing cache"
    required: false

outputs:
  NETLIFY_LOGS_URL:
    value: ${{ steps.deploy-netlify-preview.outputs.NETLIFY_LOGS_URL }}
  NETLIFY_URL:
    value: ${{ steps.deploy-netlify-preview.outputs.NETLIFY_URL }}

runs:
  using: "composite"
  steps:
    - name: Caching Gatsby
      id: gatsby-cache-build
      uses: actions/cache@v2
      with:
        path: |
          public
          .cache
        key: ${{ runner.os }}-gatsby-build-${{ github.run_id }}
        restore-keys: |
          ${{ runner.os }}-gatsby-build-

    - name: Validate prod URL
      run: |
        if [ "$APP_URL" != "https://app.useanvil.com" ]; then
          exit 1
        fi
      shell: bash
      if: inputs.NODE_ENV == 'production'
      env:
        APP_URL: ${{ inputs.APP_URL }}

    # gatsby is lame, we need to create an .env.* file instead of
    # actual env vars
    - name: Make envfile - prod
      uses: SpicyPizza/create-envfile@v1.3
      if: inputs.NODE_ENV == 'production'
      with:
        envkey_SITE_URL: https://www.useanvil.com
        envkey_APP_URL: ${{ inputs.APP_URL }}
        envkey_SEGMENT_WRITE: ${{ inputs.SEGMENT_WRITE }}
        file_name: .env.production
        fail_on_empty: true
      env:
        NODE_OPTIONS: --max_old_space_size=8192
        SEGMENT_WRITE: ${{ inputs.SEGMENT_WRITE }}

    - name: Make envfile - not prod
      if: inputs.NODE_ENV != 'production'
      run: |
        touch .env.production
        echo SITE_URL=https://deploy-preview-${{ github.event.number }}--anvil-marketing.netlify.app >> .env.production
        echo APP_URL=${{ inputs.APP_URL }} >> .env.production
        echo SEGMENT_WRITE=${{ inputs.SEGMENT_WRITE }} >> .env.production
        cat .env.production
      shell: bash
      env:
        NODE_OPTIONS: --max_old_space_size=8192
        SEGMENT_WRITE: ${{ inputs.SEGMENT_WRITE }}

    - name: Clear cache
      shell: bash
      run: |
        if [ "${{ inputs.clear_cache }}" == "true" ]; then
          echo "Cleaing Gatsby cache...."
          yarn clean
        fi

    - run: NODE_OPTIONS=--max_old_space_size=8192 yarn build
      shell: bash

    - name: Deploy to Netlify Preview
      id: deploy-netlify-preview
      # uses: netlify/actions/cli@master
      run: |
        JSON_OUTPUT=$(npx netlify-cli deploy --dir=public --alias=deploy-preview-${{ github.event.number }} --message="$PREVIEW_MESSAGE" --json)
        echo "NETLIFY_URL=$(echo $JSON_OUTPUT | python3 -c "import sys, json; print(json.load(sys.stdin)['deploy_url'])")" >> $GITHUB_OUTPUT
        echo "NETLIFY_LOGS_URL=$(echo $JSON_OUTPUT | python3 -c "import sys, json; print(json.load(sys.stdin)['logs'])")" >> $GITHUB_OUTPUT

      shell: bash
      if: inputs.NODE_ENV != 'production'
      env:
        NETLIFY_AUTH_TOKEN: ${{ inputs.NETLIFY_AUTH_TOKEN }}
        NETLIFY_SITE_ID: ${{ inputs.NETLIFY_SITE_ID }}
        PREVIEW_MESSAGE: "Deploy Preview: ${{ github.event.pull_request.title }}"
      # with:
      #   args: 'deploy --dir=public --alias=deploy-preview-${{ github.event.number }} --message="$PREVIEW_MESSAGE"'

    - name: Deploy to Netlify
      # uses: netlify/actions/cli@master
      run: |
        npx netlify-cli deploy --dir=public --prod --message='Production Deploy Successful: ${{ inputs.commit_sha }}'
      shell: bash
      if: inputs.NODE_ENV == 'production'
      env:
        NETLIFY_AUTH_TOKEN: ${{ inputs.NETLIFY_AUTH_TOKEN }}
        NETLIFY_SITE_ID: ${{ inputs.NETLIFY_SITE_ID }}
      # with:
      #   args: "deploy --dir=public --prod --message='Production Deploy Successful: ${{ inputs.commit_sha }}'"
fostimus commented 1 year ago

@erik-bershel also it make sense to create an issue for Gatsby support (i suppose the client has some kind of it) with the question about the hanging build and attached deploy/8_Build & Deploy Gatsby.txt files from the .zip's of this comment and this.

The hanging during the Gatsby build is not such a rare case and we do not have enough expertise to discover the reason in acceptable timeframe while the Gatsby can be able to solve the issue immediately

Gatsby will not be able to solve this. Builds work locally and in other environments we have tested, and we have been deploying new versions of our site without a problem. The only issue is Gatsby on GitHub Actions. If there was a way to pin an older version of a runner, we could easily rule that out as the issue. Seeing this issue (https://github.com/actions/runner-images/issues/6738), I'd guess this is a Node.js thing that Gatsby's build process uses that the new runner images don't like, so again probably not a Gatsby-specific thing.

In any case, we will be moving off of GitHub Actions for building Gatsby, so this is no longer urgent for us. Thanks for the continued support though, I'm happy to help however I can.

erik-bershel commented 1 year ago

@fostimus about using older version of runners than published. I am really sorry about that, but there are no such option technically. We have only one shipped version at every moment (except short periods when CD in action). You can build your own agents with using of elder versions of repo but it is not recommended because of bunch of reasons connected with security and maintenance. And you'll get all pros and cons of selfhosted agents in that particular case, yes.

mikhailkoliada commented 1 year ago

Hello! I am going to close the issue due to no activity. Feel free to re-open one if the problem still exists.

fostimus commented 1 year ago

This is definitely still an issue. I am trying to experiment with running a build command within GitHub Actions, and it is still hanging indefinitely