Expensify / App

Welcome to New Expensify: a complete re-imagination of financial collaboration, centered around chat. Help us build the next generation of Expensify by sharing feedback and contributing to the code.
https://new.expensify.com
MIT License
3.34k stars 2.77k forks source link

Investigate workflow job failing on main: e2ePerformanceTests / Run E2E tests in AWS device farm #49198

Open github-actions[bot] opened 5 days ago

github-actions[bot] commented 5 days ago

🚨 Failure Summary 🚨:

⚠️ Action Required ⚠️:

πŸ› οΈ A recent merge appears to have caused a failure in the job named e2ePerformanceTests / Run E2E tests in AWS device farm. This issue has been automatically created and labeled with Workflow Failure for investigation.

πŸ‘€ Please look into the following:

  1. Why the PR caused the job to fail?
  2. Address any underlying issues.

πŸ› We appreciate your help in squashing this bug!

saifelance commented 4 days ago

1. Update Node.js Version for AWS Device Farm

Current workflow is using a deprecated version of Node.js (Node 16), you need to update the realm/aws-devicefarm action to use Node.js 20.

Update .github/workflows/e2ePerformanceTests.yml

Update the workflow file to use Node.js 20, and ensure the correct version of the realm/aws-devicefarm action is used. Example Workflow (e2ePerformanceTests.yml):

name: e2ePerformanceTests

on:
  push:
    branches:
      - main

jobs:
  run-e2e-tests:
    runs-on: ubuntu-latest
    steps:
      # Checkout code
      - name: Checkout code
        uses: actions/checkout@v3

      # Setup Node.js version (Node.js 20)
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '20'

      # Install dependencies
      - name: Install dependencies
        run: npm install

      # AWS Device Farm Configuration
      - name: Run E2E tests in AWS Device Farm
        uses: realm/aws-devicefarm@v1 # Ensure to use a compatible version
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: 'us-west-2' # Set your region
          project_arn: ${{ secrets.AWS_DEVICEFARM_PROJECT_ARN }}
          device_pool_arn: ${{ secrets.AWS_DEVICEFARM_DEVICE_POOL_ARN }}
          app_path: './app/build/outputs/apk/debug/app-debug.apk' # Set your app path
          test_path: './app/build/outputs/test/debug/test-debug.apk'
          test_type: 'INSTRUMENTATION' # Set appropriate test type
          timeout_minutes: 120 # Adjust timeout if necessary

      # Upload logs
      - name: Upload test logs
        uses: actions/upload-artifact@v3
        with:
          name: debug.log
          path: path/to/debug.log # Ensure logs are correctly configured and uploaded

Increase Timeout and Handle Test Failures

2 . Install Required Dependencies

Installation of AWS CLI (locally or in CI):

# Install AWS CLI if not already installed
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Configuring AWS CLI:

aws configure # Enter AWS Access Key, Secret Access Key, region, and output format.

3 . Add Retry Logic in GitHub Actions:

  # Run E2E tests in AWS Device Farm with retry logic
      - name: Run E2E tests in AWS Device Farm
        uses: realm/aws-devicefarm@v1
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: 'us-west-2'
          project_arn: ${{ secrets.AWS_DEVICEFARM_PROJECT_ARN }}
          device_pool_arn: ${{ secrets.AWS_DEVICEFARM_DEVICE_POOL_ARN }}
          app_path: './app/build/outputs/apk/debug/app-debug.apk'
          test_path: './app/build/outputs/test/debug/test-debug.apk'
          test_type: 'INSTRUMENTATION'
        continue-on-error: true # Allow the job to continue even if it fails

      # Retry failed step
      - name: Retry E2E tests if failed
        if: failure()
        run: |
          echo "Retrying failed E2E tests..."
          # Retry logic here (you can rerun specific commands if needed)

4 . Verify Node.js Version in GitHub Actions

yaml
Copy code
      # Verify Node.js version
      - name: Verify Node.js Version
        run: node -v

By following these steps, you should be able to:

  1. Fix the Node.js deprecation warning.
  2. Address the test timeout issue.
  3. Ensure that test logs are properly uploaded.
  4. Optionally, implement a retry mechanism for intermittent test failures.

Let me know if you need further clarification or help with any part of the process!

deetergp commented 2 days ago

Started a convo about this in Slack.

deetergp commented 12 hours ago

@kirillzyusko & @Szymon20000 Are investigating and sharing their results in this Slack thread.