search

insightsengineering / teal.gallery

A Gallery of Exploratory Web Apps used for Analyzing Clinical Trial Data
https://insightsengineering.github.io/teal.gallery/
Other
29 stars 7 forks source link

[Bug]: Cypress tests gives false failures in the deployment CI #111

Open vedhav opened 11 months ago

vedhav commented 11 months ago

The issue

Some cypress tests as part of the deploy.yml workflow fail in the CI while they all succeed locally. Because of this, we are ignoring the results of the test. If this starts to work we can make sure to deploy only after we verify all the modules are loaded in the app.

This workflow ran on d19a0ba2d63f6150a24cfa9459ce1f4434215860

Workflow file ```yaml --- name: Deploy Apps 🚀 on: push: branches: - main - dev workflow_dispatch: schedule: - cron: "12 3 * * *" concurrency: group: publish-${{ github.event.pull_request.number || github.ref }} cancel-in-progress: true env: SHINYAPPSIO_ACCOUNT: genentech APP_PREFIX: NEST GITHUB_PAT: ${{ secrets.REPO_GITHUB_TOKEN }} jobs: deploy: permissions: contents: write defaults: run: shell: bash name: Publish 🗞 runs-on: ubuntu-latest container: image: ghcr.io/insightsengineering/rstudio_4.3.1_bioc_3.17:latest if: > !contains(github.event.commits[0].message, '[skip deploy]') strategy: fail-fast: false matrix: directory: [ "RNA-seq", "basic-teal", "efficacy", "exploratory", "longitudinal", "early-dev", "patient-profile", "python", "safety", ] channel: ["stable", "dev"] steps: - name: Set channel related constants run: | if [ "${{ matrix.channel }}" = "stable" ]; then echo "BRANCH_NAME=main" >> $GITHUB_ENV else echo "BRANCH_NAME=dev" >> $GITHUB_ENV fi - name: Checkout repo 🛎 uses: actions/checkout@v3 with: ref: "${{ env.BRANCH_NAME }}" - name: Check if cypress test exists id: find-cypress run: | if [ -d ${{ matrix.directory }}/tests/cypress ]; then echo "has-cypress-tests=true" >> $GITHUB_OUTPUT else echo "Skipping frontend tests because cypress directory does not exist." echo "has-cypress-tests=false" >> $GITHUB_OUTPUT fi - name: Setup system dependencies for cypress and python app run: > apt-get update && apt-get install --yes libgtk2.0-0 libgbm-dev libnotify-dev libgconf-2-4 xvfb python3.10-venv - name: Setup Node uses: actions/setup-node@v3 if: steps.find-cypress.outputs.has-cypress-tests == 'true' with: node-version: 16 - name: Restore renv from cache uses: actions/cache@v3 env: CACHE_KEY: renv-${{ runner.arch }}-${{ runner.os }}-${{ matrix.directory }}-${{ matrix.channel }} with: path: ${{ matrix.directory }}/renv/library key: ${{ env.CACHE_KEY }}-${{ hashFiles(format('{0}/renv.lock', matrix.directory)) }} restore-keys: ${{ env.CACHE_KEY }}- - name: Update renv.lock file with updated GitHub packages shell: Rscript {0} run: | setwd("${{ matrix.directory }}") lockfile <- renv::lockfile_read() pkg_name_structure <- ifelse("${{ matrix.channel }}" == "stable", "%s/%s@*release", "%s/%s") for (package in lockfile$Packages) { if (package$Source == "GitHub") { # TODO: After teal.slice is released we have to remove this change. This workaround is because update to {bslib} breaks current release of teal.slice if (package$Package == "teal.slice" && "${{ matrix.channel }}" == "stable") { renv::record("insightsengineering/teal.slice@103491c67e26a9a85d636bd105a806904a0111ec") } else { renv::record(sprintf(pkg_name_structure, package$RemoteUsername, package$Package)) } } } - name: Install R packages using renv and update the renv snapshot shell: Rscript {0} working-directory: ${{ matrix.directory }} run: | options(renv.config.cache.symlinks = FALSE) lockfile_pkgs <- renv::lockfile_read()$Package github_pkgs <- names(lockfile_pkgs)[sapply(lockfile_pkgs, function(x) x$Source == "GitHub")] renv::restore(clean = TRUE) renv::update(exclude = github_pkgs) renv::snapshot() - name: Print the new renv.lock file for ${{ matrix.directory }} working-directory: ${{ matrix.directory }} run: cat renv.lock - name: Front end test to check if the app works fine if: steps.find-cypress.outputs.has-cypress-tests == 'true' continue-on-error: true uses: cypress-io/github-action@v6 with: build: npm install cypress --save-dev working-directory: ${{ matrix.directory }}/tests start: npm run run-app wait-on: "http://localhost:3333" wait-on-timeout: 500 - name: Install deploy R package dependencies shell: Rscript {0} working-directory: ${{ matrix.directory }} run: | install.packages(c("BiocManager", "rsconnect")) - name: Deploy 🖨 ${{ matrix.directory }} 🎨 shell: Rscript {0} working-directory: ${{ matrix.directory }} run: | rsconnect::setAccountInfo( name = "${{ env.SHINYAPPSIO_ACCOUNT }}", token = "${{ secrets.SHINYAPPSIO_TOKEN }}", secret = "${{ secrets.SHINYAPPSIO_SECRET }}", server = "shinyapps.io" ) rsconnect::deployApp( appFiles = c("app.R"), appName = rsconnect::generateAppName("${{ env.APP_PREFIX }}_${{ matrix.directory }}_${{ matrix.channel }}"), appTitle = "${{ env.APP_PREFIX }}_${{ matrix.directory }}_${{ matrix.channel }}", account = "${{ env.SHINYAPPSIO_ACCOUNT }}", upload = TRUE, logLevel = "normal", lint = FALSE, forceUpdate = TRUE ) - name: Commit updated renv.lock file continue-on-error: true run: | git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com" git config --global user.name "github-actions" git config --global --add safe.directory /__w/teal.gallery/teal.gallery git fetch git stash git checkout ${{ env.BRANCH_NAME }} git pull git stash apply git add ${{ matrix.directory }}/renv.lock git add ${{ matrix.directory }}/renv/activate.R if [ -n "$(git diff --staged)" ]; then git commit -m "[skip deploy] Update renv.lock file for ${{ matrix.directory }} app" git push origin ${{ env.BRANCH_NAME }} else echo "renv.lock was not modified. Nothing to commit."; fi ```
Cypress failure log ```node Run cypress-io/github-action@v6 with: build: npm install cypress --save-dev working-directory: safety/tests start: npm run run-app wait-on: http://localhost:3333 wait-on-timeout: 500 record: false publish-summary: true component: false env: SHINYAPPSIO_ACCOUNT: genentech APP_PREFIX: NEST GITHUB_PAT: *** BRANCH_NAME: dev /usr/bin/docker exec 4ccaa337[2](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:2)be99db061c64860e5e20c9a042a12e52d21570b15e5d27ac7987227 sh -c "cat /etc/*release | grep ^ID" Received 5299526 of 5299526 (100.0%), [3](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:3)0.6 MBs/sec Cache Size: ~5 MB (5299526 B) /usr/bin/tar -xf /__w/_temp/9210877e-020c-[4](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:4)108-96e8-24fb[5](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:5)91c9979/cache.tgz -P -C /__w/teal.gallery/teal.gallery -z Cache restored successfully Received 1782510[6](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:6)[7](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:7) of 17[8](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:8)251067 (100.0%), 174.7 MBs/sec Cache Size: ~170 MB (178251067 B) /usr/bin/tar -xf /__w/_temp/[9](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:9)a72dd60-0f05-4fcf-870e-5f91963af6c5/cache.tgz -P -C /__w/teal.gallery/teal.gallery -z Cache restored successfully /__t/node/16.20.2/x64/bin/npm ci added 201 packages, and audited 202 packages in 2s 39 packages are looking for funding run `npm fund` for details 3 moderate severity vulnerabilities To address all issues, run: npm audit fix Run `npm audit` for details. /__t/node/16.20.2/x64/bin/npx cypress cache list ┌─────────┬───────────────────┐ │ version │ last used │ ├─────────┼───────────────────┤ │ 13.3.0 │ a few seconds ago │ └─────────┴───────────────────┘ build app command "npm install cypress --save-dev" current working directory "/__w/teal.gallery/teal.gallery/safety/tests" /__t/node/16.20.2/x64/bin/npm install cypress --save-dev changed 1 package, and audited 202 packages in [10](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:10)s 39 packages are looking for funding run `npm fund` for details 3 moderate severity vulnerabilities To address all issues, run: npm audit fix Run `npm audit` for details. start server command "npm run run-app" current working directory "/__w/teal.gallery/teal.gallery/safety/tests" waiting on "http://localhost:3333" with timeout of 500 seconds /__t/node/16.20.2/x64/bin/npm run run-app > run-app > cd .. && Rscript -e "shiny::runApp(port = 3333)" The following package(s) are missing entries in the cache: - packrat - rsconnect - rstudioapi These packages will need to be reinstalled. The following package(s) have broken symlinks into the cache: - packrat - rsconnect - rstudioapi Use `renv::repair()` to try and reinstall these packages. Loading required package: shiny Attaching package: ‘dplyr’ The following objects are masked from ‘package:stats’: filter, lag The following objects are masked from ‘package:base’: intersect, setdiff, setequal, union Loading required package: ggmosaic Loading required package: ggplot2 Loading required package: shinyTree Loading required package: teal Loading required package: teal.data Loading required package: teal.slice Loading required package: teal.transform Loading required package: magrittr Registered S3 method overwritten by 'teal': method from c.teal_slices teal.slice You are using teal version 0.14.0.9024 Attaching package: ‘teal’ The following objects are masked from ‘package:teal.slice’: as.teal_slices, teal_slices Loading required package: tern Loading required package: rtables Loading required package: formatters Attaching package: ‘rtables’ The following object is masked from ‘package:utils’: str Registered S3 method overwritten by 'tern': method from tidy.glm broom [INFO] 2023-[11](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:11)-28 14:08:09.6658 pid:3630 token:[] teal.modules.general Initializing tm_front_page [INFO] 2023-11-28 14:08:09.6694 pid:3630 token:[] teal.modules.general Initializing tm_data_table [INFO] 2023-11-28 14:08:09.6723 pid:3630 token:[] teal.modules.general Initializing tm_variable_browser [INFO] 2023-11-28 14:08:09.6805 pid:3630 token:[] teal.modules.clinical Initializing tm_t_summary [INFO] 2023-11-28 14:08:09.6879 pid:3630 token:[] teal.modules.clinical Initializing tm_t_events_summary [INFO] 2023-11-28 14:08:09.6949 pid:3630 token:[] teal.modules.clinical Initializing tm_t_events [INFO] 2023-11-28 14:08:09.7021 pid:3630 token:[] teal.modules.clinical Initializing tm_t_events_by_grade [INFO] 2023-11-28 14:08:09.7109 pid:3630 token:[] teal.modules.clinical Initializing tm_t_events_patyear [INFO] 2023-11-28 14:08:09.7505 pid:3630 token:[] teal.modules.clinical Initializing tm_t_smq [INFO] 2023-11-28 14:08:09.7613 pid:3630 token:[] teal.modules.clinical Initializing tm_t_summary_by [INFO] 2023-11-28 14:08:09.8386 pid:3630 token:[] teal.modules.clinical Initializing tm_t_shift_by_grade [INFO] 2023-11-28 14:08:10.1735 pid:3630 token:[] teal.modules.clinical Initializing tm_t_abnormality_by_worst_grade [INFO] 2023-11-28 14:08:10.2414 pid:3630 token:[] teal.modules.clinical Initializing tm_t_summary_by [INFO] 2023-11-28 14:08:10.2584 pid:3630 token:[] teal.modules.clinical Initializing tm_t_exposure [INFO] 2023-11-28 14:08:10.2880 pid:3630 token:[] teal.modules.clinical Initializing tm_t_abnormality [INFO] 2023-11-28 14:08:10.2966 pid:3630 token:[] teal.modules.clinical Initializing tm_t_mult_events [INFO] 2023-11-28 14:08:10.3034 pid:3630 token:[] teal.modules.clinical Initializing tm_t_shift_by_arm [INFO] 2023-11-28 14:08:10.4894 pid:3630 token:[] teal.modules.clinical Initializing tm_g_lineplot Warning in rlang::hash(hashables) : 'package:sparkline' may not be available when loading Listening on http://[12](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:12)7.0.0.1:3333 It looks like this is your first time using Cypress: [13](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:13).6.0 [STARTED] Task without title. [SUCCESS] Task without title. Opening Cypress... [3853:1128/[14](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:14)0813.765659:ERROR:object_proxy.cc(590)] Failed to call method: org.freedesktop.portal.Settings.Read: object_path= /org/freedesktop/portal/desktop: unknown error type: DevTools listening on ws://127.0.0.1:33071/devtools/browser/0b0e4dd2-2809-413a-be0b-5ebeea4a837f ==================================================================================================== (Run Starting) ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Cypress: 13.6.0 │ │ Browser: Electron 114 (headless) │ │ Node Version: v20.8.1 (/__e/node20/bin/node) │ │ Specs: 1 found (app.cy.js) │ │ Searched: cypress/e2e/**/*.cy.{js,jsx,ts,tsx} │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ──────────────────────────────────────────────────────────────────────────────────────────────────── Running: app.cy.js (1 of 1) app ✓ Starts (459ms) [INFO] 2023-11-28 14:08:17.9232 pid:3630 token:[9b1ea8f3] teal Initializing reporter_previewer_module [INFO] 2023-11-28 14:08:28.1798 pid:3630 token:[5ab3181b] teal Initializing reporter_previewer_module ✓ Has 12 tabs ([16](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:17)628ms) Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADSL-filter-ADSL_AGE-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEX-filter-ADEX_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADLB-filter-ADLB_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEG-filter-ADEG_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEX-filter-ADEX_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEX-filter-ADEX_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADLB-filter-ADLB_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADLB-filter-ADLB_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEG-filter-ADEG_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEG-filter-ADEG_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from. [INFO] [20](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:21)[23](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:24)-11-28 14:08:40.9578 pid:3630 token:[7f4a4[27](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:28)5] teal Initializing reporter_previewer_module 1) Navigates to all tabs without error 2 passing (34s) 1 failing 1) app Navigates to all tabs without error: AssertionError: Timed out retrying after 4000ms: Expected to find element: `.shiny-busy`, but never found it. Queried from: > cy.get(html) at Context.eval (webpack:///./cypress/e2e/app.cy.js:30:23) (Results) ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Tests: 3 │ │ Passing: 2 │ │ Failing: 1 │ │ Pending: 0 │ │ Skipped: 0 │ │ Screenshots: 1 │ │ Video: false │ │ Duration: 33 seconds │ │ Spec Ran: app.cy.js │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ (Screenshots) - /__w/teal.gallery/teal.gallery/safety/tests/cypress/screenshots/app.cy.js/app -- (1[28](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:29)0x720) Navigates to all tabs without error (failed).png ==================================================================================================== (Run Finished) Spec Tests Passing Failing Pending Skipped ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ✖ app.cy.js 00:[33](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:34) 3 2 1 - - │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ✖ 1 of 1 failed ([100](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:101)%) 00:33 3 2 1 - - Error: Cypress tests: 1 failed ```
cicdguy commented 11 months ago

Agreed. Ideally we'd want to deploy the apps following successful test runs. I'll take a look into possible solution to probe for readiness+liveness of the apps prior to the test execution.