mwouts / jupytext

Jupyter Notebooks as Markdown Documents, Julia, Python or R scripts
https://jupytext.readthedocs.io
MIT License
6.65k stars 386 forks source link

Retry flaky UI tests in CI #1262

Closed mahendrapaipuri closed 3 months ago

mahendrapaipuri commented 4 months ago

See comment

Closes #1230

github-actions[bot] commented 4 months ago

Thank you for making this pull request.

Did you know? You can try it on Binder: Binder:lab or Binder:notebook.

Also, the version of Jupytext developed in this PR can be installed with pip:

HATCH_BUILD_HOOKS_ENABLE=true pip install git+https://github.com/mahendrapaipuri/jupytext.git@plywrgt_config

(this requires nodejs, see more at Developing Jupytext)

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.91%. Comparing base (6089897) to head (a34d772).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1262 +/- ## ========================================== - Coverage 95.93% 95.91% -0.03% ========================================== Files 29 29 Lines 4476 4476 ========================================== - Hits 4294 4293 -1 - Misses 182 183 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

mwouts commented 3 months ago

Hi @mahendrapaipuri , I am not sure this test is "flaky" - I had relaunched the publish CI multiple times and it failed on each attempt (https://github.com/mwouts/jupytext/actions/runs/9958694800/job/28138283717) - curious to learn what makes you think that retry would improve this?

mahendrapaipuri commented 3 months ago

Well, if you look into the CI runs of this PR, playwright actually marked that test as flaky. I guess the first run of that test is always a failure and it passes in the subsequent retries.

Anyways, that test does not add any value and hence, I removed it. I added a workflow to update snapshots directly from a PR by commenting "update snapshots". Let me know what you think!

mwouts commented 3 months ago

Thank you @mahendrapaipuri , this looks great! I will have to fix https://github.com/mwouts/jupytext/issues/1266 first but I am looking forward to taking your PR! Thanks