google / blockly

The web-based visual programming editor.
https://developers.google.com/blockly/
Apache License 2.0
12.49k stars 3.71k forks source link

Debug and reenable macOS CI on GitHub Actions #2114

Open rachel-fenichel opened 5 years ago

rachel-fenichel commented 5 years ago

Generator tests require firefox instead of chrome, but our current setup has issues with geckodriver on osx. Low priority because we do regularly build on osx, and because ubuntu tests are running.

cpcallen commented 3 years ago

Working on this in PR #4903. Looks like the generator tests are fine on macOS, but mocha tests are failing in CI (not locally).

cpcallen commented 3 years ago

I looked at this a bit but I am unable to figure out what is going wrong. Comparing test logs from my own machine vs. our CI servers it looks like the CI servers are actually running MORE mocha tests than get run locally. Here are the test output files, trimmed to start with the mocha tests and modified to remove timestamps and element IDs for easier diffing:

Could it be that tests are passing locally only because we are not running all of them?

I'm going to leave this for the time being because I do not understand mocha, selenium, webdriver, etc.

cpcallen commented 3 years ago

One further possible clue about what is going on: it seems that the mocha tests are flaky on macOS, and sometimes succeed and sometimes not; e.g., for the same commit:

I don't think its v16 specific, however, because at earlier commits v12 passed but v14 and v16 failed.

rachel-fenichel commented 1 year ago

To work on this bug:

alicialics commented 12 months ago

@BeksOmega looks like all tests passes on macos for now: https://github.com/alicialics/blockly/actions/runs/6830615442/job/18578823354?pr=2

maribethb commented 11 months ago

Reverting #7640 in #7692 because CI does not pass consistently, but is unrelated to macos actually failing.