apache / incubator-stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm
https://stormcrawler.apache.org/
Apache License 2.0
890 stars 260 forks source link

Apple Silicon emulation issue in unit tests #1209

Closed joshfischer1108 closed 6 months ago

joshfischer1108 commented 6 months ago

When compiling Stormcrawler from source on Apple Silicon we are hitting timeout issues in selenium tests due to emulation issues.

Steps to reproduce:

Using an Apple M3: From the top level directory run:

mvn clean install

First we get this warning.

The architecture 'amd64' for image 'selenium/standalone-chrome:120.0' (ID sha256:deff784da2138b912b66e2941cc976ced4ecba3a4e6941ca3bfa2b8c88886b75) does not match the Docker server architecture 'arm64'. This will cause the container to execute much more slowly due to emulation and may lead to timeout failures.

Then we get this error:

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 27.48 s <<< FAILURE! -- in org.apache.stormcrawler.protocol.selenium.ProtocolTest
[ERROR] org.apache.stormcrawler.protocol.selenium.ProtocolTest.testBlocking -- Time elapsed: 27.44 s <<< ERROR!
org.awaitility.core.ConditionTimeoutException: Condition with org.apache.stormcrawler.protocol.selenium.ProtocolTest was not fulfilled within 10 seconds.

Then the error should appear

rzo1 commented 6 months ago

Guess it is more or less an upstream thing: https://github.com/SeleniumHQ/docker-selenium/issues/1076

Might be worth adding a profile which exclude those tests from being run on arm64, or to write a vustom JUnit 5 extension to disable them in OSX ARM, or to try Geckodriver, or to play with the timeouts.

Guess we can start with increasing timeouts or switching to Geckodriver (which should be arm ready from cross reading).

joshfischer1108 commented 6 months ago

I'm looking at the test. Are these the timeouts? I've changed them to much higher values such as 100000 and the tests seem to timeout about the same time on my machine (which is around 27 seconds)

 timeouts.put("implicit", 10000);
timeouts.put("pageLoad", 10000);
timeouts.put("script", 10000);

conf.put("selenium.timeouts", timeouts);
joshfischer1108 commented 6 months ago

I forgot to add the link. Here is where I am looking