search

web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
5k stars 3.1k forks source link

webkitgtk nightly test runs failed #24752

Closed jdm closed 4 years ago

jdm commented 4 years ago

In particular, the logs show that it stalled while installing packages:

Setting up x11proto-record-dev (2019.2-1ubuntu1) ...
Setting up libndp0:amd64 (1.7-0ubuntu1) ...
Setting up gstreamer1.0-pulseaudio:amd64 (1.16.2-1ubuntu2) ...
Setting up keyboard-configuration (1.194ubuntu3) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
Configuring keyboard-configuration
----------------------------------

The layout of keyboards varies per country, with some countries having multiple
common layouts. Please select the country of origin for the keyboard of this
computer.

  1. Afghani
  2. Albanian
  3. Amharic
  4. Arabic
  5. Arabic (Morocco)
  6. Arabic (Syria)
  7. Armenian
  8. Azerbaijani
  9. Bambara
  10. Bangla
  11. Belarusian
  12. Belgian
  13. Berber (Algeria, Latin)
  14. Bosnian
  15. Braille
  16. Bulgarian
  17. Burmese
  18. Chinese
  19. Croatian
  20. Czech
  21. Danish
  22. Dhivehi
  23. Dutch
  24. Dzongkha
  25. English (Australian)
  26. English (Cameroon)
  27. English (Ghana)
  28. English (Nigeria)
  29. English (South Africa)
  30. English (UK)
  31. English (US)
  32. Esperanto
  33. Estonian
  34. Faroese
  35. Filipino
  36. Finnish
  37. French
  38. French (Canada)
  39. French (Democratic Republic of the Congo)
  40. French (Guinea)
  41. French (Togo)
  42. Georgian
  43. German
  44. German (Austria)
  45. Greek
  46. Hebrew
  47. Hungarian
  48. Icelandic
  49. Indian
  50. Indonesian (Arab Melayu, phonetic)
  51. Indonesian (Javanese)
  52. Iraqi
  53. Irish
  54. Italian
  55. Japanese
  56. Japanese (PC-98)
  57. Kazakh
  58. Khmer (Cambodia)
  59. Korean
  60. Kyrgyz
  61. Lao
  62. Latvian
  63. Lithuanian
  64. Macedonian
  65. Malay (Jawi, Arabic Keyboard)
  66. Maltese
  67. Maori
  68. Moldavian
  69. Mongolian
  70. Montenegrin
  71. Nepali
  72. Norwegian
  73. Persian
  74. Polish
  75. Portuguese
  76. Portuguese (Brazil)
  77. Romanian
  78. Russian
  79. Serbian
  80. Sinhala (phonetic)
  81. Slovak
  82. Slovenian
  83. Spanish
  84. Spanish (Latin American)
  85. Swahili (Kenya)
  86. Swahili (Tanzania)
  87. Swedish
  88. Switzerland
  89. Taiwanese
  90. Tajik
  91. Thai
  92. Tswana
  93. Turkish
  94. Turkmen
  95. Ukrainian
  96. Urdu (Pakistan)
  97. Uzbek
  98. Vietnamese
  99. Wolof
Country of origin for the keyboard: 
[taskcluster:error] Task timeout after 7200 seconds. Force killing container.

This is likely a regression from the docker image update in https://github.com/web-platform-tests/wpt/pull/24673.

stephenmcgruer commented 4 years ago

Log from a failing run

Looks like we're trying to select keyboard-configuration (not sure why yet), which is then prompting input. https://askubuntu.com/questions/876240/how-to-automate-setting-up-of-keyboard-configuration-package implies you can pass DEBIAN_FRONTEND=noninteractive to that specific package to avoid its setup. (On a side-note, in a similar question there is someone advocating that you shouldn't be doing GUI in Docker ever, which is interesting...)

So a good start would be figuring out why its trying to select this package. I suspect maybe its from /opt/webkitgtk/nightly/install-dependencies (tools/ci/run_tc.py) cc @clopez

clopez commented 4 years ago

Looks like we're trying to select keyboard-configuration (not sure why yet), which is then prompting input. https://askubuntu.com/questions/876240/how-to-automate-setting-up-of-keyboard-configuration-package implies you can pass DEBIAN_FRONTEND=noninteractive to that specific package to avoid its setup. (On a side-note, in a similar question there is someone advocating that you shouldn't be doing GUI in Docker ever, which is interesting...)

That install-dependencies script is not optimized to install only the minimum dependencies and installs a bunch of stuff that is not strictly needed for running webkitgtk.

I'm working on the generation of improved bundles that will be more light and require less dependencies.

So a good start would be figuring out why its trying to select this package. I suspect maybe its from /opt/webkitgtk/nightly/install-dependencies (tools/ci/run_tc.py) cc @clopez

For now, I have modified that install-dependencies script that is shipped in those bundles to use DEBIAN_FRONTEND=noninteractive. It should be effective on the next nightly bundle.

Let's keep this open until that happens and I can verify the issue is fixed

stephenmcgruer commented 4 years ago

@clopez looks from wpt.fyi that we still don't have webkitgtk runs - can you check out the latest epochs/daily run and see what is going on now?

jdm commented 4 years ago 
sudo apt-get -qqy -t bionic-wpt-webkit-updates install webkit2gtk-driver
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
E: Unable to correct problems, you have held broken packages.
Traceback (most recent call last):
  File "./tools/ci/run_tc.py", line 493, in <module>
    main()
  File "./tools/ci/run_tc.py", line 485, in main
    setup_environment(args)
  File "./tools/ci/run_tc.py", line 329, in setup_environment
    install_webkitgtk(args.channel)
  File "./tools/ci/run_tc.py", line 224, in install_webkitgtk
    install_webkitgtk_from_apt_repository(channel)
  File "./tools/ci/run_tc.py", line 174, in install_webkitgtk_from_apt_repository
    run(["sudo", "apt-get", "-qqy", "-t", "bionic-wpt-webkit-updates", "install", "webkit2gtk-driver"])
  File "./tools/ci/run_tc.py", line 69, in run
    return f(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', 'apt-get', '-qqy', '-t', 'bionic-wpt-webkit-updates', 'install', 'webkit2gtk-driver']' returned non-zero exit status 100
clopez commented 4 years ago

That log above is for the stable runs, which has a different problem than nightlies. Let's try to fix first the nightlies as they are more interesting and also I'm planning to revamp how the stable bundles are generated to use tarballs like nightlies instead of deb packages.

Regarding the failure for nightlies I see this:

2020-08-04 00:52:23,538 - tc-run - INFO - Running all tests
2020-08-04 00:52:23,538 - tc-run - INFO - Executing command: python ./wpt run --channel=nightly --log-wptreport=../artifacts/wpt_report.json --log-wptscreenshot=../artifacts/wpt_screenshot.txt --no-fail-on-unexpected --this-chunk=1 --total-chunks=16 --test-type=testharness --log-mach-level=info --log-mach=- -y --no-pause --no-restart-on-unexpected --install-fonts --no-headless --verify-log-full webkitgtk_minibrowser
CRITICAL:tools.wpt.utils:('/usr/local/bin/virtualenv', '/home/test/web-platform-tests/_venv2', '-p', '/usr/bin/python') exited with return code 1
CRITICAL:tools.wpt.utils:ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'
Traceback (most recent call last):
  File "./wpt", line 21, in <module>
    wpt.main()
  File "/home/test/web-platform-tests/tools/wpt/wpt.py", line 150, in main
    venv = setup_virtualenv(main_args.venv, main_args.skip_venv_setup, props)
  File "/home/test/web-platform-tests/tools/wpt/wpt.py", line 126, in setup_virtualenv
    venv.start()
  File "/home/test/web-platform-tests/tools/wpt/virtualenv.py", line 104, in start
    self.create()
  File "/home/test/web-platform-tests/tools/wpt/virtualenv.py", line 48, in create
    call(self.virtualenv, self.path, "-p", sys.executable)
  File "/home/test/web-platform-tests/tools/wpt/utils.py", line 54, in call
    return subprocess.check_output(args).decode('utf8')
  File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '('/usr/local/bin/virtualenv', '/home/test/web-platform-tests/_venv2', '-p', '/usr/bin/python')' returned non-zero exit status 1
[taskcluster 2020-08-04 00:52:24.596Z] === Task Finished ===

Don't know why that happens or what causes it. I'll try to debug it tomorrow (busy today)

clopez commented 4 years ago

Don't know why that happens or what causes it. I'll try to debug it tomorrow (busy today)

Found the issue. The bundle was not working on Ubuntu-20.04 because it was built for Ubuntu-18.04 and for running on Ubuntu-20.04 it missed some libraries, so it was not starting. I fixed this by adding this extra libraries to the bundle.

Should work now. Let's keep this open until I can verify it works on the WPT CI

clopez commented 4 years ago

It didn't worked. It turns out there were other issue on top of the previous:

From log https://community.taskcluster-artifacts.net/ZemInQXHTVWB4uMwbPMo1w/0/public/logs/live_backing.log

2020-08-06 00:55:22,560 - tc-run - INFO - Running all tests
2020-08-06 00:55:22,560 - tc-run - INFO - Executing command: python ./wpt run --channel=nightly --log-wptreport=../artifacts/wpt_report.json --log-wptscreenshot=../artifacts/wpt_screenshot.txt --no-fail-on-unexpected --this-chunk=4 --total-chunks=16 --test-type=testharness --log-mach-level=info --log-mach=- -y --no-pause --no-restart-on-unexpected --install-fonts --no-headless --verify-log-full webkitgtk_minibrowser
CRITICAL:tools.wpt.utils:('/usr/local/bin/virtualenv', '/home/test/web-platform-tests/_venv2', '-p', '/usr/bin/python') exited with return code 1
CRITICAL:tools.wpt.utils:ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

Traceback (most recent call last):
  File "./wpt", line 21, in <module>
    wpt.main()
  File "/home/test/web-platform-tests/tools/wpt/wpt.py", line 152, in main
    venv = setup_virtualenv(main_args.venv, main_args.skip_venv_setup, props)
  File "/home/test/web-platform-tests/tools/wpt/wpt.py", line 128, in setup_virtualenv
    venv.start()
  File "/home/test/web-platform-tests/tools/wpt/virtualenv.py", line 104, in start
    self.create()
  File "/home/test/web-platform-tests/tools/wpt/virtualenv.py", line 48, in create
    call(self.virtualenv, self.path, "-p", sys.executable)
  File "/home/test/web-platform-tests/tools/wpt/utils.py", line 54, in call
    return subprocess.check_output(args).decode('utf8')
  File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '('/usr/local/bin/virtualenv', '/home/test/web-platform-tests/_venv2', '-p', '/usr/bin/python')' returned non-zero exit status 1
[taskcluster 2020-08-06 00:55:24.024Z] === Task Finished ===
[taskcluster 2020-08-06 00:55:24.678Z] Unsuccessful task run with exit code: 1 completed in 343.896 seconds

It seems there is an issue on Debian/Ubuntu where the system version of python-virtualenv conflicts with the one installed via pip. See https://github.com/pypa/virtualenv/issues/1873

And it only happens on the WebKitGTK runs, because the install-dependencies script of the WebKitGTK bundle installs virtualenv via apt-get. I have modified the install-dependencies script to not do this (since its not needed four use case).

I think now it should be working back, but let's keep this open until I can see it fixed.

I'm also working on a different way of generating the bundles for testing on the CI that would require way less dependencies to be installed.

clopez commented 4 years ago

The issue its still not fixed, but I suspect there is a new problem now somewhere on the taskcluster logic (perhaps from a recent change). For some reason I don't know I can't see anymore the wpt-webkitgtk_minibrowser-nightly-* runs on the commits of the epochs/daily branch. Neither a failed run, neither a good run. For example on the commit 875cbea8c68714da40c0f06b5f49e133619c7a8c there should be several task-cluster runs with the prefix wpt-webkitgtk_minibrowser-nightly-*, but I can't see any of themt. Any idea?

stephenmcgruer commented 4 years ago

I wonder if that's a GitHub UI problem. If one looks at https://community-tc.services.mozilla.com/tasks/groups/JNm1aj4fSlCJxgvByPlkHQ, which I believe is the Taskcluster group for the latest epochs/daily push, it looks like wpt-webkitgtk_minibrowser-nightly-* did happen and all passed.

But also, if one looks in wpt.fyi, it isn't there on the runs view! But it is there (I think?) on https://wpt.fyi/api/runs?sha=875cbea&max-count=100&product=webkitgtk ... @Hexcles ?

Hexcles commented 4 years ago

webkitgtk isn't a default product. Try https://wpt.fyi/runs?label=master&from=2020-08-04T00%3A00&product=webkitgtk

stephenmcgruer commented 4 years ago

They appear to be missing the 'experimental' label.

Hexcles commented 4 years ago

Ahh sorry that's a regression from my https://github.com/web-platform-tests/wpt.fyi/pull/2081 (not surprisingly, as we didn't have a test for webkitgtk)

Hexcles commented 4 years ago

This has been fixed in https://staging.wpt.fyi/runs?label=master&label=experimental&max-count=100&product=webkitgtk

The fix will be released into prod next week.

KyleJu commented 4 years ago

This has been fixed in https://staging.wpt.fyi/runs?label=master&label=experimental&max-count=100&product=webkitgtk

The fix will be released into prod next week.

Also fixed in prod after the release

clopez commented 4 years ago

That install-dependencies script is not optimized to install only the minimum dependencies and installs a bunch of stuff that is not strictly needed for running webkitgtk.

I'm working on the generation of improved bundles that will be more light and require less dependencies.

I submitted a PR for using this new bundles at #25263