[BUG] Using libsoup2 and libsoup3 in the same process is not supported.

[lunika]

System info

• Playwright Version: [v1.38.0]
• Operating System: Docker image mcr.microsoft.com/playwright:v1.38.0-jammy
• Browser: [WebKit]
• Other info: The image is ran on circle-ci. The same image as no problem with firefox and chrome browser. We don't have any issue running the same code locally.

A similar issue exists and is closed : https://github.com/microsoft/playwright/issues/26653 This issue should be fixed with Playwright 1.38.0 but we still have it but only with the safari browser.

If you clone the repo, you should not have any error locally, you can test by running the following command :

$ make bootstrap
$ make e2e

You can also check the result on circle-ci with the following link : https://app.circleci.com/pipelines/github/openfun/marsha/14253/workflows/28f88761-d423-43c1-b4c1-4702c879a09f/jobs/421630

Both libsoup 2 and 3 are indeed installed in this image:

$ docker run --rm -it mcr.microsoft.com/playwright:v1.38.0-jammy /bin/bash -c "dpkg -l | grep libsoup"
ii  libsoup-3.0-0:amd64                  3.0.7-0ubuntu1                          amd64        HTTP library implementation in C -- Shared library
ii  libsoup-3.0-common                   3.0.7-0ubuntu1                          all          HTTP library implementation in C -- Common files
ii  libsoup2.4-1:amd64                   2.74.2-3                                amd64        HTTP library implementation in C -- Shared library
ii  libsoup2.4-common                    2.74.2-3                                all          HTTP library implementation in C -- Common files

If libsoup2 is removed (just for testing), it also deletes gstreamer plugins that seems used by webkit.

Source code

Link to the GitHub repository with the repro

https://github.com/openfun/marsha/blob/804a873683fa650623e0c48e92313b072d31f3bf/.circleci/config.yml#L620-L747

This commit is part of this PR: https://github.com/openfun/marsha/pull/2433

Expected

The test suite should run without issue with libsoup. Libsoup compatibility issue is fixed with version 1.38.0 and we don't have this issue with firefox and chrome browsers.

[yury-s]

Yeah, this issue is supposed to have been fixed in https://github.com/microsoft/playwright/issues/26653 but looks like it is not. Does it reproduce inside mcr.microsoft.com/playwright:v1.38.0-jammy outside of circle ci?

[lunika]

No we reproduce it only on circleci.

[tobbsel]

We have the same issue in a Jenkins pipeline. We use mcr.microsoft.com/playwright:v1.38.1-jammy as a base image. Using FROM mcr.microsoft.com/playwright:v1.38.1-jammy RUN apt-get remove -y libsoup2.4 in our docker image fixes our problem for now. Not sure if the mentioned gstreamer plugins are actually no problem or if they just aren't necessary in our concrete use-case. (We also use webkit.)

[yury-s]

Yeah, gstreamer plugins are needed in some scenarios that involve videos.

[lunika]

And this is our case indeed

[tolu]

I've been seeing this same issue in GitLab CI. Our tests do not include video (even though our app does) but was resolved via this before_script: (thanks to @tobbsel comment 🙇 )

playwright_tests:
  stage: e2e-test
  image: mcr.microsoft.com/playwright:v1.38.1-jammy
  before_script:
    - apt-get remove -y libsoup2.4
  script:
    - npm ci
    - npm run test:e2e

[americos]

FWIW I have the same issue in Semaphore CI. Also using mcr.microsoft.com/playwright:v1.38.1-jammy

[oleksii-fedorenko-94]

Same on Buildkite CI using mcr.microsoft.com/playwright:latest. So it seems the issue exists even in 1.39 version.

[mxschmitt]

Steps to reproduce:

1. Create a Ubuntu 20.04 VM
2. Install Docker & run 22.04 PW Docker container with WebKit:

   curl -L get.docker.com | sh
   docker run --rm -it mcr.microsoft.com/playwright:v1.39.0-jammy /bin/bash
   cd
   npm init playwright -y testpw
   # Press Enter a few times
   cd testpw
   npx playwright test --project=webkit

Expected: It works Actual: It does not

    Error: browserType.launch: Browser closed.
    ==================== Browser output: ====================
    <launching> /ms-playwright/webkit-1921/pw_run.sh --inspector-pipe --headless --no-startup-window
    <launched> pid=963
    [pid=963][err] 
    [pid=963][err] (MiniBrowser:985): GStreamer-WARNING **: 13:40:22.894: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.
    [pid=963][err] 
    [pid=963][err] (MiniBrowser:985): libsoup-ERROR **: 13:40:23.080: libsoup3 symbols detected. Using libsoup2 and libsoup3 in the same process is not supported.
    [pid=963][err] /ms-playwright/webkit-1921/pw_run.sh: line 24:   985 Trace/breakpoint trap   (core dumped) WEBKIT_FORCE_COMPLEX_TEXT="1" "$MINIBROWSER" "$@"
    [pid=963] <process did exit: exitCode=133, signal=null>
    [pid=963] starting temporary directories cleanup
    =========================== logs ===========================
    <launching> /ms-playwright/webkit-1921/pw_run.sh --inspector-pipe --headless --no-startup-window
    <launched> pid=963
    [pid=963][err] 
    [pid=963][err] (MiniBrowser:985): GStreamer-WARNING **: 13:40:22.894: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.
    [pid=963][err] 
    [pid=963][err] (MiniBrowser:985): libsoup-ERROR **: 13:40:23.080: libsoup3 symbols detected. Using libsoup2 and libsoup3 in the same process is not supported.
    [pid=963][err] /ms-playwright/webkit-1921/pw_run.sh: line 24:   985 Trace/breakpoint trap   (core dumped) WEBKIT_FORCE_COMPLEX_TEXT="1" "$MINIBROWSER" "$@"
    [pid=963] <process did exit: exitCode=133, signal=null>
    [pid=963] starting temporary directories cleanup
    ============================================================

Main issue to investigate: Why does it work when Docker Host Ubuntu version matches and does not if its Ubuntu 20.04?

Note: If a user runs the Docker container with --privileged it works as expected.

[j-hit]

I also had this issue with mcr.microsoft.com/playwright:v1.39.0-jammy.

I was able to work around this issue by deleting libsoup2.4 explicitly before starting the e2e tests.

My .gitlab-ci.yml file looks something like this.

script:
    - pnpm install
    - pnpm dlx playwright install --with-deps
    - pnpm dlx playwright install chrome
    - pnpm dlx playwright install msedge
    - apt-get remove -y libsoup2.4 # needed because of https://github.com/microsoft/playwright/issues/27313
    - pnpm e2e # executes playwright test

[mxschmitt]

cc @TingPing since you were adding this check in libsoup.

Do you know maybe why there is a difference in loading when using a privileged vs unprivileged Docker container? @dpino and me did some experiments (with strace and LD_DEBUG) and in the privileged one there was no occurrence of libsoup at all which made us wonder why thats the case. First we don't understand how privileged affects module loading of libsoup and second that there is no occurrence of it in the privileged logs. Would appreciate a ton, thanks!

Summary: https://github.com/microsoft/playwright/issues/27313#issuecomment-1782979478

[TingPing]

I can't imagine any reason that would matter.

The solution is just using the soup module from GStreamer 1.20 or newer as it detects which version of libsoup is in use and only uses that one. Earlier versions could only be built against a single version, thus breaking anything mixing the other version.

[mxschmitt]

Some more investigation notes while debugging it with @dpino. The libsoup2 module load chain is the following:

gst-plugins-bad:
-> libgstwebrtc.so:
  -> libnice.so:
    -> libgupnp:
      -> libsoup2:

We tried then to debug gstreamer plugin loader, since around this area its behaving differently. You can do it via the following env var:

GST_DEBUG=GST_PLUGIN_LOADING:7 /ms-playwright/webkit-1921/pw_run.sh --inspector-pipe --headless --no-startup-window &> gstdebug.txt

this produces the following line which is only there in an unprivileged container, a privileged container does not have it: ->

0:00:00.002668171  3184 0x5556b1563ad0 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:442:gst_plugin_loader_try_helper: Spawning gst-plugin-scanner helper failed: Failed to close file descriptor for child process (Operation not permitted)

(Full logs are here.)

This brought up:

• In both scenarios, privileged, unprivileged, the source code, docker image, libraries are 100% the same.
• ideally gstreamer should never load libgstwebrtc, since we (Playwright) are using libwebrtc instead in WebKit.
• When the plugin loading fails, gstreamer will behave weirdly and load for some reason as per above libsoup2 via the dependency tree.
• Why is there no plugin-scanner on the privileged container? -> we don't load any plugins there for some reason. Since we don't load any plugins, we never load libgstwebrtc so we are "good". Can we understand this based on the logs?
• This affects both of our WPE/GTK builds

This suggested the following next steps:

• Build ourselves, same gstreamer version as the one in the system. (ideally, it still fails there). Then build a more recent version and ideally it passes there. This means its an upstream gstreamer bug.
• With our custom gstreamer build, we can also disable the webrtc backend.

[TingPing]

gst-plugins-bad:
-> libgstwebrtc.so:
  -> libnice.so:
    -> libgupnp:
      -> libsoup2:

Ah good find. libgupnp also switched to libsoup3 as of 1.5.0 (dev) / 1.6.0 (stable).

Though just disabling libgstwebrtc makes a lot of sense too.

[mxschmitt]

Looks like here we fallback here to their default loader which is in-process compared to their out-of-process loader as per here:

out-of-process will then not clash with libsoup version mismatch I believe.

Looks like as per gst_registry_scan_path_level we don't have much control about which plugins get loaded or not. plugin_loader_create_blacklist_plugin seems too late.

Based on that I see two ways of proceeding:

• being able somehow to deny-list certain plugins (most likely not possible as how I was reading the source code).
• bundling gstreamer and making sure that our gstreamer plugins get loaded, they seem to not get loaded right now as per the logs.

This seems to solve all the leftover questions.

[yury-s]

Also the fact that gstreamer in-process loader starts loading the plugins eagerly even when there is no media in the web page seems wrong to me. As far as I understand, out-of-process loader does not do this and we don't see any plugins loaded at all on simple pages (which is expected behavior), not sure if it's just a bug in the in-process loader or there is a technical rationale for such behavior.

[mxschmitt]

Closing since we implemented a workaround in the Docker container.

[lunika]

@mxschmitt can you share a docker image tag to test please ?

[mxschmitt]

It gets released as part of 1.40.

[cexbrayat]

@mxschmitt Thanks for looking into this. If I understand correctly the docker image has a fix, but the issue persists if we run the tests directly on a CI (I'm still seeing this even with v1.40 on CircleCi when running tests inside cimg/openjdk:17.0-browsers).

Will there be a fix for those of us who run tests outside of the playwright docker image?

[mikew]

@mxschmitt Will there be a fix for those of us who run tests outside of the playwright docker image?

Looks to be this: https://github.com/microsoft/playwright/commit/6d3913f459570df5d12cdda65f6a2e0a50cd6900

For one of circleci's -browser images, I had to use

# https://github.com/microsoft/playwright/commit/6d3913f459570df5d12cdda65f6a2e0a50cd6900
- run: sudo rm -f /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstwebrtc.so