Is the playwright integration broken?

[LuySeiwert]

Hello,

I am a student from Saarland University looking to use foxhound for my bachelor thesis. I have been able to build the most recent version of foxhound, but not the integration with playwright.

My starting point is a newly installed Ubuntu 20.04 virtual machine. I cloned playwright and edited the UPSTREAM_CONFIG.sh as described in the wiki, after which I ran the prepare_checkout.sh script.

For me it was not possible to run the build command right after as dependencies are still missing. The mach bootstrap from the foxhound version that the playwright checkout cloned failed with toolchains not being found and it does not contain the firefox-releases branch, so I cloned foxhound in a different location, switched to the releases branch and ran the bootstrap command successfully.

Still, the build command fails with some, for me mostly undecipherable, error messages. Two errors stand out, one about redefinition of 'ROOT_CLIP_CHAIN' and one about failed to run custom build command for gecko-profiler v0.1.0.

Here's the full output of the build command: https://pastebin.com/nKUzT0k7

Do you have any idea what could be the issue? Would you perhaps be able to confirm that the build still works for you, and if so, what environment you're using?

[tmbrbr]

Hi @LuySeiwert and thanks for your interest in Foxhound!

At the moment we are supporting Ubuntu 22.04 builds (ie Jammy Jellyfish), I know that there are issues with other versions, perhaps you could try updating your distro? We do daily clean builds on 22.04 which are currently succeeding.

Otherwise, I just saw that the wiki is outdated, which version of Playwright did you use to do the patching?

[leeN]

Hey,

The readme in the wiki is outdated. I maintain a script, that does all the setup stuff for you to build mainline foxhound with Playwright integration. You can use it to (look for the required steps to) build with playwright integration enabled.

[LuySeiwert]

Thank you both for the help!

@tmbrbr As per the wiki, I used playwright v1.21.1.

@leeN Thanks for maintaining such a script! However, on my Ubuntu 22.04 vm I am running into the same issue as https://github.com/SAP/project-foxhound/issues/222. Is this not happening for you?

Using the script, I managed to build foxhound with playwright on a Ubuntu 24.04 virtual machine. Running the binary starts the browser, but trying to start it from within a node script (specifically the jaw crawler https://github.com/SoheilKhodayari/JAW) errors. This might not be an issue with foxhound, but I thought I give it a chance asking you, since perhaps you might have encountered this yourselves before:

node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

browserType.launch: Protocol error (Browser.enable): Browser closed.
==================== Browser output: ====================
<launching> /home/ubuntu/builder-test/foxhound/foxhound-bin -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-TViLAO -juggler-pipe -silent
<launched> pid=4151
[pid=4151][err] *** You are running in headless mode.
[pid=4151][out] console.warn: services.settings: Ignoring preference override of remote settings server
[pid=4151][out] console.warn: services.settings: Allow by setting MOZ_REMOTE_SETTINGS_DEVTOOLS=1 in the environment
[pid=4151][out] console.error: "Warning: unrecognized command line flag" "-juggler-pipe"
[pid=4151][err] JavaScript error: resource://gre/modules/AsyncShutdown.sys.mjs, line 719: Error: Phase "quit-application-granted" is finished, it is too late to register completion condition "UserIdleBlocker"
[pid=4151][out] console.warn: services.settings: main/fingerprinting-protection-overrides sync interrupted by shutdown
[pid=4151][out] console.error: (new TypeError("NetworkError: Network request failed", "resource://services-settings/Utils.sys.mjs", 236))
[pid=4151][out] console.error: Region.sys.mjs: "Error fetching region" (new TypeError("NetworkError when attempting to fetch resource.", ""))
[pid=4151][out] console.error: Region.sys.mjs: "Failed to fetch region" (new Error("NO_RESULT", "resource://gre/modules/Region.sys.mjs", 411))
=========================== logs ===========================
<launching> /home/ubuntu/builder-test/foxhound/foxhound-bin -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-TViLAO -juggler-pipe -silent
<launched> pid=4151
[pid=4151][err] *** You are running in headless mode.
[pid=4151][out] console.warn: services.settings: Ignoring preference override of remote settings server
[pid=4151][out] console.warn: services.settings: Allow by setting MOZ_REMOTE_SETTINGS_DEVTOOLS=1 in the environment
[pid=4151][out] console.error: "Warning: unrecognized command line flag" "-juggler-pipe"
[pid=4151][err] JavaScript error: resource://gre/modules/AsyncShutdown.sys.mjs, line 719: Error: Phase "quit-application-granted" is finished, it is too late to register completion condition "UserIdleBlocker"
[pid=4151][out] console.warn: services.settings: main/fingerprinting-protection-overrides sync interrupted by shutdown
[pid=4151][out] console.error: (new TypeError("NetworkError: Network request failed", "resource://services-settings/Utils.sys.mjs", 236))
[pid=4151][out] console.error: Region.sys.mjs: "Error fetching region" (new TypeError("NetworkError when attempting to fetch resource.", ""))
[pid=4151][out] console.error: Region.sys.mjs: "Failed to fetch region" (new Error("NO_RESULT", "resource://gre/modules/Region.sys.mjs", 411))
============================================================
    at launch_firefox (/home/ubuntu/JAW/crawler/crawler-taint.js:827:23)
    at /home/ubuntu/JAW/crawler/crawler-taint.js:904:22
    at Object.<anonymous> (/home/ubuntu/JAW/crawler/crawler-taint.js:945:3)

Node.js v18.19.1

The error seems to be in the region fetching? But perhaps this might also be a problem of how the browser is launched?

For completeness, this is the function launching the browser:

async function launch_firefox(headless_mode) {

    var ff;
    const ffopts = {}; // { "dom.push.serverURL": "wss://push.services.mozilla.com/" };

    if(should_use_foxhound){
        if(foxhound_executable_path === ''){
            ff = await firefox.launch({ headless: headless_mode, firefoxUserPrefs: ffopts });
        }else{
        // Copy modified config file
        fs.copyFileSync(pathModule.join(BASE_DIR, 'crawler', 'playwright.cfg'),
                pathModule.join(pathModule.dirname(foxhound_executable_path), 'playwright.cfg'));

            ff = await firefox.launch({
                // executablePath: "/mnt/workspace/playwright/project-foxhound/obj-build-playwright/dist/bin/firefox",
                executablePath: foxhound_executable_path,
                headless: headless_mode,
        firefoxUserPrefs: ffopts
            });

        }
    }else{
        ff = await firefox.launch({ headless: headless_mode });
    }

    return ff;
}

So the specific function call to launch the browser is the most deeply neested one, which uses the path to my foxhound binary.

P.S. After describing this issue I'm still facing, I looked through the other issues and it seems to be the same case as https://github.com/SAP/project-foxhound/issues/179?

[leeN]

Hi,

the following line:

[pid=4151][out] console.error: "Warning: unrecognized command line flag" "-juggler-pipe"

means that Playwright is not enabled. This can happen if the patch fails to apply. I've updated the script to stop if that's the case. Can you try again using the updated build script?

[LuySeiwert]

I think you were correct about the patches not being applied correctly!

I pulled your changes and reran the build script which now successfully applied the patches and allowed me to build foxhound with playwright and I can start the browser without any issues, thanks :)

... though I do still wonder what the issue might have been, because as far as I can tell you did not update anything about how the patches are being applied, right?

[LuySeiwert]

Closing this issue since the project-foxhound-builder script works. The part about the patches not being applied seems to have been a user issue more than anything:

As I am trying to modify/ extend foxhound, I wanted to disable the part of the build script which is updating and resetting foxhound as this discards my changes, but this leads to the patches failing to apply. As a workaround, I am using the script to build foxhound once with patches applied and add my changes afterwards. Then I can just rebuild and repackage my modified version of foxhound "by hand".

[tmbrbr]

@LuySeiwert, great that the build is working now!

If you have success modifying and extending Foxhound, consider opening a PR after you are done! We are always looking for active contributions to the project!