Certain compilations do not instrument location members correctly

cdaringe commented 1 year ago

What is your Scenario?

window.location.assign("/foo") should navigate my app to test_cafe_origin/test_cafe_session_id/foo, but the browser window is routed absolutely to /foo, ejecting out of TestCafe entirely.

What is the Current behavior?

window.location.assign("/foo") => /foo

What is the Expected behavior?

window.location.assign("/foo") => http://test_cafe_hostname:test_cafe_port/http://my_app/foo

What is your public website URL? (or attach your complete example)

I cannot share this, but want to discuss diagnostic tips s.t. I can triage and possibly root-cause and patch testcafe
note--this doesn't happen in all cases, just one application i have & maintain

What is your TestCafe test code?

n/a. my application is doing the window.location.assign

test.page(
    `${BASE_URL}/bar?tid=0&returnUrl=${encodedReturnUrl}`,
  )
)(
  "can do great work",
  async (t) => {
    await t
      .typeText(selector("input").withAttribute("type", "email"), testEmail)
      .click(selector("button").withAttribute("type", "submit")); // this el, on click, issues a window.location.assign(...)
  }
)

Your complete configuration file

:( i'd love to share this

Your complete test report

No response

Screenshots

No response

Steps to Reproduce

run testcafe
observe: app loads, behaves normally
on window.location.assign, my browser visits that page directly, ejecting from the test cafe env

TestCafe version

1.17.4

Node.js version

16.10.1

Command-line arguments

launched programatically

Browser name(s) and version(s)

chrome, chrome carnary

Platform(s) and version(s)

macos 12.16.1

Other

when building my next.js application using production settings, everything works great
when building my next.js application using development settings, testcafe exhibits this behavior

it seems to be associated with some hammerhead instrumentation.

hammerhead seems to instument __get$ and __call$ functions. these functions are instrumented in my production build, but are not instrumented in my dev build.

what diagnostics can i execute internally to discern why instrumentation is different between my own applications compilations? where in the process is my source-code instrumented by testcafe source-code? can i inspect the TC instrumentation of my source?

cdaringe commented 1 year ago

hmm.

node_modules/.pnpm/testcafe-hammerhead@17.1.20/node_modules/testcafe-hammerhead/lib/processing/script/transformers/method-call.js

this module, i have found, transforms/instruments my browser code in my production builds. however, in development builds, i don't even enter this module for transformation.

it's a bit hard to discern what conditions lead into my browser modules getting transformed or not. there's a findTransform fn somewhere in there that matches in my production application, that i'm not sure is even attempted for my development version application.

i put a unique sigil into my code right next to the window.location.assign(...) call, e.g. console.log({ foobarbaz: "sigil" }), and I did discover that function processScript does receive the script! however, it's still not updated with __call$ and __get$ xforms

github-actions[bot] commented 1 year ago

Thank you for submitting this issue. Since you are using an outdated version of TestCafe, we recommend you update TestCafe to the latest version to check if this issue has been addressed. We constantly improve our tools, and there is a chance that this issue has been already resolved and/or is no longer reproducible in the latest version. We look forward to your response.

cdaringe commented 1 year ago

Issue traced today down to webpack using https://developer.mozilla.org/en-US/docs/Web/API/TrustedScript

hammerhead is/was not aware of TrustedScript in its AST re-writer. when __procScript$ is invoked on an eval, it only looks for apply calls or scripts that are of type string, not of type TrustedScript.

i monkey patched hammerhead to do trustedScript.toString(), and it worked

Roy412 commented 1 year ago

I have the same issue with the latest testcafe version(2.1.0) and you can reproduce it by following steps.

To reproduce 1) Clone the repository - https://github.com/Roy412/msw-testcafe-error/tree/window_assign_error 2) Install the packages - yarn 3) Run the test - yarn testcafe

Initial screen path: https://localhost:1337/knsBl4Sg0*b465RSTNT/http://localhost:3000/knsBl4Sg0*b465RSTNT/http://localhost:3000/account Navigate code: window.location.assign('/account2') Navigated second screen path: https://localhost:1337/account2

miherlosev commented 1 year ago

Hi @Roy412 ,

You've created the https://github.com/DevExpress/testcafe/issues/7429 issue with the same example. At first glance, the cause of the issue from https://github.com/DevExpress/testcafe/issues/7429 is related to the wrong processing scope calculation for the service worker. The cause of this issue is the wrong processing of the trusted script. Maybe, both of these issues are pointing to the same problematic place, but it's difficult to say anything for sure without debugging. So, we will accept that there is no simple reproducible example for this issue.

However, @cdaringe can try using the --experimental-proxyless option as a workaround. Option syntax:

// Command-line
testcafe chrome tests --experimental-proxyless
// Programmatic
const testcafe = await createTestCafe({ experimentalProxyless: true });

// Configuration file
{
"experimentalProxyless": "true"
}

Set up the testcafe@2.2.0-rc.1 version and try running your tests in Proxyless mode. I look forward to your results. Note that, at present, it is in experimental mode. Also, the Proxyless mode is implemented only in Google Chrome. It will not work correctly if you run tests in a non-Chrome browser or in a combination of other browsers.

cdaringe commented 1 year ago

both of these issues are pointing to the same problematic place, but it's difficult to say anything for sure without debugging

Concretely, testcafe hammerhead, when it processes eval scripts, has no awareness of TrustedScripts.

Here's the associated lines of code: https://github.com/DevExpress/testcafe-hammerhead/blob/4751c9fd97df3e4a0d505cc909cbd191fd1d227a/src/client/sandbox/code-instrumentation/index.ts#L60-L77

A reproduction can be furnished easily:

A fresh next.js installation (e.g. npm create-next-app ...)
wiring that app thru testcafe with the next.js dev server (next dev),
...and invocation of window.location.assign(...) issued within the next.js development application's source code...

ought demonstrate the issue, if debugging is needed. however, debugging probably isn't strictly needed here, as the codepath is known. it's not surprising to observe learn that processScript may receive a TrustedScript instance. Currently, that function is hardcoded to look only for string scripts. No fault of TestCafe's--it's a new API that browser application frameworks are starting to integrate with!

ps, excited to hear about proxyless mode! curious how that works? is there a thread i can follow to learn more about that?

cdaringe commented 1 year ago

I've tried the experimental proxyless mode. TC launches, TC loads my web app, my web-app loads, but:

no TC banner/controls render on the page
TC halts... it seems to not be present/active. The TC server stdio emits nothing, and the browser is just stopped there, with my app loaded 🤔 . after some timeout, TC reloads the the browser and tries again, but... nil. looking forward to digging in further there

miherlosev commented 1 year ago

@Roy412,

Thank you for this investigation. I will mark this issue as a duplicate of https://github.com/DevExpress/testcafe/issues/7429. All future updates will be posted there.

@cdaringe

Could you please clarify what Node.js version and operating system you use?

cdaringe commented 1 year ago

node 16.x, osx 12.x. However the issue affects all versions. Curious why youve closed the issue? Should I read into that? 😀

cdaringe commented 1 year ago

Oh, marked as dupe. Whoops! Will read the other issue

cdaringe commented 1 year ago

I looked at the other issue. Seems like different failures, although the same symptoms.

miherlosev commented 1 year ago

node 16.x, osx 12.x. However the issue affects all versions. Curious why youve closed the issue? Should I read into that?

I run a simple test with testcafe@2.2.0-rc.1, Node.js (16.13.1), and MacOS Ventura (13.0.1) and it works fine.

testcafe chrome test.js --experimental-proxyless

fixture `Fixture`
    .page('https://example.com');

test('test', async t => {
     await t.click('body');
});

Could you please share a simple example that we can run locally?

cdaringe commented 1 year ago

@miherlosev,

have you read my above diagnostics?
does your simple script on example.com use TrustedScript to testcafe __procScript$ instrumentation?
did you consider the link I provided above to the offending lines of code?

He're a MVP demo: https://github.com/cdaringe/trustedscript-eval-testcafe-bug . Simple instructions are in the readme.

As I've shared above, this code is wrong: https://github.com/DevExpress/testcafe-hammerhead/blob/4751c9fd97df3e4a0d505cc909cbd191fd1d227a/src/client/sandbox/code-instrumentation/index.ts#L60-L77

The final line does:

return script

Returning the script from the function directly is ignoring failure to process the script, silently.

It should instead have instead either:

option-1:

throw new Error(`unknown type: ${typeof script} detected. cannot process script`)

or, option-2 (preferred): option-1 and handle TrustedScript instances

miherlosev commented 1 year ago

@cdaringe

I ran the tested application and it threw an error.

Please fix it.

DevExpress / testcafe