search

mswjs / msw

Industry standard API mocking for JavaScript.
https://mswjs.io
MIT License
16.01k stars 527 forks source link

Request bodies of URLSearchParams can not be read #1365

Closed markdon closed 1 year ago

markdon commented 2 years ago

Prerequisites

Environment check

Node.js version

v16.13.0

Reproduction repository

https://github.com/markdon/cra-msw

Reproduction steps

clone repo npm i DEBUG=* npm run test -- --watchAll=false

Current behavior

I think this is related to https://github.com/mswjs/msw/issues/1318

When handling a fetch request where the request body is a URLSearchParameters object, the body can not be read in the handler. This was previously available through request.body (v0.27.1). request.body is now deprecated.

I would expect to be able to just use the newer request.text(), however this seems to throw an error parsing the body.

Debug logs. See ERR_INVALID_ARG_TYPE 
➜  cra-msw git:(main) DEBUG=* npm run test -- --watchAll=false

> cra-msw@0.1.0 test
> react-scripts test "--watchAll=false"

  babel:config:loading:files:plugins Loaded preset '/Users/mark/projects/cra-msw/node_modules/babel-preset-react-app/index.js' from '/Users/mark/projects/cra-msw'. +0ms
  babel:config:loading:files:plugins Loaded preset '/Users/mark/projects/cra-msw/node_modules/babel-preset-jest/index.js' from '/Users/mark/projects/cra-msw'. +2ms
  http constructing the interceptor... +0ms
  xhr constructing the interceptor... +0ms
  setup-server constructing the interceptor... +0ms
  http:on adding "request" event listener: setupServerListener +0ms
  async-event-emitter:on adding "request" listener... +0ms
  xhr:on adding "request" event listener: setupServerListener +0ms
  async-event-emitter:on adding "request" listener... +0ms
  http:on adding "response" event listener:  +0ms
  async-event-emitter:on adding "response" listener... +0ms
  xhr:on adding "response" event listener:  +0ms
  async-event-emitter:on adding "response" listener... +0ms
  setup-server:apply applying the interceptor... +0ms
  async-event-emitter:activate set state to: ACTIVE +0ms
  setup-server:apply activated the emiter! ACTIVE +0ms
  setup-server retrieved global instance: undefined +4ms
  setup-server:apply no running instance found, setting up a new instance... +1ms
  setup-server:setup applying all 2 interceptors... +0ms
  setup-server:setup applying "ClientRequestInterceptor" interceptor... +0ms
  http:apply applying the interceptor... +0ms
  async-event-emitter:activate set state to: ACTIVE +0ms
  http:apply activated the emiter! ACTIVE +0ms
  http retrieved global instance: undefined +4ms
  http:apply no running instance found, setting up a new instance... +0ms
  http:setup native "http" module patched! +0ms
  http:setup native "https" module patched! +0ms
  http set global instance! http +0ms
  setup-server:setup adding interceptor dispose subscription +1ms
  setup-server:setup applying "XMLHttpRequestInterceptor" interceptor... +0ms
  xhr:apply applying the interceptor... +0ms
  async-event-emitter:activate set state to: ACTIVE +0ms
  xhr:apply activated the emiter! ACTIVE +0ms
  xhr retrieved global instance: undefined +5ms
  xhr:apply no running instance found, setting up a new instance... +0ms
  xhr:setup patching "XMLHttpRequest" module... +0ms
  xhr:setup native "XMLHttpRequest" module patched! XMLHttpRequestOverride +0ms
  xhr set global instance! xhr +0ms
  setup-server:setup adding interceptor dispose subscription +0ms
  setup-server set global instance! setup-server +1ms
  xhr:request POST /login open {
  method: 'POST',
  url: '/login',
  async: true,
  user: undefined,
  password: undefined
} +0ms
  xhr:request POST /login reset +1ms
  xhr:request POST /login readyState change 0 -> 1 +1ms
  xhr:request POST /login triggering readystate change... +0ms
  xhr:request POST /login trigger "readystatechange" (1) +0ms
  xhr:request POST /login resolve listener for event "readystatechange" +0ms
  xhr:request POST /login set request header "content-type" to "application/x-www-form-urlencoded;charset=UTF-8" +0ms
  xhr:request POST /login send POST /login +0ms
  xhr:request POST /login request headers HeadersPolyfill {
  [Symbol(normalizedHeaders)]: { 'content-type': 'application/x-www-form-urlencoded;charset=UTF-8' },
  [Symbol(rawHeaderNames)]: Map(1) { 'content-type' => 'content-type' }
} +1ms
  xhr:request POST /login emitting the "request" event for 1 listener(s)... +0ms
  async-event-emitter:emit emitting "request" event... +0ms
  async-event-emitter:openListenerQueue opening "request" listeners queue... +0ms
  async-event-emitter:openListenerQueue no queue found, creating one... +0ms
  async-event-emitter:emit appending a one-time cleanup "request" listener... +0ms
  async-event-emitter:openListenerQueue opening "request" listeners queue... +0ms
  async-event-emitter:openListenerQueue returning an exising queue: [] +0ms
  async-event-emitter:on awaiting the "request" listener... +11ms
  xhr:request POST /login awaiting mocked response... +2ms
  async-event-emitter:on "TypeError [ERR_INVALID_ARG_TYPE]: The "input" argument must be an instance of ArrayBuffer or ArrayBufferView. Received an instance of URLSearchParams" listener has rejected! +13ms
  xhr:request POST /login middleware function threw an exception! TypeError: The "input" argument must be an instance of ArrayBuffer or ArrayBufferView. Received an instance of URLSearchParams
    at new NodeError (node:internal/errors:371:5)
    at TextDecoder.decode (node:internal/encoding:413:15)
    at Object.decodeBuffer (/Users/mark/projects/cra-msw/node_modules/@mswjs/interceptors/src/utils/bufferUtils.ts:11:18)
    at RestRequest. (/Users/mark/projects/cra-msw/node_modules/@mswjs/interceptors/src/IsomorphicRequest.ts:59:12)
    at step (/Users/mark/projects/cra-msw/node_modules/@mswjs/interceptors/lib/IsomorphicRequest.js:33:23)
    at Object.next (/Users/mark/projects/cra-msw/node_modules/@mswjs/interceptors/lib/IsomorphicRequest.js:14:53)
    at /Users/mark/projects/cra-msw/node_modules/@mswjs/interceptors/lib/IsomorphicRequest.js:8:71
    at new Promise ()
    at Object..__awaiter (/Users/mark/projects/cra-msw/node_modules/@mswjs/interceptors/lib/IsomorphicRequest.js:4:12)
    at RestRequest.Object..IsomorphicRequest.text (/Users/mark/projects/cra-msw/node_modules/@mswjs/interceptors/lib/IsomorphicRequest.js:73:16) {
  code: 'ERR_INVALID_ARG_TYPE'
} +12ms
  xhr:request POST /login trigger "error" (1) +8ms
  xhr:request POST /login resolve listener for event "error" +1ms
  xhr:request POST /login abort +0ms
  xhr:request POST /login readyState change 1 -> 0 +0ms
  xhr:request POST /login trigger "abort" (0) +0ms
  xhr:request POST /login resolve listener for event "abort" +0ms
  async-event-emitter:emit cleaned up "request" listeners queue! +27ms
  setup-server:dispose disposing the interceptor... +0ms
  setup-server retrieved global instance: BatchInterceptor +49ms
  setup-server cleared global instance! setup-server +0ms
  setup-server:dispose global symbol deleted: undefined +0ms
  setup-server:dispose disposing of 2 subscriptions... +0ms
  http:dispose disposing the interceptor... +0ms
  http retrieved global instance: ClientRequestInterceptor +51ms
  http cleared global instance! http +0ms
  http:dispose global symbol deleted: undefined +0ms
  http:dispose disposing of 2 subscriptions... +0ms
  http:setup native "http" module restored! +51ms
  http:setup native "https" module restored! +0ms
  http:dispose disposed of all subscriptions! 0 +0ms
  async-event-emitter:deactivate removing all listeners... +0ms
  async-event-emitter:removeAllListeners event: undefined +0ms
  async-event-emitter:removeAllListeners cleared the listeners queue! Map(0) {} +0ms
  async-event-emitter:deactivate set state to: DEACTIVATED +0ms
  http:dispose destroyed the listener! +0ms
  xhr:dispose disposing the interceptor... +0ms
  xhr retrieved global instance: XMLHttpRequestInterceptor +50ms
  xhr cleared global instance! xhr +0ms
  xhr:dispose global symbol deleted: undefined +0ms
  xhr:dispose disposing of 1 subscriptions... +0ms
  xhr:setup native "XMLHttpRequest" module restored! XMLHttpRequest +51ms
  xhr:dispose disposed of all subscriptions! 0 +1ms
  async-event-emitter:deactivate removing all listeners... +0ms
  async-event-emitter:removeAllListeners event: undefined +0ms
  async-event-emitter:removeAllListeners cleared the listeners queue! Map(0) {} +0ms
  async-event-emitter:deactivate set state to: DEACTIVATED +0ms
  xhr:dispose destroyed the listener! +0ms
  setup-server:dispose disposed of all subscriptions! 0 +2ms
  async-event-emitter:deactivate removing all listeners... +0ms
  async-event-emitter:removeAllListeners event: undefined +0ms
  async-event-emitter:removeAllListeners cleared the listeners queue! Map(0) {} +0ms
  async-event-emitter:deactivate set state to: DEACTIVATED +0ms
  setup-server:dispose destroyed the listener! +0ms
 FAIL  src/Login.test.ts
  fetch
    ✕ can login (35 ms)

  ● fetch › can login

    expect(received).resolves.toBeInstanceOf()

    Received promise rejected instead of resolved
    Rejected to value: [TypeError: Network request failed]

      2 | describe('fetch', ()=>{
      3 |   it('can login', async ()=>{
    > 4 |     await expect(fetch('/login', {
        |           ^
      5 |       method: 'POST',
      6 |       body: new URLSearchParams('username=admin&password=admin')
      7 |     })).resolves.toBeInstanceOf(Response);

      at expect (node_modules/expect/build/index.js:178:15)
      at Object. (src/Login.test.ts:4:11)
      at TestScheduler.scheduleTests (node_modules/@jest/core/build/TestScheduler.js:333:13)
      at runJest (node_modules/@jest/core/build/runJest.js:404:19)
      at _run10000 (node_modules/@jest/core/build/cli/index.js:320:7)
      at runCLI (node_modules/@jest/core/build/cli/index.js:173:3)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        0.834 s, estimated 1 s
Ran all test suites.

Expected behavior

A request body of type URLSearchParameters should be able to be read in a handler. I suggest via request.text(), which would return the value of URLSearchParameters.toString().

kettanaito commented 2 years ago

Hey, @markdon. Thanks for reporting this.

We are working on improving how request bodies are read, and this use case is going to the list of things we need to account for. Meanwhile, you can still use req.body if that used to work previously. The logic there should remain the same.

markdon commented 2 years ago

Actually, I only spent the time investigating this after req.body.toString() started throwing a very similar error :( Just accessing req.body at all throws the same error.

I can refactor the few of my tests this affected to move to latest msw.

Debug log snippet 
xhr:request POST /login middleware function threw an exception! TypeError: The "input" argument must be an instance of ArrayBuffer or ArrayBufferView. Received an instance of URLSearchParams
    at new NodeError (node:internal/errors:371:5)
    at TextDecoder.decode (node:internal/encoding:413:15)
    at decodeBuffer (/Users/mark/projects/cra-msw/node_modules/@mswjs/interceptors/src/utils/bufferUtils.ts:11:18)
    at RestRequest.body (/Users/mark/projects/cra-msw/node_modules/msw/src/utils/request/MockedRequest.ts:111:18)
    at /Users/mark/projects/cra-msw/src/setupTests.ts:8:44
    at /Users/mark/projects/cra-msw/node_modules/msw/src/handlers/RequestHandler.ts:232:55
    at RestHandler.run (/Users/mark/projects/cra-msw/node_modules/msw/src/handlers/RequestHandler.ts:215:34)
    at /Users/mark/projects/cra-msw/node_modules/msw/src/utils/getResponse.ts:50:34
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at getResponse (/Users/mark/projects/cra-msw/node_modules/msw/src/utils/getResponse.ts:41:18) {
  code: 'ERR_INVALID_ARG_TYPE'
} +12ms
kettanaito commented 2 years ago

Oh, so the reason is not to read the body but to construct it. He'd have to support that.

We're using something called IsomorphicRequest abstract that allows us to handle different request body types in the same manner. That abstraction only accepts ArrayBuffer as request body (which we want to keep) but the consumer (MSW in this case) can transform the body however they need to eventually provide the buffer.

So, I'd imagine this support to be:

  1. Check when the body is URLSearchParams.
  2. Serialize that somehow.
  3. Turn it into an ArrayBuffer.
  4. Make sure it's de-serializable as well.

Contributions are welcome!

markdon commented 2 years ago

Sounds pretty straight forward to implement! I would love to contribute, if I hadn't just signed up to a project with a tight timeline for the next few months.

I will check back later!

ianldgs commented 2 years ago

My workaround, which was actually for FormData, is the following:

jest.spyOn(global, "FormData").mockImplementation(mockFormDataForMSW);
jest.mock("util", mockNodeUtilForMSW);

function mockFormDataForMSW() {
  return new FormDataFakeArrayBuffer(0) as unknown as FormData;
}

function mockNodeUtilForMSW() {
  const real = jest.requireActual("util") as typeof import("util");
  return {
    ...real,
    TextDecoder: class extends real.TextDecoder {
      decode(
        input?: NodeJS.ArrayBufferView | ArrayBuffer | null,
        options?: { stream?: boolean | undefined },
      ): string {
        if (input instanceof FormDataFakeArrayBuffer) return input.toString();

        return super.decode(input, options);
      }
    },
  };
}

class FormDataFakeArrayBuffer extends ArrayBuffer {
  private params = new URLSearchParams();

  append(param: string, value: string) {
    return this.params.append(param, value);
  }

  toString() {
    return this.params.toString();
  }
}

For reference/SEO, the error was:

The input argument must be an instance of ArrayBuffer or ArrayBufferView. Received an instance of FormData

or

The input argument must be an instance of ArrayBuffer or ArrayBufferView. Received an instance of URLSearchParams
markdon commented 2 years ago

I'm not sure why I didn't consider this before but surely the easiest workaround (for URLSearchParams) is to create the request with a string instead of URLSearchParams object. e.g. use URLSearchParams.toString() for the body when calling fetch.

The MSW handler will always need to parse the text body with new URLSearchParams(await body.text()) as there is no Request method that gets the body as a URLSearchParams object.

ddolcimascolo commented 2 years ago

Same issue as mine in #1327 but with FormData bodies. msw just plain broke reading request bodies in v0.44.0.

New features keep getting added in the 0.x line so that it's totally OK to break stuff from a semver perspective. And bugs are not fixed.

sadness

chawax commented 2 years ago

Same problem here.

chawax commented 2 years ago

I used Axios instead of Fetch and it works !

kettanaito commented 2 years ago

@ddolcimascolo, a true sadness is having a 7M/week downloaded library used by FAANG and having to develop it over weekends instead of spending time with one's family (which I've promised myself not to do anymore, thus you will have issues that I don't have time/desire to work on). But hey, you can take a step towards resolving that sadness, here you go.

This is going to be fixed by #1436 so I'm not going to do any work around this until that API lands. If you're blocked by this, consider contributing, I'd do my best to review your code and release it.

ddolcimascolo commented 2 years ago

Yeah. My message above was itself sad, sorry about that, probably written while closing yet another upgrade merge request on MSW that I can't merge because of this blocker. You do a great job in maintaining OSS projects, keep up the great work!

Since my message above I've contributed a fix, even though you implemented #1436 in the end which is a really nice addition.

Cheers, David

markdon commented 2 years ago

I will check back later!

It looks like a lot of progress has been made towards the Fetch API update that replaces the related code and fixes this issue. There's a beta available to try.

I had a quick look at making a fix for the current release of msw but @mswjs/interceptors has already moved on from the related IsomorphicRequest class and the Fetch API update looks pretty close to done anyway.

DusanPausly commented 1 year ago

Hey, just little question from me. I am using latest msw, node and react-query@^3.39.2. And I am trying to connect with a Keycloak (newest stable version) for fresh token before running tests. I have to use URLSearchParams formatting in body to construct fetch and I am getting this mentioned error from msw:

"The "input" argument must be an instance of ArrayBuffer or ArrayBufferView. Received an instance of URLSearchParams."

I guess, #1436 is fixing it also for this case, right? Otherwise what is the status of completition @kettanaito ?

erperejildo commented 1 year ago

https://stackoverflow.com/questions/75535744/cant-get-body-using-msw-and-redux-toolkit/75543115#75543115

kettanaito commented 1 year ago

@DusanPausly, correct. The current version of MSW doesn't support URLSearchParams as the request body. The Fetch API feature will add that support.

kettanaito commented 1 year ago

Released: v2.0.0 🎉

This has been released in v2.0.0!

Make sure to always update to the latest version (npm i msw@latest) to get the newest features and bug fixes.

Predictable release automation by @ossjs/release.