bug: CapacitorHTTP no longer passes user-agent header in fetch requests on Android

[fabian-meisinger]

Bug Report

Capacitor Version

   Capacitor Doctor   

Latest Dependencies:

  @capacitor/cli: 5.5.1
  @capacitor/core: 5.5.1
  @capacitor/android: 5.5.1
  @capacitor/ios: 5.5.1

Installed Dependencies:

  @capacitor/cli: 5.5.1
  @capacitor/android: 5.5.1
  @capacitor/ios: 5.5.1
  @capacitor/core: 5.5.1

[success] Android looking great! 👌
[error] Xcode is not installed

Platform(s)

Android

Current Behavior

After updating to 5.5.1 (from 5.3.0) fetch removes any headers from the "forbidden header" list.

Expected Behavior

Like before, it should allow free control over headers.

Code Reproduction

Simply start any app with the http plugin enabled and use a developer console to run a fetch call: fetch("https://example.com", { headers: {"User-Agent": "test" }}) The console log from the native bridge will show that options.headers is empty.

Other Technical Details

none

Additional Context

The issue was introduced due to changes in this pull request. The line in question can be found here. Before the changes it would take the headers directly from the passed options. After the change it will take them from the request object, which filters forbidden header names. Looking at the code, it seems sensible to revert the line in question to the code from before and instead check if resource is of type Request to copy any headers from there before applying the ones from options.

try {
    const { body, method } = request;
    const {
      data: requestData,
      type,
      headers,
    } = await convertBody(body || undefined);

    let requestHeaders = {}
    if (resource instanceof Request) {
      requestHeaders = Object.fromEntries(resource.headers.entries());
    }
    let optionHeaders = options.headers;
    if (options.headers instanceof Headers) {
      optionHeaders = Object.fromEntries(options.headers.entries());
    }
    const nativeResponse: HttpResponse = await cap.nativePromise(
      'CapacitorHttp',
      'request',
      {
        url: request.url,
        method: method,
        data: requestData,
        dataType: type,
        headers: {
          ...headers,
          ...requestHeaders,
          ...optionHeaders,
        },
      },
    );

[aeharding]

I have the same problem - https://github.com/ionic-team/capacitor/issues/7013

[ionitron-bot[bot]]

This issue needs more information before it can be addressed. In particular, the reporter needs to provide a minimal sample app that demonstrates the issue. If no sample app is provided within 15 days, the issue will be closed. Please see the Contributing Guide for how to create a Sample App. Thanks! Ionitron 💙

[fabian-meisinger]

Here you go: https://github.com/fabian-meisinger/capacitor-http-header-issue-repro Simply clone the repo and run the following commands:

npm i
npm run build
npx cap sync android
npx cap open android

Then launch the app, connect with chrome dev tools and click on the "send request" button. Afterwards you will see that the options passed to fetch contain a user-agent header which will get filtered before the request is passed to the native plugin.

The code for the request is located in src/js/capacitor-welcome.js#L96-L103

[ionitron-bot[bot]]

This issue has been labeled as type: bug. This label is added to issues that that have been reproduced and are being tracked in our internal issue tracker.

[jcesarmobile]

Marking it as bug, but not 100% sure this is a bug, as it has never worked on web platform. Will discuss with the team.

In version 5.7.1 there was an additional change where the GET requests go through a "proxy", where the fetch request is the browser fetch, which doesn't allow to change the user-agent, so I don't think this is going to be possible for GET requests.

[fabian-meisinger]

I'd say it is a bug for sure. The whole point of the CapacitorHttp plugin is to not get restricted by the browser sandbox by offering a native way to send request. So of course it won't behave the same on the web platform, where you do not have a native plugin api. The situations where you have to make use of this plugin don't usually involve the web platform, so trying to compare the behaviour is kind of a moot point.

If this plugin started to compare everything to the web platform and were to impose the same browser restrictions, what would it even do that's different from the original browser fetch function? I'd be forced to write and maintain my own native functions and at that point I'd honestly question why I even use Capacitor/Ionic, when I have to write my own native code anyway.

[jcesarmobile]

It should work (and it works) if you use the CapacitorHttp API directly, but not so sure about the patched fetch, the patched fetch is supposed to work exactly the same way the web fetch works, except for the CORS issues.

BTW, since fetch uses the WebView user agent, you can set it with overrideUserAgent or appendUserAgent config options , but that's read at build time, so all requests will use the same, you can't change it by request.

EDIT: I've done more testing and Safari allows to change the user agent, so on iOS it works. Also works on web if using Safari or Firefox. But doesn't work on web if using Chrome, nor on Android since the WebView uses Chrome/Chromium under the hood.

[fabian-meisinger]

I'll have to give that a try then. However, the documentation on the plugin page makes it sound as if the patched fetch should behave the same as using the CapacitorHttp API directly and is only there as a convenience so you don't have to replace all fetch calls in your code. So maybe it's worth mentioning the difference?

I do use overrideUserAgent, but at least in the version of capacitor I started the project with, it did not work when used together with the old community http plugin and I haven't tried it without setting the header directly since then. Will give it a try when I get around to update to the latest version.

And yes, on iOS it works fine for me too, so I only reported it for Android. Guess I should have mentioned that in my initial post. Sorry about that.

[aeharding]

the patched fetch is supposed to work exactly the same way the web fetch works, except for the CORS issues.

Maybe the best course of action (instead of making exceptions for how the behavior is different than the web fetch api) is to treat the Capacitor fetch api like the Node or Deno (backend) fetch API, instead of browser API. After all, it bypasses CORS.

FWIW, this same conversation was also had in Deno and Node fetch API projects in recent years:

• https://github.com/denoland/deno/issues/6868
• https://github.com/nodejs/undici/issues/1262

(TLDR AFAIK of above issues: You can set all headers except for host)

[MirazMac]

Any update on this? It seems patched requests are stripping headers like Cookie, Referer etc. But using CapacitorHttp directly works. Any way to keep them? Perhaps this can be implemented inside CapacitorHttp? Something like if we specifiy

X-Custom-Cookie X-Custom-Referer

It will be transformed into just

"Cookie" or "Referer" inside the Capacitor interceptor?