dart-lang / http

A composable API for making HTTP requests in Dart.
https://pub.dev/packages/http
BSD 3-Clause "New" or "Revised" License
1.02k stars 354 forks source link

The headers default to adding "Dart3.0" to the user-agent, resulting in some image sources detecting it as Dart's user agent and returning it directly to Forbidden #990

Open popsams opened 1 year ago

popsams commented 1 year ago

The headers default to adding "Dart3.0" to the user-agent, resulting in some image sources detecting it as Dart's user agent and returning it directly to Forbidden. Can the author optimize it? The headers are what they pass, so don't add other default content by default. Thank you

SamJakob commented 1 year ago

The user agent Dart/3.0 (dart:io) (in your case) is set in dart:io which this package uses under-the-hood when running on platforms that have dart:io (so macOS/Windows/Linux/Android/iOS as opposed to a browser). When running in the browser, it will use the browser's user agent by default (also, some browsers prevent setting the user agent - my understanding is that User-Agent used to be a blocked/protected header but isn't anymore) so it might not be feasible to add this as a general option (as it would then be expected to work consistently on both dart:io platforms and the web).

With dart:io there wouldn't be a 'default user agent' that this package or the dart:io library can inherit like there is with a browser (because the HTTP client stack is Dart's own but also because native platforms don't often set their own user agent), so there's nothing else to get a default from. On the other hand, not sending a user agent at all is very often problematic. For example, anti-DDOS or firewall services like CloudFlare generally block requests without a user agent (albeit depending on the settings for the site), so in dart:io the user agent Dart/<version> (dart:io) is set as a reasonable default (but some services might not recognize this user agent or flag it as suspicious due to relatively small usage compared to ordinary web browsers).

A simple workaround (if you're not targeting Browsers - i.e., Flutter for Web) is to set one explicitly - that is; just create the IOClient with a custom user agent (e.g., one from this list of most common user agents) and pass it into a dart:http client. Like so:

import 'package:http/http.dart';

// Add these imports - but note they won't work on browsers (i.e., Flutter for Web)
import 'dart:io';
import 'package:http/io_client.dart';

/// Create a package:http Client with a custom [userAgent].
/// Alternatively, set [userAgent] to null to not send any user agent.
Client createCustomClient({
  // User agent for Chrome on Windows - should avoid any issues.
  final String? userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
      "AppleWebKit/537.36 (KHTML, like Gecko) "
      "Chrome/114.0.0.0 "
      "Safari/537.36",
}) {
  // Initialize a (lower-level) HttpClient from dart:io with a custom user agent.
  // (Don't use this directly).
  final innerClient = HttpClient()
    ..userAgent = userAgent;

  // Pass the lower-level client into an IOClient object (from the http package).
  return IOClient(innerClient);
}

Future<void> main() async {
  // Place the following where you initialize your client in your code.

  final client = createCustomClient();

  // You can now use `client`. For example:

  // echo.hoppscotch.io is a service that echoes back the request headers
  // and body as a JSON response.
  final response = await client.get(Uri.parse('https://echo.hoppscotch.io'));

  // The headers.user-agent property should match the value passed into
  // `createCustomClient`.
  // (Or it'll be the default in `createCustomClient`).
  print(response.body);
}

If you instead use get, post, etc., as opposed to calling them on a specific Client instance, you can use runWithClient. Wrap as much of your code (probably whatever you call in your main method) with a call to runWithClient and use the createCustomClient provided above.

If you're using Flutter, you can just wrap your call to runApp like so:

// Paste the `createCustomClient` function from above.

void main() {
  runWithClient(() {
      runApp(const MyApp());
  }, () => createCustomClient());
}

Then, calls to get, post, etc., - even when not called explicitly on a client - will use your custom user agent.


If you need to support Dart/Flutter on the Web too, that's still doable but you'll need to use a conditional import - probably by placing createCustomClient in an external file and making one for web, using this one for dart:io and a stub one, and then conditionally importing the one relevant to the current platform. (Feel free to ask for further details).