search

microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
64.96k stars 3.54k forks source link

[BUG] `Content-Length` header missing from `page.request.response.AllHeadersAsync` #28152

Closed lonix1 closed 9 months ago

lonix1 commented 9 months ago

System info

Source code

page.Request += (_, request) => {
  Console.WriteLine(request.Url);
  foreach (var header in await response.AllHeadersAsync())
     Console.WriteLine($" {header.Key}: {string.Join(", ", header.Value)}");
};

Example:

https://fonts.googleapis.com/css?family=Barlow:400,400i,700,700i,600,600i,900,900i,800,800i%7CNunito+Sans:400,400i,700,700i%7CBarlow+Semi+Condensed:600,600i%7CMontserrat:400,400i,700,700i,600,600i%7CBarlow+Condensed:400,400i,800,800i&subset=latin,latin-ext&display=auto
access-control-allow-origin: *
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
cache-control: private, max-age=86400, stale-while-revalidate=604800
content-encoding: gzip
content-type: text/css; charset=utf-8
cross-origin-opener-policy: same-origin-allow-popups
cross-origin-resource-policy: cross-origin
date: Wed, 15 Nov 2023 09:03:10 GMT
expires: Wed, 15 Nov 2023 09:03:10 GMT
last-modified: Wed, 15 Nov 2023 09:03:10 GMT
link: <https://fonts.gstatic.com>; rel=preconnect; crossorigin
server: ESF
strict-transport-security: max-age=31536000
timing-allow-origin: *
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 0

With curl, one can see that the header is being served:

$ curl --head https://fonts.googleapis.com/css?family=Barlow:400,400i,700,700i,600,600i,900,900i,800,800i%7CNunito+Sans:400,400i,700,700i%7CBarlow+Semi+Condensed:600,600i%7CMontserrat:400,400i,700,700i,600,600i%7CBarlow+Condensed:400,400i,800,800i&subset=latin,latin-ext&display=auto
HTTP/2 200 
content-type: text/css; charset=utf-8         # <-----
access-control-allow-origin: *
timing-allow-origin: *
content-length: 5563                          # <-----
expires: Wed, 15 Nov 2023 09:17:52 GMT
date: Wed, 15 Nov 2023 09:17:52 GMT
cache-control: private, max-age=86400
cross-origin-opener-policy: same-origin-allow-popups
cross-origin-resource-policy: cross-origin
server: ESF
x-xss-protection: 0
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Steps

Expected

Output should include Content-Length header.

Actual

Output excludes Content-Length header.

NOTES The above url was just an example, it happens for other cases too (it's not an edge case).

Content-Length is missing whenever content-type is text/css; charset=utf-8 or text/html; charset=UTF-8 and probably others too.

mxschmitt commented 9 months ago

Can you provide us a self contained reproducible? I tried the following but it was working for me:

import { chromium } from 'playwright';

(async () => {
  const context = await chromium.launch({
    headless: false,
  });
  const page = await context.newPage();
  page.on('response', async response => {
    const contentType = (await response.allHeaders())['content-type'];
    if (!contentType)
      throw new Error('No content-type in response for url: ' + response.url());
    console.log(contentType);
  });
  await page.goto('https://fonts.google.com/specimen/Roboto');
  await page.waitForTimeout(100_000);
  await context.close();
  await browser.close();
})();
lonix1 commented 9 months ago

Full program:

using Microsoft.Playwright;

public class Program
{

  public static async Task Main(params string[] args)
  {
    using var playwright = await Playwright.CreateAsync();
    await using var browser = await playwright.Chromium.LaunchAsync();
    var page = await browser.NewPageAsync();

    page.Request += (_, request) => Handler(request);

    await page.GotoAsync("https://getbootstrap.com");
    await Task.Delay(10_000);
  }

  public static async void Handler(IRequest request)
  {
    var response = await request.ResponseAsync();
    if (response == null) return;
    var headers = await response.AllHeadersAsync();

    var contentLength = headers.ContainsKey("content-length") ? headers["content-length"] : "";
    var contentType   = headers.ContainsKey("content-type")   ? headers["content-type"]   : "";

    Console.WriteLine($"{request.Url}\n  content-type   = {contentType}\n  content-length = {contentLength}");
  }

}

Results:

https://getbootstrap.com/
  content-type   = text/html; charset=utf-8
  content-length =
https://getbootstrap.com/docs/5.3/dist/css/bootstrap.min.css
  content-type   = text/css; charset=utf-8
  content-length =
https://getbootstrap.com/docs/5.3/assets/css/docs.css
  content-type   = text/css; charset=utf-8
  content-length =
https://getbootstrap.com/docs/5.3/dist/js/bootstrap.bundle.min.js
  content-type   = application/javascript; charset=utf-8
  content-length =
https://getbootstrap.com/docs/5.3/assets/brand/bootstrap-logo-shadow.png
  content-type   = image/png
  content-length = 46142
https://cdn.jsdelivr.net/npm/@docsearch/css@3
  content-type   = text/css; charset=utf-8
  content-length = 3388
https://cdn.jsdelivr.net/npm/@docsearch/js@3
  content-type   = application/javascript; charset=utf-8
  content-length = 31966
https://getbootstrap.com/docs/5.3/assets/js/color-modes.js
  content-type   = application/javascript; charset=utf-8
  content-length =
https://getbootstrap.com/docs/5.3/assets/js/docs.min.js
  content-type   = application/javascript; charset=utf-8
  content-length =
https://cdn.usefathom.com/script.js
  content-type   = application/javascript
  content-length =
https://getbootstrap.com/docs/5.3/assets/img/webpack.svg
  content-type   = image/svg+xml
  content-length =
https://getbootstrap.com/docs/5.3/assets/img/parcel.png
  content-type   = image/png
  content-length = 6003
https://getbootstrap.com/docs/5.3/assets/img/vite.svg
  content-type   = image/svg+xml
  content-length =
https://www.google-analytics.com/analytics.js
  content-type   = text/javascript
  content-length = 20994
https://cdn.carbonads.com/carbon.js?serve=CKYIKKJL&placement=getbootstrapcom
  content-type   = application/javascript
  content-length = 9644
https://cdn.usefathom.com/?h=https%3A%2F%2Fgetbootstrap.com&p=%2F&r=&sid=ITUSEYJG&qs=%7B%7D&cid=74340965
  content-type   = text/html; charset=UTF-8
  content-length = 7
https://www.google-analytics.com/j/collect?v=1&_v=j101&aip=1&a=2006215410&t=pageview&_s=1&dl=https%3A%2F%2Fgetbootstrap.com%2F&ul=en-us&de=UTF-8&dt=Bootstrap%20%C2%B7%20The%20most%20popular%20HTML%2C%20CSS%2C%20and%20JS%20library%20in%20the%20world.&sd=24-bit&sr=1280x720&vp=1280x720&je=0&_u=YEBAAAABAAAAACAAI~&jid=2064396960&gjid=1054847076&cid=1451063877.1700061873&tid=UA-146052-10&_gid=1486541707.1700061873&_r=1&_slc=1&z=1072701052
  content-type   = text/plain
  content-length = 15
https://www.googletagmanager.com/gtag/js?id=G-W2VMDSX0CE&cx=c&_slc=1
  content-type   = application/javascript; charset=UTF-8
  content-length = 85416
https://srv.carbonads.net/ads/CKYIKKJL.json?segment=placement:getbootstrapcom
  content-type   = application/json; charset=utf-8
  content-length = 742
https://www.google-analytics.com/g/collect?v=2&tid=G-W2VMDSX0CE&gtm=45je3b81v9126315764&_p=1700061873511&gcd=11l1l1l1l2&dma=0&ul=en-us&sr=1280x720&cid=1451063877.1700061873&ir=1&uaa=x86&uab=64&uafvl=HeadlessChrome%3B119.0.6045.9%7CChromium%3B119.0.6045.9%7CNot%253FA_Brand%3B24.0.0.0&uamb=0&uam=&uap=Linux&uapv=5.15.0&uaw=0&_eu=EBAI&_s=1&dl=https%3A%2F%2Fgetbootstrap.com%2F&dt=Bootstrap%20%C2%B7%20The%20most%20popular%20HTML%2C%20CSS%2C%20and%20JS%20library%20in%20the%20world.&sid=1700061874&sct=1&seg=0&en=page_view&_fv=1&_ss=1&_ee=1&tfd=2280
  content-type   = text/plain
  content-length = 0
https://srv.carbonads.net/static/30242/d0e1cb2a7998823f9119a073e8cce55b669a3936
  content-type   = image/png
  content-length =

Here is another run, for the url you used, https://fonts.google.com/specimen/Roboto:

https://fonts.google.com/specimen/Roboto
  content-type   = text/html; charset=utf-8
  content-length =
https://www.gstatic.com/_/fonts/_/ss/k=fonts.app.B4dOlbCf2eY.L.W.O/am=CAM/d=0/br=1/rs=AE60u5eEoH3uiStWw81mOiOYBiHgYmBpQQ
  content-type   = text/css; charset=UTF-8
  content-length = 42136
https://www.gstatic.com/_/fonts/_/js/k=fonts.app.en_US.fVeX5sheqyQ.2018.O/am=CAM/d=1/br=1/rs=AE60u5eDCEG27Zq1axZeEgkcrZ-caK8dFw/m=core
  content-type   = text/javascript; charset=UTF-8
  content-length = 285101
https://www.googletagmanager.com/gtag/js?id=G-XPW1QSKFW4
  content-type   = application/javascript; charset=UTF-8
  content-length = 91650
https://www.gstatic.com/feedback/js/help/prod/service/lazy.min.js
  content-type   = text/javascript
  content-length = 36058
https://ssl.gstatic.com/external_hosted/lottie/lottie_light.js
  content-type   = text/javascript
  content-length = 35295
https://www.gstatic.com/_/fonts/_/js/k=fonts.app.en_US.fVeX5sheqyQ.2018.O/am=CAM/d=1/exm=core/ed=1/br=1/rs=AE60u5eDCEG27Zq1axZeEgkcrZ-caK8dFw/m=KHJXJ
  content-type   = text/javascript; charset=UTF-8
  content-length = 30029
https://www.gstatic.com/_/fonts/_/js/k=fonts.app.en_US.fVeX5sheqyQ.2018.O/am=CAM/d=1/exm=KHJXJ,core/ed=1/br=1/rs=AE60u5eDCEG27Zq1axZeEgkcrZ-caK8dFw/m=AL0Bn
  content-type   = text/javascript; charset=UTF-8
  content-length = 38839
https://fonts.gstatic.com/s/googlesans/v29/4UaGrENHsxJlGDuGo1OIlL3Owp4.woff2
  content-type   = font/woff2
  content-length = 39516
https://www.google-analytics.com/g/collect?v=2&tid=G-XPW1QSKFW4&gtm=45je3b81v876567358&_p=1700062056303&gcd=11l1l1l1l1&dma=0&cid=859968966.1700062057&ul=en-us&sr=1280x720&uaa=x86&uab=64&uafvl=HeadlessChrome%3B119.0.6045.9%7CChromium%3B119.0.6045.9%7CNot%253FA_Brand%3B24.0.0.0&uamb=0&uam=&uap=Linux&uapv=5.15.0&uaw=0&_s=1&sid=1700062057&sct=1&seg=0&dl=https%3A%2F%2Ffonts.google.com%2Fspecimen%2FRoboto&dt=Roboto%20-%20Google%20Fonts&en=page_view&_fv=1&_nsi=1&_ss=1&_ee=1&ep.anonymize_ip=true&ep.referrer=&tfd=2669
  content-type   = text/plain
  content-length = 0
https://fonts.google.com/metadata/fonts
  content-type   = application/json; charset=utf-8
  content-length =
https://fonts.google.com/metadata/fonts/Roboto
  content-type   = application/json; charset=utf-8
  content-length =
https://fonts.google.com/metadata/lang
  content-type   = application/json; charset=utf-8
  content-length =
https://fonts.google.com/sampletext?family=Roboto
  content-type   = application/json; charset=utf-8
  content-length =
https://fonts.google.com/metadata/page
  content-type   = application/json; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%400%2C300&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%401%2C300&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%401%2C500&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%400%2C400&directory=3&display=block
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%400%2C900&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%401%2C900&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%400%2C500&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%400%2C100&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%401%2C100&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%401%2C400&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.gstatic.com/s/img/knowledge/modules/choosing_type/lessons/choosing_reliable_typefaces/images/thumbnail_411126311.svg
  content-type   = image/svg+xml
  content-length = 657
https://fonts.gstatic.com/s/img/knowledge/modules/choosing_type/lessons/a_checklist_for_choosing_type/images/thumbnail_439626857.svg
  content-type   = image/svg+xml
  content-length = 9289
https://fonts.gstatic.com/s/img/knowledge/modules/choosing_type/lessons/emotive_considerations_for_choosing_typefaces/images/thumbnail_411126311.svg
  content-type   = image/svg+xml
  content-length = 46121
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%401%2C700&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.googleapis.com/css2?family=Roboto%3Aital%2Cwght%400%2C700&directory=3&display=block&text=%20Wacdefghinorsty
  content-type   = text/css; charset=utf-8
  content-length =
https://fonts.gstatic.com/l/font?kit=KFOlCnqEu92Fr1MmSU5vBgUtCpaT9uwrHRzcaDdwEUesIw&skey=11ce8ad5f54705ca&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3220
https://fonts.gstatic.com/l/font?kit=KFOjCnqEu92Fr1Mu51TjARc4GNHSwuktHx7SZjF2DVuqKZJ1&skey=8f644060176e1f7e&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3472
https://fonts.gstatic.com/l/font?kit=KFOlCnqEu92Fr1MmYUtvBgUtCpaT9uwrHRzcaDdwEUesIw&skey=934406f772f9777d&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3304
https://fonts.gstatic.com/l/font?kit=KFOlCnqEu92Fr1MmEU9vBgUtCpaT9uwrHRzcaDdwEUesIw&skey=ee881451c540fdec&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3240
https://fonts.gstatic.com/l/font?kit=KFOjCnqEu92Fr1Mu51TLBBc4GNHSwuktHx7SZjF2DVuqKZJ1&skey=b80be3241fe40325&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3476
https://fonts.gstatic.com/l/font?kit=KFOkCnqEu92Fr1MmgWxKMScdfaCR8e0oHBPdbzZtEECh&skey=5473b731ec7fc9c1&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3180
https://fonts.gstatic.com/l/font?kit=KFOjCnqEu92Fr1Mu51S7ABc4GNHSwuktHx7SZjF2DVuqKZJ1&skey=c985e17098069ce0&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3504
https://fonts.gstatic.com/l/font?kit=KFOkCnqEu92Fr1Mu52xKMScdfaCR8e0oHBPdbzZtEECh&skey=c608c610063635f9&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3532
https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu4mxK.woff2
  content-type   = font/woff2
  content-length = 15744
https://fonts.gstatic.com/l/font?kit=KFOiCnqEu92Fr1Mu51QrIzIPOuGl9OsqHh3TaTBxDEarLp8&skey=8f53aa2e7deadc4a&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3340
https://fonts.gstatic.com/l/font?kit=KFOjCnqEu92Fr1Mu51TzBhc4GNHSwuktHx7SZjF2DVuqKZJ1&skey=dd030d266f3beccc&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3340
https://fonts.gstatic.com/l/font?kit=KFOlCnqEu92Fr1MmWUlvBgUtCpaT9uwrHRzcaDdwEUesIw&skey=c06e7213f788649e&v=v30
  content-type   = text/html; charset=utf-8
  content-length = 3160
https://www.google-analytics.com/g/collect?v=2&tid=G-XPW1QSKFW4&gtm=45je3b81v876567358&_p=1700062056303&gcd=11l1l1l1l1&dma=0&cid=859968966.1700062057&ul=en-us&sr=1280x720&uaa=x86&uab=64&uafvl=HeadlessChrome%3B119.0.6045.9%7CChromium%3B119.0.6045.9%7CNot%253FA_Brand%3B24.0.0.0&uamb=0&uam=&uap=Linux&uapv=5.15.0&uaw=0&_eu=AAAC&_s=2&dp=%2Fspecimen%2FRoboto&sid=1700062057&sct=1&seg=1&dl=https%3A%2F%2Ffonts.google.com%2Fspecimen%2FRoboto&dt=Roboto%20-%20Google%20Fonts&en=page_view&_ee=1&tfd=7755
  content-type   = text/plain
  content-length = 0

Notice that many content-length headers were missing. This occurs for all sorts of content types, it isn't predictable as far as I can tell.

mxschmitt commented 9 months ago

These requests are served over http3. Http3 does not rely on the content-length header to read the body, thats why its missing. If you open Chrome DevTools and navigate to it, you'll see also that this header is missing.

If you try to measure the network traffic, I recommend a proxy.

Closing since this is by spec.

lonix1 commented 9 months ago

Thanks, you taught me something new! :+1:

lonix1 commented 9 months ago

Background info for anyone who needs it.