[feature] zstd codec support

[tas33n]

Describe the bug when capturing networks form facebook/messenger networks, the response are always Hex data , theres no option to show as utf-8 or readble text. The devtool form chrome show the data as readble json or script data. i've also cheked in burp suite that also shows as json data but reqable always shows hex..

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. See error

Expected behavior It should show the data same as chrome dev tools shows, or at least have option to show hex and utf8 data.

Screenshots Reqable:

Chrome dev tool:

Burp Suite ((different api but same type data))

Information

• Platform: Windows
• OS: 11
• Arch: 64
• App Version: 2.19.0

[MegatronKing]

@tas33n Please attach the response headers here.

[tas33n]

response header from reqable :

:status: 200
vary: Accept-Encoding
content-encoding: zstd
content-type: text/javascript; charset=utf-8
reporting-endpoints: coop_report="https://www.facebook.com/browser_reporting/coop/?minimize=0", coep_report="https://www.facebook.com/browser_reporting/coep/?minimize=0", default="https://www.facebook.com/ajax/comet_error_reports/?device_level=mod-low&brsid=7387264710118225710", permissions_policy="https://www.facebook.com/ajax/browser_error_reports/"
report-to: {"max_age":2592000,"endpoints":[{"url":"https:\/\/www.facebook.com\/browser_reporting\/coop\/?minimize=0"}],"group":"coop_report","include_subdomains":true}, {"max_age":86400,"endpoints":[{"url":"https:\/\/www.facebook.com\/browser_reporting\/coep\/?minimize=0"}],"group":"coep_report"}, {"max_age":259200,"endpoints":[{"url":"https:\/\/www.facebook.com\/ajax\/comet_error_reports\/?device_level=mod-low&brsid=7387264710118225710"}]}, {"max_age":21600,"endpoints":[{"url":"https:\/\/www.facebook.com\/ajax\/browser_error_reports\/"}],"group":"permissions_policy"}
content-security-policy: default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com *.fbcdn.net 'unsafe-eval';script-src *.facebook.com *.fbcdn.net *.facebook.net 127.0.0.1:* 'unsafe-inline' blob: data: 'self' connect.facebook.net 'unsafe-eval' https://*.google-analytics.com *.google.com;style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline' https://fonts.googleapis.com;connect-src *.facebook.com facebook.com *.fbcdn.net *.facebook.net wss://*.facebook.com:* wss://*.whatsapp.com:* wss://*.fbcdn.net attachment.fbsbx.com ws://localhost:* blob: *.cdninstagram.com 'self' http://localhost:3103 wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d.facebook.com/ v.whatsapp.net *.fbsbx.com *.fb.com https://*.google-analytics.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com https://fonts.gstatic.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net connect.facebook.net *.carriersignal.info blob: android-webview-video-poster: *.whatsapp.net *.fb.com *.oculuscdn.com *.tenor.co *.tenor.com *.giphy.com https://paywithmybank.com/ https://*.paywithmybank.com/ https://www.googleadservices.com https://googleads.g.doubleclick.net https://*.google-analytics.com;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data: *.tenor.co *.tenor.com https://*.giphy.com;frame-src *.facebook.com *.fbsbx.com fbsbx.com data: www.instagram.com *.fbcdn.net https://paywithmybank.com/ https://*.paywithmybank.com/ https://www.googleadservices.com https://googleads.g.doubleclick.net https://www.google.com https://td.doubleclick.net *.google.com *.doubleclick.net;worker-src blob: *.facebook.com data:;block-all-mixed-content;upgrade-insecure-requests;
document-policy: force-load-at-top
permissions-policy: accelerometer=(), attribution-reporting=(self), autoplay=(), battery=(self), bluetooth=(), camera=(self), ch-device-memory=(), ch-downlink=(), ch-dpr=(), ch-ect=(), ch-rtt=(), ch-save-data=(), ch-ua-arch=(), ch-ua-bitness=(), ch-viewport-height=(), ch-viewport-width=(), ch-width=(), clipboard-read=(self), clipboard-write=(self), compute-pressure=(), display-capture=(self), encrypted-media=(self), fullscreen=(self), gamepad=*, geolocation=(self), gyroscope=(), hid=(), idle-detection=(), interest-cohort=(), keyboard-map=(), local-fonts=(), magnetometer=(), microphone=(self), midi=(), otp-credentials=(), payment=(), picture-in-picture=(self), private-state-token-issuance=(), publickey-credentials-get=(self), screen-wake-lock=(), serial=(), shared-storage=(), shared-storage-select-url=(), private-state-token-redemption=(), usb=(), usb-unrestricted=(), unload=(self), window-management=(), xr-spatial-tracking=(self);report-to="permissions_policy"
cross-origin-resource-policy: same-origin
cross-origin-embedder-policy-report-only: require-corp;report-to="coep_report"
cross-origin-opener-policy: same-origin-allow-popups;report-to="coop_report"
pragma: no-cache
cache-control: private, no-cache, no-store, must-revalidate
expires: Sat, 01 Jan 2000 00:00:00 GMT
x-content-type-options: nosniff
x-xss-protection: 0
x-frame-options: DENY
origin-agent-cluster: ?0
strict-transport-security: max-age=15552000; preload
x-fb-debug: /3JEfxMr+S6AOFf+2tPZwVf3Czv/h7geW+L36Y4808X5Lr9YtTQYFx+qeg7FjTkEjHgmq2Cpc0csDIlEE8ovZA==
date: Wed, 03 Jul 2024 04:39:57 GMT
x-fb-connection-quality: EXCELLENT; q=0.9, rtt=27, rtx=0, c=10, mss=1392, tbw=3577, tp=-1, tpl=-1, uplat=272, ullat=0
alt-svc: h3=":443"; ma=86400

[MegatronKing]

response header from reqable :

:status: 200
vary: Accept-Encoding
content-encoding: zstd
content-type: text/javascript; charset=utf-8
reporting-endpoints: coop_report="https://www.facebook.com/browser_reporting/coop/?minimize=0", coep_report="https://www.facebook.com/browser_reporting/coep/?minimize=0", default="https://www.facebook.com/ajax/comet_error_reports/?device_level=mod-low&brsid=7387264710118225710", permissions_policy="https://www.facebook.com/ajax/browser_error_reports/"
report-to: {"max_age":2592000,"endpoints":[{"url":"https:\/\/www.facebook.com\/browser_reporting\/coop\/?minimize=0"}],"group":"coop_report","include_subdomains":true}, {"max_age":86400,"endpoints":[{"url":"https:\/\/www.facebook.com\/browser_reporting\/coep\/?minimize=0"}],"group":"coep_report"}, {"max_age":259200,"endpoints":[{"url":"https:\/\/www.facebook.com\/ajax\/comet_error_reports\/?device_level=mod-low&brsid=7387264710118225710"}]}, {"max_age":21600,"endpoints":[{"url":"https:\/\/www.facebook.com\/ajax\/browser_error_reports\/"}],"group":"permissions_policy"}
content-security-policy: default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com *.fbcdn.net 'unsafe-eval';script-src *.facebook.com *.fbcdn.net *.facebook.net 127.0.0.1:* 'unsafe-inline' blob: data: 'self' connect.facebook.net 'unsafe-eval' https://*.google-analytics.com *.google.com;style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline' https://fonts.googleapis.com;connect-src *.facebook.com facebook.com *.fbcdn.net *.facebook.net wss://*.facebook.com:* wss://*.whatsapp.com:* wss://*.fbcdn.net attachment.fbsbx.com ws://localhost:* blob: *.cdninstagram.com 'self' http://localhost:3103 wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d.facebook.com/ v.whatsapp.net *.fbsbx.com *.fb.com https://*.google-analytics.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com https://fonts.gstatic.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net connect.facebook.net *.carriersignal.info blob: android-webview-video-poster: *.whatsapp.net *.fb.com *.oculuscdn.com *.tenor.co *.tenor.com *.giphy.com https://paywithmybank.com/ https://*.paywithmybank.com/ https://www.googleadservices.com https://googleads.g.doubleclick.net https://*.google-analytics.com;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data: *.tenor.co *.tenor.com https://*.giphy.com;frame-src *.facebook.com *.fbsbx.com fbsbx.com data: www.instagram.com *.fbcdn.net https://paywithmybank.com/ https://*.paywithmybank.com/ https://www.googleadservices.com https://googleads.g.doubleclick.net https://www.google.com https://td.doubleclick.net *.google.com *.doubleclick.net;worker-src blob: *.facebook.com data:;block-all-mixed-content;upgrade-insecure-requests;
document-policy: force-load-at-top
permissions-policy: accelerometer=(), attribution-reporting=(self), autoplay=(), battery=(self), bluetooth=(), camera=(self), ch-device-memory=(), ch-downlink=(), ch-dpr=(), ch-ect=(), ch-rtt=(), ch-save-data=(), ch-ua-arch=(), ch-ua-bitness=(), ch-viewport-height=(), ch-viewport-width=(), ch-width=(), clipboard-read=(self), clipboard-write=(self), compute-pressure=(), display-capture=(self), encrypted-media=(self), fullscreen=(self), gamepad=*, geolocation=(self), gyroscope=(), hid=(), idle-detection=(), interest-cohort=(), keyboard-map=(), local-fonts=(), magnetometer=(), microphone=(self), midi=(), otp-credentials=(), payment=(), picture-in-picture=(self), private-state-token-issuance=(), publickey-credentials-get=(self), screen-wake-lock=(), serial=(), shared-storage=(), shared-storage-select-url=(), private-state-token-redemption=(), usb=(), usb-unrestricted=(), unload=(self), window-management=(), xr-spatial-tracking=(self);report-to="permissions_policy"
cross-origin-resource-policy: same-origin
cross-origin-embedder-policy-report-only: require-corp;report-to="coep_report"
cross-origin-opener-policy: same-origin-allow-popups;report-to="coop_report"
pragma: no-cache
cache-control: private, no-cache, no-store, must-revalidate
expires: Sat, 01 Jan 2000 00:00:00 GMT
x-content-type-options: nosniff
x-xss-protection: 0
x-frame-options: DENY
origin-agent-cluster: ?0
strict-transport-security: max-age=15552000; preload
x-fb-debug: /3JEfxMr+S6AOFf+2tPZwVf3Czv/h7geW+L36Y4808X5Lr9YtTQYFx+qeg7FjTkEjHgmq2Cpc0csDIlEE8ovZA==
date: Wed, 03 Jul 2024 04:39:57 GMT
x-fb-connection-quality: EXCELLENT; q=0.9, rtt=27, rtx=0, c=10, mss=1392, tbw=3577, tp=-1, tpl=-1, uplat=272, ullat=0
alt-svc: h3=":443"; ma=86400

Thanks, not support content-encoding: zstd currently, would you like to share me the url, I can have a test.

[tas33n]

U can check facebook web, all behind the page api/scripts returns this 'zstd' encoding. below is a curl from my testing account..

curl 'https://www.facebook.com/ajax/bulk-route-definitions/' \
  -H 'accept: */*' \
  -H 'accept-language: en-GB,en;q=0.9' \
  -H 'content-type: application/x-www-form-urlencoded' \
  -H 'cookie: sb=EfGEZm-zQUpIhVZ5BfqMLc6f; datr=EfGEZoeyLDBoQvTGAa4obEpn; locale=en_GB; c_user=100085763542628; xs=14%3AxQULrMSiNcLsyA%3A2%3A1719988576%3A-1%3A9665; presence=C%7B%22t3%22%3A%5B%5D%2C%22utc3%22%3A1719988594651%2C%22v%22%3A1%7D; wd=1366x158; fr=0f8yY6iOna12l5yWj.AWX3mU60cfPJHVZUymZl-rioKOY.BmhPER..AAA.0.0.BmhPF7.AWWzZom-Nfk' \
  -H 'dnt: 1' \
  -H 'origin: https://www.facebook.com' \
  -H 'priority: u=1, i' \
  -H 'referer: https://www.facebook.com/' \
  -H 'sec-ch-prefers-color-scheme: light' \
  -H 'sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"' \
  -H 'sec-ch-ua-full-version-list: "Not/A)Brand";v="8.0.0.0", "Chromium";v="126.0.6478.127", "Google Chrome";v="126.0.6478.127"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-model: ""' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-ch-ua-platform-version: "6.0.0"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36' \
  -H 'x-asbd-id: 129477' \
  -H 'x-fb-lsd: xuui1CV-ERnG2AsjhARNk1' \
  --data-raw 'route_urls[0]=%2F&routing_namespace=fb_comet&__aaid=0&__user=100085763542628&__a=1&__req=2&__hs=19907.HYP%3Acomet_pkg.2.1..2.1&dpr=1&__ccg=EXCELLENT&__rev=1014644081&__s=48nhtm%3Akh2k0h%3Akf9vui&__hsi=7387294801491474419&__dyn=7AzHK4HwkEng5K8G6EjBAg5S3G2O5U4e2C17xt3odE98K360CEboG0x8bo6u3y4o2Gwn82nwb-q7oc81xoswMwto886C11wBz83WwgEcEhwGxu782lwv89kbxS1Fwc61awkovwRwlE-U2exi4UaEW2G1jwUBwJK2W5olwUwgojUlDw-wUwxwjFovUaU3VBwFKq2-azo2NwwwOg2cwMwhEkxebwHwNxe6Uak0zU8oC1hxB0qo4e16wWwjHDzUiwRK6E4-8wLwHw&__csr=gR112cegxdb8x4D2dbfbsRNnObhmLp54ibNYQOnlmh4G8QTiuil8yTEzHPRKL99uFdBL9WXARSF8ZbXiVo8paGFAlfBKXyXgK4Vp8Cm59fCVVp649rx6lap7UOby9ry8hx3x6ubDKfBGdz8y325EtK4V4fzUlxe9xa8wyyV8dVEboaEqCxeeglw8iaxeawUzod8vwkU2Iwywj8aEjwNw8W1LwnE7i0li18wOw7Dw1xq08ZwPwv82Jw0kQE03TACw33A0qi04TE08sE1BoW2a015uxG01Asw9q0cLK7E&__comet_req=15&fb_dtsg=NAcPnLk42f-NczaPobcvzYezP530xuRjsm6ejoNH-DYwbjPtVXp-JGA%3A14%3A1719988576&jazoest=25623&lsd=xuui1CV-ERnG2AsjhARNk1&__spin_r=1014644081&__spin_b=trunk&__spin_t=1719988603'

[tas33n]

when i used this curl in postman, it returned 'br' encoding instead of 'zstd'

Postman:

Chrome:

[MegatronKing]

when i used this curl in postman, it returned 'br' encoding instead of 'zstd'

Postman:

Chrome:

Please check Accept-Encoding in request headers.

[MegatronKing]

We will support zstd compression in the next version.

[tas33n]

Today i tried rewrite the request header for facebook request, i removed old accept encoding request and added new header accept encoding without zstd in it. and now all request are readble in br (i think, that what shown)

results

[MegatronKing]

Done! Please update to 2.20.0.