lightbody / browsermob-proxy

A free utility to help web developers watch and manipulate network traffic from their AJAX applications.
http://bmp.lightbody.net
Apache License 2.0
2.17k stars 658 forks source link

browsermobproxy HAR file response content size is 0 always #902

Open allenbaby opened 1 year ago

allenbaby commented 1 year ago

I am trying to get the network log and gnerating a HAR file with browsermobproxy and I am not able to get the response content data, ie the content data has size 0 always.

for example when I go to this url: https://github.com/lightbody/browsermob-proxy/issues/626 mobproxy HAR file :

{
    "log": {
        "version": "1.2",
        "creator": {
            "name": "BrowserMob Proxy",
            "version": "2.1.4",
            "comment": ""
        },
        "pages": [
            {
                "id": "https://github.com/lightbody/browsermob-proxy/issues/626",
                "startedDateTime": "2023-07-04T14:06:44.515+05:30",
                "title": "https://github.com/lightbody/browsermob-proxy/issues/626",
                "pageTimings": {
                    "comment": ""
                },
                "comment": ""
            }
        ],
        "entries": [
            {
                "pageref": "https://github.com/lightbody/browsermob-proxy/issues/626",
                "startedDateTime": "2023-07-04T14:06:46.931+05:30",
                "request": {
                    "method": "GET",
                    "url": "https://avatars.githubusercontent.com/u/4196299?s=80&u=0345202e847a88c47546f29b0ba10f1455a3ccb7&v=4",
                    "httpVersion": "HTTP/1.1",
                    "cookies": [],
                    "headers": [],
                    "queryString": [
                        {
                            "name": "s",
                            "value": "80"
                        },
                        {
                            "name": "u",
                            "value": "0345202e847a88c47546f29b0ba10f1455a3ccb7"
                        },
                        {
                            "name": "v",
                            "value": "4"
                        }
                    ],
                    "headersSize": 683,
                    "bodySize": 0,
                    "comment": ""
                },
                "response": {
                    "status": 200,
                    "statusText": "OK",
                    "httpVersion": "HTTP/1.1",
                    "cookies": [],
                    "headers": [],
                    "content": {
                        "size": 0,
                        "mimeType": "image/jpeg",
                        "comment": ""
                    },
                    "redirectURL": "",
                    "headersSize": 950,
                    "bodySize": 1940,
                    "comment": ""
                },
                "cache": {},
                "timings": {
                    "comment": "",
                    "wait": 49,
                    "ssl": 629,
                    "send": 0,
                    "blocked": 0,
                    "receive": 0,
                    "dns": 0,
                    "connect": 680
                },
                "serverIPAddress": "185.199.111.133",
                "comment": "",
                "time": 731
            }

"content": {"size": 0,"mimeType": "image/jpeg","comment": "" } Notice the content size is 0 but other details are fetching alright. I have 110 records similar to this in the HAR file all having content size as 0. When I generated the HAR from the chrome dev tools, I can see the data of the response (as byte46) and the content size is not 0. What am I doing wrong, please help!

My main aim to get the HAR file is to get a certain blob from LinkedIn from a network call which cannot be directly accessed because of the security. I have to see the response content data using mobproxy, I have tried adding SSL certifications This is the my selenium and mobproxy code block

server.start()
proxy = server.create_proxy(params={"trustAllServers": "true"})
# Configure the browser proxy in chrome options
options = webdriver.ChromeOptions()
options.add_argument("--ignore-certificate-errors")
options.add_argument("--proxy-server={}".format(proxy.proxy))
driver = webdriver.Chrome(
    executable_path="C:/Users/baby.allen/Downloads/chromedriver.exe",
    options=options,
)
allenbaby commented 1 year ago

Maybe related to #900 , I am not quite sure. Forgive me If this is a duplicate, but in my case, every entry inside the HAR file has a content size 0.

Mouad-scriptz commented 1 year ago

same issue with me

NeoUKR commented 1 year ago

Maybe related to #900 , I am not quite sure. Forgive me If this is a duplicate, but in my case, every entry inside the HAR file has a content size 0.

The solve of the problem is here: https://github.com/lightbody/browsermob-proxy/issues/598 .