search

resphinas / shopee_goods_spider

虾皮12国家全站点爬虫 shopee goods's spider ，thumbs up if you like this project
8 stars 2 forks source link

br_headers are not accepted #1

Open rafaelribeiroo opened 7 months ago

rafaelribeiroo commented 7 months ago

I've tried to send requests with br_headers, but it not works anymore...

resphinas commented 7 months ago

I've tried to send requests with br_headers, but it not works anymore...

The security system of websites is constantly changing in real-tiime,especially regarding web crawlers,this is un avoidable, but you can refer to some of the content in the code, if you actually need help, i can do some work for a fee, this is very time-consuming for most people,but as time goes by,i cannot guarantee that I have the ability to redo it.

rafaelribeiroo commented 7 months ago

I've tried to send requests with br_headers, but it not works anymore...

The security system of websites is constantly changing in real-tiime,especially regarding web crawlers,this is un avoidable, but you can refer to some of the content in the code, if you actually need help, i can do some work for a fee, this is very time-consuming for most people,but as time goes by,i cannot guarantee that I have the ability to redo it.

Oh i appreciate a lot if u could help me, current code below..

from requests import get, Session
from pdb import set_trace

headers = {
    "accept": "application/json",
    "accept-encoding": "gzip, deflate, br",
    "accept-language": "zh-CN,zh;q=0.9",
    "af-ac-enc-dat": "AAcyLjUuMC0yAAABhS4izBUAAA7CAuAAAAAAAAAAAuvlR3weVVU60ykHUkkzSmQs+0sol/82EyfDx/bVRcPaaRvYm6/AMx6LSpoTdPFTsS3fKgun0IrV61IXZnIllxCRFup2D1gYRPH79G8+i2Aeb7zsNtGDj9cL8Asw813ikLnKoCVr4l3fuyugWFRp1Qe6rTHJGZBVPbG6m89nsDLLDwgG5xoTlPCAqrBcwADPmfermZMS99ksTip4iyWZxQP+WmEm5EuoJQgI6nY5eCmyhkmD4HThl5TjUnyZisue7aWxtehPu7N70xtTDcHg3wmMUa2R+Pauq8cKY3SotpKL1pPKWJf7Ii6ulIGCuoQClFlVhPyX/zYTJ8PH9tVFw9ppG9ibnkHEUB14rHUebq8JcaWvLTOCoBzhRKmUO439gBG9LIjBUDPJ/Fz4D5PsTtJ62WeGnbI2KS5LUeXSWBpl59AI5gYJCX7CT9Ut1dzLOvVE0Rl8EQl6E1RwOgFd2KVjgKprW5JdW8YfywX1otVvVkVU1wsNXts5GyAjI8vnGItu2NO3eLkh8HnUyi6Hw1/itIk+dvrx2cDNAh/WiazMgcgB2IpWoI3lzvoPKDx96WVQBxEd5/le2ngmePC+WGsrwe6QUQJfq3K1AG1EpKO616ktiwbY7P9KHML6aQtF4FlKd6x7LmbUKDtOOQPXo6adSCgMwsXNNYTyK3HSzCGoV/4BXUckfn/aqZ2EUBMC3PvgG/UJYfJy7sHEMhm1yOdrF0U4C7jmWDcFCMr+GYq1E5L5+opWoI3lzvoPKDx96WVQBxHpaZ1BcwHcCXLOb3JmD/34hAJG77Wiv3h8+QSVcnFcYr1EtvsTmxjnCzBEa6+AecM5gAAJEAbg9IkzYPLXPwi598jZ/901tpYwiYTjGF/2xIkHl1HqyESYMNSS8O5naAcTeUAhbodJtmu4kLBVAN/IDe4LpjRu9mRyVhJkALpVyW9oi9wjYiPXVXoHt38VZKt5R7jEhjvz+WyTm/WWgrub",
    "cache-control": "no-cache",
    "content-type": "application/json",
    "pragma": "no-cache",
    "referer": "https://shopee.com.br/search?filters=9&keyword=funko&locations=Nacional&noCorrection=true&page=0&sortBy=relevancy",
    "sec-ch-ua": "\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Linux\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin",
    # "sz-token": "obGBwsb9uFAMEgaTl4te+A==|WHI4PAmPPw9NeYxDHtb8HE65+39NAYsMqrqf2A3jHJargCyGL30tkSA3wqcGfZ81s4KsH+2UlzfUt8kuSlsxvcaS8MrSAU8gDQ==|Va3Eo3y9J9sOKDAO|06|3",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
    "x-api-source": "pc",
    "x-csrftoken": "tD86kbYtdrqFVkM1b7dA9iYgiqZj77Rh",
    "x-requested-with": "XMLHttpRequest",
    # "x-sap-access-f": "3.2.107.2.0|13|2.5.0-2_5.1.0_0_158|f9f99d436d624a12aa4761d29c07ee1a5faf4142d78242|10900|1100",
    # "x-sap-access-s": "4N_RTpkpvb-MRTNNJSUvNbi4FM4zEOAKjdeKuC8hv5E=",
    # "x-sap-access-t": "1671516644",
    "x-shopee-language": "pt-BR"
}

params = {
    "filters": "9",
    "keyword": "funko",
    "locations": "Nacional",
    "noCorrection": "true",
    "page": "0",
    "sortBy": "relevancy",
}

with Session() as session:
        res = session.get("https://shopee.com.br/search", params=params, headers=headers)
        print(res)
        set_trace()
resphinas commented 7 months ago

I've tried to send requests with br_headers, but it not works anymore...

The security system of websites is constantly changing in real-tiime,especially regarding web crawlers,this is un avoidable, but you can refer to some of the content in the code, if you actually need help, i can do some work for a fee, this is very time-consuming for most people,but as time goes by,i cannot guarantee that I have the ability to redo it.

Oh i appreciate a lot if u could help me, current code below..

from requests import get, Session
from pdb import set_trace

headers = {
    "accept": "application/json",
    "accept-encoding": "gzip, deflate, br",
    "accept-language": "zh-CN,zh;q=0.9",
    "af-ac-enc-dat": "AAcyLjUuMC0yAAABhS4izBUAAA7CAuAAAAAAAAAAAuvlR3weVVU60ykHUkkzSmQs+0sol/82EyfDx/bVRcPaaRvYm6/AMx6LSpoTdPFTsS3fKgun0IrV61IXZnIllxCRFup2D1gYRPH79G8+i2Aeb7zsNtGDj9cL8Asw813ikLnKoCVr4l3fuyugWFRp1Qe6rTHJGZBVPbG6m89nsDLLDwgG5xoTlPCAqrBcwADPmfermZMS99ksTip4iyWZxQP+WmEm5EuoJQgI6nY5eCmyhkmD4HThl5TjUnyZisue7aWxtehPu7N70xtTDcHg3wmMUa2R+Pauq8cKY3SotpKL1pPKWJf7Ii6ulIGCuoQClFlVhPyX/zYTJ8PH9tVFw9ppG9ibnkHEUB14rHUebq8JcaWvLTOCoBzhRKmUO439gBG9LIjBUDPJ/Fz4D5PsTtJ62WeGnbI2KS5LUeXSWBpl59AI5gYJCX7CT9Ut1dzLOvVE0Rl8EQl6E1RwOgFd2KVjgKprW5JdW8YfywX1otVvVkVU1wsNXts5GyAjI8vnGItu2NO3eLkh8HnUyi6Hw1/itIk+dvrx2cDNAh/WiazMgcgB2IpWoI3lzvoPKDx96WVQBxEd5/le2ngmePC+WGsrwe6QUQJfq3K1AG1EpKO616ktiwbY7P9KHML6aQtF4FlKd6x7LmbUKDtOOQPXo6adSCgMwsXNNYTyK3HSzCGoV/4BXUckfn/aqZ2EUBMC3PvgG/UJYfJy7sHEMhm1yOdrF0U4C7jmWDcFCMr+GYq1E5L5+opWoI3lzvoPKDx96WVQBxHpaZ1BcwHcCXLOb3JmD/34hAJG77Wiv3h8+QSVcnFcYr1EtvsTmxjnCzBEa6+AecM5gAAJEAbg9IkzYPLXPwi598jZ/901tpYwiYTjGF/2xIkHl1HqyESYMNSS8O5naAcTeUAhbodJtmu4kLBVAN/IDe4LpjRu9mRyVhJkALpVyW9oi9wjYiPXVXoHt38VZKt5R7jEhjvz+WyTm/WWgrub",
    "cache-control": "no-cache",
    "content-type": "application/json",
    "pragma": "no-cache",
    "referer": "https://shopee.com.br/search?filters=9&keyword=funko&locations=Nacional&noCorrection=true&page=0&sortBy=relevancy",
    "sec-ch-ua": "\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Linux\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin",
    # "sz-token": "obGBwsb9uFAMEgaTl4te+A==|WHI4PAmPPw9NeYxDHtb8HE65+39NAYsMqrqf2A3jHJargCyGL30tkSA3wqcGfZ81s4KsH+2UlzfUt8kuSlsxvcaS8MrSAU8gDQ==|Va3Eo3y9J9sOKDAO|06|3",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
    "x-api-source": "pc",
    "x-csrftoken": "tD86kbYtdrqFVkM1b7dA9iYgiqZj77Rh",
    "x-requested-with": "XMLHttpRequest",
    # "x-sap-access-f": "3.2.107.2.0|13|2.5.0-2_5.1.0_0_158|f9f99d436d624a12aa4761d29c07ee1a5faf4142d78242|10900|1100",
    # "x-sap-access-s": "4N_RTpkpvb-MRTNNJSUvNbi4FM4zEOAKjdeKuC8hv5E=",
    # "x-sap-access-t": "1671516644",
    "x-shopee-language": "pt-BR"
}

params = {
    "filters": "9",
    "keyword": "funko",
    "locations": "Nacional",
    "noCorrection": "true",
    "page": "0",
    "sortBy": "relevancy",
}

with Session() as session:
        res = session.get("https://shopee.com.br/search", params=params, headers=headers)
        print(res)
        set_trace()

abive all,it never work at request function of python, the features has been recorded by le anticheat bot, try to use socket like my file "request_function.py" ,also the go lang works too, you can try , but i am not sure if it work now.