search

yifeikong / curl_cffi

Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
https://curl-cffi.readthedocs.io/
MIT License
1.63k stars 210 forks source link

Strange behavior on the site when using impersonate #247

Closed r00t-Taurus closed 1 week ago

r00t-Taurus commented 4 months ago

curl-cffi==0.6.0b9

If I understand correctly, then all the options in the impersonate parameter should be legal for the site? But I came across a site with strange behavior

def get_random_string(lenf):
    return "".join(
        secrets.choice(string.ascii_letters + string.digits) for _ in range(lenf)
    )

url = "https://gateway.kleinanzeigen.de/auth/login"
json = {"username": "fs432dffdsfdsf@gmail.com", "password": "Qfds1f23+v"}
headers = {
    "X-Ebayk-Wenkse-Session-Id": get_random_string(32),
    "X-Ebayk-App": f"{get_random_string(8)}-{get_random_string(4)}-{get_random_string(4)}-{get_random_string(4)}-{get_random_string(25)}",
    "X-Ebayk-Userid-Token": "",
    "X-Ebayk-Groups": "BAND-7832-Category-Alerts_B|BAND-7912-Revise-Buy-Now_B|BLN-19260-cis-login_B|BLN-24652_category_alert_B|backend_ab_bln13364_A|backend_ab_bln404_B|backend_ab_bln418_A|backend_ab_bln_abc_A|backend_ab_bln_abc2_A",
    "User-Agent": "Kleinanzeigen/15.18.0 (Android 7.1.2; google G011A)",
    "X-Ecg-User-Agent": "ebayk-android-app-15.18.0",
    "X-Ecg-User-Version": "15.18.0",
    "X-Acf-Sensor-Data": "3,a,ZW4fpMxYm9METyXMaPQRtahvaE1MOO4Uo34a+sqhF7Zb/HpUHkqGXb1tFcpVO/0i/kwnz+4Uc3q33qiR8MLBKp2cfgDPqDVwlXYgeKaDDLk0tWRjv0Gf7pJzwM9/8QCikGXTCUG/YNB9wq3MgvXyjbyhuw+XiEfzKgFEvM4KInE=,WT3ePQThSzpu1q/VcKDZfXU9ekK25gqILRKREGT4vwORw5zzE+cgsCxfvTsBlv9D9j58fRRK7FC5masivzbDocrRDht/95JDK7U2GXo1/eq2TXm+BFfqp5Hd0kijZJ5OcrjyjY/VhuaG8vzDBuYby8IXZ4DRLzWodDWsNWhUqNo=$RGlR/LEy5xDOuNqZabjgy4czyDHc9sHA5JTYQCfQg8VLXBSylMgZoz4R4N78lnHc0hhVI8GxSpMEJLX6BZdD/swVBI7X6nW802hbmqtRcgrTB56F8H9gX1DIcMmbmS8XHMTH2JhELoSH5VW6QPQfrAIYFpj7LZnbISlqwOKLIfW5z/QFOhCSAYZPU7qDcG5n9Q86dUGsmk6Ayre6VB1yZQ76o4pdyPmrYBPgMFm5zYRXd+JUNmM/DDdxqrPc9F9qNeCWaWzmbZPLYZxqKKzox+43/t50CLcKFDrvavIi8J+77/iqHvt3FRp4CHCLkPSUdyJeOo+KFqCz5wxbsrIMmHIz2ZQfrJPpMb3xiW/F7c6wV3QfDILfj7QcewqbGeAmu7nuWZIzyopgkpgdmF4RLURFOSwML2/H310nZLErri18T+gIbBG8LixqnjGfRUb7IWDBeofMNC5yWz+htu522mVi468IwhLm2I48SbTO/jsn7G8SEzCGd40er1cRidOXz7A0mtMXCHTLE5TZVyUQUQuX0/4LpB6qBpGzPKVwQxVrL7aE/3vpELR6+wn2L65AwincaKXAgSdefeRkzbP1r3/Z93MYgAJDBR4dXBDJV6YdXbA+H7bR6H5mr+pjEfkkg6y5OjxSjKccYUwDiaYsku6aXbxGMR8IoWPCvEh5y4tr7TYwSdAh7MNOoa2/54NqQKH/cDHlQMEWxCakOmmCOqJWgOrE/YBXfKOX2kb/edF9VPooJZnGEp3y8y60bS/vT4aNbbN5kNRPsOUxW3yDS3tC9nQ4agEa/Ky2dMpASj+VcS9GOODiUVtkA9RcjFOTOPCK5xvAgsarBSND1IhKaG7jNvtsjDjoiC44OaFEyiASd26yH5vM2DWlhfBI7TWN4DKywJ6eY0S5ky1sX7ndWy0j0G/AO0lCOBUmd/AZ7j3Lr9zW6dCYn//mybarDYFAG15GqzgIzHByjp0eFrmeyNzpKBUJqFbQP6N/QoOfZ5efy/IzDgYNvUMoJWX+bQS3eybYRrwluoJwRsT9m9H5ndcpqQBUWN0w9kk2EXu3CccrIKFhHyHtfqEIM4qDUaR9VFhoWV3dd7UgfmqaHRHJJw8tMyMn056mO/yMHzTnB3LpNBOF0HqrjwtY5iJd5FvSjsQM8daOTWR6bQG9QTn40STIE16X0QG75Vm7h8xfsvanWGVnI5RhX7/XZyBH4PtiTlZjxCRxs1rIvLstJINdCiB99hgbKbPMGYnEp/2+LaR7O5iRc0kobd1e4//haWGTOUkyUyfYjUouJ5CXnTnArE9VVT18SPD1LCrQBSZVKaW9KB16bcDoSIMmiC5HY7AEdygUibJGEOixvB2p6I26Rk71MTQ8S+lnYGVV2gFCEmbLodRfELQsw2El7NrqkqnZpG11QyBTqOioa15Ja6/B+xuvlg+wSzvDC7xserXy2Dt8rGD3Vjw5wdvthevByhnEzMt6z6TO5fZ+bSsk9IgHZEzPHQw7qhz/6I9d7ZCQRkYtVE3CDbaJl/KtaRwaDiRDTezOdiBmAUEd+VAFRODYbUBOO4PKqKymPohLtuSbuoucsD8HB4Y5YSM6V1zL+OQDOZ6AVR36WKsHleDzD7S8Se8zbMyrIbsBExdpo1kQPuy+DeBXTm0fFA2Y+Y+qggNGBOZawi6XnCJtuKZ6OO8emrwaBLH/SbIdcIiZe6FQ07hZ29SA8UZMHk30LjlEHxuWYc2vZ9R/70cT4E9cxi5ckpIGmtAwpCopDahmkE3V++EjwQ9IA92BU5kFkDrUPbvq3DlHIRzyv4m28JmJsApYByzK6jRXaoGjBF6PgVMmSGyEu+oKuOinoF/NIBh4c0kOTmzTpvJ+xcj9BTWtEZCcJ8HvX/sAXUI5eAXHCjGslPG+9Ve8kZdor9mbT8iX3HAdfHDH98aPH0olUpENLf+/TqS8ogQUm0u9lJHRZH0s5aAw38RPrNNhmmqOe6/d5+JTbT3+ZM/Zzo9TVClSUJZZzc5bUjLBfr6qjJAnG+1zfRf04cAp1XWSzhVinctsjrue1rapIJNoPU2LpBdAYaSn8tQNztdNHWOLfouswRhWWEG+0jUMLAwAVb+ANSrwS83/VZvCXltfklNhKyEbecvsjt761D//5Nuk1aKTH/uCD9SQgrFDlcKwGYlCl7RSBMFjSq6XO4sOoeE6hQl5uMnvN1tGQiIQ3KLmcGl4o4i7ZBJWQ/rjcqFGDbaUP9lFx4iyhcBzrstmZT30UPc98G0h3sFLlN/ia/AGOsXHE98g0Nn70gheKr3KNWEUhsfqgOoB1pmZ1CR6F7YBQD7h23mIJnuaJMveKL7ho8hhtSmVbdR1SH6T02eun8Wk6NpDa5QUn/eqj2/U/BFu6Udu8dcJg3qjX69IChZfhKWFBc4sLwDAFAB8hVHDnV943smavxATPTXktE3dy1NRQ4WOCUYoowNQPrl6NjmJ98c/pJ6Lg9I=$18,4,7$$$",
    "Content-Type": "application/json; charset=utf-8",
    "Accept-Encoding": "gzip, deflate, br",
}

async def main():
    async with requests.AsyncSession() as session:
        item = await session.post(
            url,
            headers=headers,
            json=json,
            impersonate="chrome99_android",
            timeout=(1.5, 9),
        )
        print(item.status_code)
        print(item.text)

if __name__ == "__main__":
    import asyncio

    asyncio.run(main())

If you execute this code, it returns 403 and says that the ip is blocked, it really can be ip blocking, so you can try to install a proxy But if you change chrome99_android to any safari, for example safari17_0, then everything starts working (sometimes you need to make a request a couple of times for the site to start skipping it), why is that? after all, it would be more logical to use chrome99_android, since the headers indicate "User-Agent": "Kleinanzeigen/15.18.0 (Android 7.1.2; google G011A)", But this is not the whole strangeness.

def get_random_string(lenf):
    return "".join(
        secrets.choice(string.ascii_letters + string.digits) for _ in range(lenf)
    )

url2 = "https://api.kleinanzeigen.de/api/account/registration"
headers = {
    "X-Ebayk-Wenkse-Session-Id": get_random_string(32),
    "X-Ebayk-App": f"{get_random_string(8)}-{get_random_string(4)}-{get_random_string(4)}-{get_random_string(4)}-{get_random_string(25)}",
    "X-Ebayk-Userid-Token": "",
    "X-Ebayk-Groups": "BAND-7832-Category-Alerts_B|BAND-7912-Revise-Buy-Now_B|BLN-19260-cis-login_B|BLN-24652_category_alert_B|backend_ab_bln13364_A|backend_ab_bln404_B|backend_ab_bln418_A|backend_ab_bln_abc_A|backend_ab_bln_abc2_A",
    "User-Agent": "Kleinanzeigen/15.18.0 (Android 7.1.2; google G011A)",
    "X-Ecg-User-Agent": "ebayk-android-app-15.18.0",
    "X-Ecg-User-Version": "15.18.0",
    "X-Acf-Sensor-Data": "3,a,ZW4fpMxYm9METyXMaPQRtahvaE1MOO4Uo34a+sqhF7Zb/HpUHkqGXb1tFcpVO/0i/kwnz+4Uc3q33qiR8MLBKp2cfgDPqDVwlXYgeKaDDLk0tWRjv0Gf7pJzwM9/8QCikGXTCUG/YNB9wq3MgvXyjbyhuw+XiEfzKgFEvM4KInE=,WT3ePQThSzpu1q/VcKDZfXU9ekK25gqILRKREGT4vwORw5zzE+cgsCxfvTsBlv9D9j58fRRK7FC5masivzbDocrRDht/95JDK7U2GXo1/eq2TXm+BFfqp5Hd0kijZJ5OcrjyjY/VhuaG8vzDBuYby8IXZ4DRLzWodDWsNWhUqNo=$RGlR/LEy5xDOuNqZabjgy4czyDHc9sHA5JTYQCfQg8VLXBSylMgZoz4R4N78lnHc0hhVI8GxSpMEJLX6BZdD/swVBI7X6nW802hbmqtRcgrTB56F8H9gX1DIcMmbmS8XHMTH2JhELoSH5VW6QPQfrAIYFpj7LZnbISlqwOKLIfW5z/QFOhCSAYZPU7qDcG5n9Q86dUGsmk6Ayre6VB1yZQ76o4pdyPmrYBPgMFm5zYRXd+JUNmM/DDdxqrPc9F9qNeCWaWzmbZPLYZxqKKzox+43/t50CLcKFDrvavIi8J+77/iqHvt3FRp4CHCLkPSUdyJeOo+KFqCz5wxbsrIMmHIz2ZQfrJPpMb3xiW/F7c6wV3QfDILfj7QcewqbGeAmu7nuWZIzyopgkpgdmF4RLURFOSwML2/H310nZLErri18T+gIbBG8LixqnjGfRUb7IWDBeofMNC5yWz+htu522mVi468IwhLm2I48SbTO/jsn7G8SEzCGd40er1cRidOXz7A0mtMXCHTLE5TZVyUQUQuX0/4LpB6qBpGzPKVwQxVrL7aE/3vpELR6+wn2L65AwincaKXAgSdefeRkzbP1r3/Z93MYgAJDBR4dXBDJV6YdXbA+H7bR6H5mr+pjEfkkg6y5OjxSjKccYUwDiaYsku6aXbxGMR8IoWPCvEh5y4tr7TYwSdAh7MNOoa2/54NqQKH/cDHlQMEWxCakOmmCOqJWgOrE/YBXfKOX2kb/edF9VPooJZnGEp3y8y60bS/vT4aNbbN5kNRPsOUxW3yDS3tC9nQ4agEa/Ky2dMpASj+VcS9GOODiUVtkA9RcjFOTOPCK5xvAgsarBSND1IhKaG7jNvtsjDjoiC44OaFEyiASd26yH5vM2DWlhfBI7TWN4DKywJ6eY0S5ky1sX7ndWy0j0G/AO0lCOBUmd/AZ7j3Lr9zW6dCYn//mybarDYFAG15GqzgIzHByjp0eFrmeyNzpKBUJqFbQP6N/QoOfZ5efy/IzDgYNvUMoJWX+bQS3eybYRrwluoJwRsT9m9H5ndcpqQBUWN0w9kk2EXu3CccrIKFhHyHtfqEIM4qDUaR9VFhoWV3dd7UgfmqaHRHJJw8tMyMn056mO/yMHzTnB3LpNBOF0HqrjwtY5iJd5FvSjsQM8daOTWR6bQG9QTn40STIE16X0QG75Vm7h8xfsvanWGVnI5RhX7/XZyBH4PtiTlZjxCRxs1rIvLstJINdCiB99hgbKbPMGYnEp/2+LaR7O5iRc0kobd1e4//haWGTOUkyUyfYjUouJ5CXnTnArE9VVT18SPD1LCrQBSZVKaW9KB16bcDoSIMmiC5HY7AEdygUibJGEOixvB2p6I26Rk71MTQ8S+lnYGVV2gFCEmbLodRfELQsw2El7NrqkqnZpG11QyBTqOioa15Ja6/B+xuvlg+wSzvDC7xserXy2Dt8rGD3Vjw5wdvthevByhnEzMt6z6TO5fZ+bSsk9IgHZEzPHQw7qhz/6I9d7ZCQRkYtVE3CDbaJl/KtaRwaDiRDTezOdiBmAUEd+VAFRODYbUBOO4PKqKymPohLtuSbuoucsD8HB4Y5YSM6V1zL+OQDOZ6AVR36WKsHleDzD7S8Se8zbMyrIbsBExdpo1kQPuy+DeBXTm0fFA2Y+Y+qggNGBOZawi6XnCJtuKZ6OO8emrwaBLH/SbIdcIiZe6FQ07hZ29SA8UZMHk30LjlEHxuWYc2vZ9R/70cT4E9cxi5ckpIGmtAwpCopDahmkE3V++EjwQ9IA92BU5kFkDrUPbvq3DlHIRzyv4m28JmJsApYByzK6jRXaoGjBF6PgVMmSGyEu+oKuOinoF/NIBh4c0kOTmzTpvJ+xcj9BTWtEZCcJ8HvX/sAXUI5eAXHCjGslPG+9Ve8kZdor9mbT8iX3HAdfHDH98aPH0olUpENLf+/TqS8ogQUm0u9lJHRZH0s5aAw38RPrNNhmmqOe6/d5+JTbT3+ZM/Zzo9TVClSUJZZzc5bUjLBfr6qjJAnG+1zfRf04cAp1XWSzhVinctsjrue1rapIJNoPU2LpBdAYaSn8tQNztdNHWOLfouswRhWWEG+0jUMLAwAVb+ANSrwS83/VZvCXltfklNhKyEbecvsjt761D//5Nuk1aKTH/uCD9SQgrFDlcKwGYlCl7RSBMFjSq6XO4sOoeE6hQl5uMnvN1tGQiIQ3KLmcGl4o4i7ZBJWQ/rjcqFGDbaUP9lFx4iyhcBzrstmZT30UPc98G0h3sFLlN/ia/AGOsXHE98g0Nn70gheKr3KNWEUhsfqgOoB1pmZ1CR6F7YBQD7h23mIJnuaJMveKL7ho8hhtSmVbdR1SH6T02eun8Wk6NpDa5QUn/eqj2/U/BFu6Udu8dcJg3qjX69IChZfhKWFBc4sLwDAFAB8hVHDnV943smavxATPTXktE3dy1NRQ4WOCUYoowNQPrl6NjmJ98c/pJ6Lg9I=$18,4,7$$$",
    "Content-Type": "application/json; charset=utf-8",
    "Accept-Encoding": "gzip, deflate, br",
}
json2 = {
    "accountType": "PRIVATE",
    "contactName": "Kecas",
    "email": "fdterwrtegfgdf@outlook.com",
    "marketingOptIn": False,
    "password": "fdsfdFDSFDSFsfreewr+",
    "passwordConfirmation": "fdsfdFDSFDSFsfreewr+",
}
async def main():
    async with requests.AsyncSession() as session:
        item = await session.post(
            url2,
            headers=headers,
            json=json2,
            impersonate="safari17_0",
            timeout=(1.5, 9),
        )
        print(item.status_code)
        print(item.text)

if __name__ == "__main__":
    import asyncio

    asyncio.run(main())

If you run this you will get 403 I tried all the parameters available in impersonate, but they all give 403, but if you remove the impersonate parameter, then everything starts to work, why is that? after all, these are legitimate tls fingerprints, can't it block real devices? As far as I know, the site uses akamai. Is it possible to issue a different ja3 every time?

async def main():
    async with requests.AsyncSession() as session:
        item = await session.post(
            url2,
            headers=headers,
            json=json2,
            # impersonate="safari17_0",
            timeout=(1.5, 9),
        )
        print(item.status_code)
        print(item.text)
yifeikong commented 4 months ago

AFAIK, TLS fingerprints on iOS should be more uniformed, since there is not other browser engine on iOS except for webkit(Safari). Akamai also considers the http2 fingerprints along with the TLS ones.

You can change the fingerprints for each request with options like CURLOPT_CIPHER_SUITES and those mentioned on yifeikong/curl-impersonate.