wangluozhe / requests

用于快速请求HTTP或HTTPS,并支持修改ja3、ja4指纹
GNU General Public License v3.0
417 stars 93 forks source link

默认并未使用http2.0 #20

Open dream2333 opened 1 year ago

dream2333 commented 1 year ago

如图,请求抖音,图一python,图二go,并没有发现http2协商过程,有没有一种方案可以强制设置http2.0 image image

wangluozhe commented 1 year ago

可以附上代码吗?涉及私密的地方可以用*代替,但请不要忽略不写。

dream2333 commented 1 year ago

可以附上代码吗?涉及私密的地方可以用*代替,但请不要忽略不写

代码比较复杂,涉及到逆向,抖音网页版全站都是http2.0的,您可以试一下,抓包出来的结果是http1.1的

dream2333 commented 1 year ago

可以附上代码吗?涉及私密的地方可以用*代替,但请不要忽略不写。

rawurl := "https://www.douyin.com/aweme/v1/web/aweme/post/?" ori_params := device_platform=webapp&aid=6383&channel=channel_pc_web&sec_user_id=MS4wLjABAAAA_nwyX0DfxMKDTbjs7GpTBjbDsKnN9eszwAa7r8MthWM&max_cursor=1635588600000&locate_query=false&show_live_replay_strategy=1&count=10&publish_video_strategy_type=2&pc_client_type=1&version_code=170400&version_name=17.4.0&cookie_enabled=true&screen_width=1920&screen_height=1080&browser_language=zh-CN&browser_platform=Linux+x86_64&browser_name=Edge&browser_version=112.0.1722.34&browser_online=true&engine_name=Blink&engine_version=112.0.0.0&os_name=Linux&os_version=x86_64&cpu_core_num=12&device_memory=8&platform=PC&downlink=10&effective_type=4g&round_trip_time=150&webid=7221748043930371587&msToken=ibF7wcwyHIkkMdztr4IJ_GAr9gwU8pOqx6O0Ond4m2io0U7ezpzT2NA7TFR8EiVIM5JQUoqLLV4CFLJ3LH7vdOwPc5KCkPaL2yLqhjDrMao9sL2hgaWwZX0Y4qtPmwaNRQ==&X-Bogus=DFSzswSLFcsANCIIteg2Az9WcBjW HTTP/1.1 headers := url.NewHeaders() headers.Set("accept", "my-app/0.0.1") headers.Set("user-agent", "application/json, text/plain, /") headers.Set("accept-language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6") headers.Set("cookie", "s_v_web_id=verify_lgg0huen_daX7wtLx_0tle_4eit_8ovH_Dg65iBlNNvDo; _tea_utm_cache_2018=undefined; ttwid=1%7C1w5kilF7Hm3tpF_ulVKyASmNqAIf3NFZt_E2_l9lA7I%7C1681444261%7Cbf6c415ec2fb9ad4f9d986d20beadd9e42e84f992935655545db65f45fdd3193; passport_csrf_token=fe226ac31e4cb9a8fb3778722671d2fe; passport_csrf_token_default=fe226ac31e4cb9a8fb3778722671d2fe; ttcid=a308c7ee6b7f4e37a4fdd482d4b0194e34; download_guide=%223%2F20230414%22; douyin.com; bd_ticket_guard_client_data=eyJiZC10aWNrZXQtZ3VhcmQtdmVyc2lvbiI6MiwiYmQtdGlja2V0LWd1YXJkLWl0ZXJhdGlvbi12ZXJzaW9uIjoxLCJiZC10aWNrZXQtZ3VhcmQtY2xpZW50LWNzciI6Ii0tLS0tQkVHSU4gQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tXHJcbk1JSUJEVENCdFFJQkFEQW5NUXN3Q1FZRFZRUUdFd0pEVGpFWU1CWUdBMVVFQXd3UFltUmZkR2xqYTJWMFgyZDFcclxuWVhKa01Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRUMyY1VncW1iYWJDemxXYmFuRXMzd3A5MlxyXG5yNHJLMmlLNEUyZlB3czhmUGxlV2VVWEhIWHRhSXRDODhZdEQ4M1cwNU5OZUdjck1RK09EMWhIWkRQcVc3NkFzXHJcbk1Db0dDU3FHU0liM0RRRUpEakVkTUJzd0dRWURWUjBSQkJJd0VJSU9kM2QzTG1SdmRYbHBiaTVqYjIwd0NnWUlcclxuS29aSXpqMEVBd0lEUndBd1JBSWdGMzhEb1dQSjAwcDZpeTEwU204TStqSkgvaGRMOUJkb21OZzBzZDROSk9nQ1xyXG5JRnRMRkt2Q3BIemkycW45TXFFR0JvSUtucVQxNW9NY2NEQ3ZseGMzdTl5MFxyXG4tLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS1cclxuIn0=; csrf_session_id=88788e107da38c65034fff85f99f331e; strategyABtestKey=%221681868544.997%22; __ac_nonce=0643f544d0011ca9a247c; __ac_signature=_02B4Z6wo00f01F4Nj3gAAIDBcMxlCW6Av5ReLYvAAHPD5BBoxE0z4V0Z7pI8IF-xzsFU60Yd7VZa4CapVmH.t6naIU3HYX8D5YbnPoVAGpEIFIyT0BkLZwnrXP2vC9oW5Md7skqd6tvrdAqT2a; VIDEO_FILTER_MEMO_SELECT=%7B%22expireTime%22%3A1682476749862%2C%22type%22%3A1%7D; msToken=a0B1-QoQ0DyNHbn6ovUVO3a4zMbX5NfvOva82_GHbkCTnc7_yd9iVXXhCwaITyylRZWluTgVCfj-ktwitx0yTBJLS6gKJvF9mdz0IuB2vK-UQ3PrAuvJZMdX4efkMv824A==; tt_scid=neXSttxYg6FHT7IBM58VjP9IwK8qP8gAO1h77BU5zHWoRyPGwsvrIMK1sgOOfOmh4c82; pwa2=%223%7C0%22; home_can_add_dy_2_desktop=%221%22; msToken=ibF7wcwyHIkkMdztr4IJ_GAr9gwU8pOqx6O0Ond4m2io0U7ezpzT2NA7TFR8EiVIM5JQUoqLLV4CFLJ3LH7vdOwPc5KCkPaL2yLqhjDrMao9sL2hgaWwZX0Y4qtPmwaNRQ==") headers.Set("referer", "https://www.douyin.com/user/MS4wLjABAAAA_nwyX0DfxMKDTbjs7GpTBjbDsKnN9eszwAa7r8MthWM") headers.Set("user-agent", ua) req := url.NewRequest() req.Headers = headers // req.Proxies = "http://127.0.0.1:8888" // req.Ja3 = "771,4865-4866-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,45-0-10-17513-16-35-18-43-51-23-13-65281-27-11-5-21,29-23-24,0" res,err:=requests.Get(rawurl+ori_params,req)

这里是测试代码

wangluozhe commented 1 year ago

新版本已更新,请你更新到1.1.0以后尝试一下看看

dream2333 commented 1 year ago

新版本已更新,请你更新到1.1.0以后尝试一下看看

依旧不行,这里是测试代码,TestRequest是使用requests请求,TestHTTP是使用net/HTTP请求,net/HTTP正常而requests库失败,请求的页面为抖音网页版用户已发布的视频列表页

`package main

import ( "fmt" "github.com/wangluozhe/requests" "github.com/wangluozhe/requests/url" "io" "log" "net/http" )

func TestRequest() { rawurl := "https://www.douyin.com/aweme/v1/web/aweme/post/?device_platform=webapp&aid=6383&channel=channel_pc_web&sec_user_id=MS4wLjABAAAACX0jKoZARKXONKiywn2yeDglLJ8oKoU5E_iPLflzKqM&max_cursor=1676028071000&locate_query=false&show_live_replay_strategy=1&count=10&publish_video_strategy_type=2&pc_client_type=1&version_code=170400&version_name=17.4.0&cookie_enabled=true&screen_width=2560&screen_height=1440&browser_language=zh-CN&browser_platform=Win32&browser_name=Edge&browser_version=113.0.1774.42&browser_online=true&engine_name=Blink&engine_version=113.0.0.0&os_name=Windows&os_version=10&cpu_core_num=12&device_memory=8&platform=PC&downlink=10&effective_type=4g&round_trip_time=50&webid=7214076632230282789&msToken=PJkiDtDJPv_gkTg0Lxkoo2yQfJOQP_CFLO0s0MwZdVj5Bt9p8ye7i22qXvlAjX3fMeQC053OkJR8ukzJZB1TS1yeIWIskE0SPtE02jE4YcYlpwqMhLxXVV6488dR6s8=&X-Bogus=DFSzswVYQOvANamNttwrDToB6lQc" headers := url.NewHeaders() headers.Set("authority", "www.douyin.com") headers.Set("accept", "application/json, text/plain, /") headers.Set("accept-language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6") headers.Set("cookie", "ttwid=1%7Cwkw7ngkVUZ3CuQne2ijSq3yEOofa-kCE_rpwkDChoHQ%7C1679658117%7C91361174e5a36c322ebcdf8c06f4316645139f0e804fa3db01e41d154995c6ec; passport_csrf_token=18cc42990eee1bfe86924b16e55bddf1; passport_csrf_token_default=18cc42990eee1bfe86924b16e55bddf1; s_v_web_id=verify_lflzxgmo_ZGYZwYa7_rHkq_4laW_BZ7u_Ooj719O2pU4P; xgplayer_user_id=295352157608; pwa2=%223%7C0%22; download_guide=%223%2F20230507%22; __ac_nonce=0646077350040fab2831c; __ac_signature=_02B4Z6wo00f01K3IqkgAAIDBvZZPcVNWNbit6K7AAE8XeVBcDAO-s1c1w4CMvfO6J.i7i.RmoCXC3-DMQ5DsjIspI69eooizvceFOPgmJZc0M2IY23AZwmd72QWRMVEJyhoY4PApw2AYnl9M72; douyin.com; strategyABtestKey=%221684043574.82%22; VIDEO_FILTER_MEMO_SELECT=%7B%22expireTime%22%3A1684648374831%2C%22type%22%3A1%7D; csrf_session_id=9e4805e3154a3ebc1e3c7737c533f2d6; tt_scid=OrOeB1qYNNIVw1GQfhLH6Nb87zdxWZoQJbqUR-1S7BCfsWPDD5JxuGP99NA3f5z0eaba; bd_ticket_guard_client_data=eyJiZC10aWNrZXQtZ3VhcmQtdmVyc2lvbiI6MiwiYmQtdGlja2V0LWd1YXJkLWl0ZXJhdGlvbi12ZXJzaW9uIjoxLCJiZC10aWNrZXQtZ3VhcmQtY2xpZW50LWNzciI6Ii0tLS0tQkVHSU4gQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tXHJcbk1JSUJEVENCdFFJQkFEQW5NUXN3Q1FZRFZRUUdFd0pEVGpFWU1CWUdBMVVFQXd3UFltUmZkR2xqYTJWMFgyZDFcclxuWVhKa01Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRXJwUVQyTm5rYkJQeGNteXk2NTZxU29PRFxyXG5LTk51WkdTamIxR0lJWjcrZXo0NFBnQ3B0R1FsUlBSSTBGTTZ1Y1FpZmd6ZlFLRm9SaEtCSC9oSUR6VDJ5NkFzXHJcbk1Db0dDU3FHU0liM0RRRUpEakVkTUJzd0dRWURWUjBSQkJJd0VJSU9kM2QzTG1SdmRYbHBiaTVqYjIwd0NnWUlcclxuS29aSXpqMEVBd0lEUndBd1JBSWdGY2NhSittdTQyT0tsNVlNUDA2bGtYR3MwaDFLUi96VElDRGhhY01GQUhzQ1xyXG5JQ2sxWXVXc08wemlwTlRuVi9Vc3A5cDI3Wnc2QlNJRmFUcGlFYjhRK2RrVVxyXG4tLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS1cclxuIn0=; msToken=fuXeZhpRb65tedIa3jLQjkn6ZM2lxDVsB5bEkq5qGoJvql34gdEr_DduzpxOxacWDnV3YqPJn5bitDYs7LMevFoq6Tt7jWVGPimBbTKjrYC_a5rB7ccP2NthLP6smKI=; home_can_add_dy_2_desktop=%221%22; msToken=PJkiDtDJPv_gkTg0Lxkoo2yQfJOQP_CFLO0s0MwZdVj5Bt9p8ye7i22qXvlAjX3fMeQC053OkJR8ukzJZB1TS1yeIWIskE0SPtE02jE4YcYlpwqMhLxXVV6488dR6s8=") headers.Set("referer", "https://www.douyin.com/user/MS4wLjABAAAACX0jKoZARKXONKiywn2yeDglLJ8oKoU5E_iPLflzKqM") headers.Set("sec-ch-ua", "Microsoft Edge";v="113", "Chromium";v="113", "Not-A.Brand";v="24") headers.Set("sec-ch-ua-mobile", "?0") headers.Set("sec-ch-ua-platform", "Windows") headers.Set("sec-fetch-dest", "empty") headers.Set("sec-fetch-mode", "cors") headers.Set("sec-fetch-site", "same-origin") headers.Set("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.42") req := url.NewRequest() req.Headers = headers res, _ := requests.Get(rawurl, req) fmt.Println(res.Text) }

func TestHTTP() { client := &http.Client{} req, _ := http.NewRequest("GET", "https://www.douyin.com/aweme/v1/web/aweme/post/?device_platform=webapp&aid=6383&channel=channel_pc_web&sec_user_id=MS4wLjABAAAACX0jKoZARKXONKiywn2yeDglLJ8oKoU5E_iPLflzKqM&max_cursor=1676028071000&locate_query=false&show_live_replay_strategy=1&count=10&publish_video_strategy_type=2&pc_client_type=1&version_code=170400&version_name=17.4.0&cookie_enabled=true&screen_width=2560&screen_height=1440&browser_language=zh-CN&browser_platform=Win32&browser_name=Edge&browser_version=113.0.1774.42&browser_online=true&engine_name=Blink&engine_version=113.0.0.0&os_name=Windows&os_version=10&cpu_core_num=12&device_memory=8&platform=PC&downlink=10&effective_type=4g&round_trip_time=50&webid=7214076632230282789&msToken=PJkiDtDJPv_gkTg0Lxkoo2yQfJOQP_CFLO0s0MwZdVj5Bt9p8ye7i22qXvlAjX3fMeQC053OkJR8ukzJZB1TS1yeIWIskE0SPtE02jE4YcYlpwqMhLxXVV6488dR6s8=&X-Bogus=DFSzswVYQOvANamNttwrDToB6lQc", nil) req.Header.Set("authority", "www.douyin.com") req.Header.Set("accept", "application/json, text/plain, /") req.Header.Set("accept-language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6") req.Header.Set("cookie", "ttwid=1%7Cwkw7ngkVUZ3CuQne2ijSq3yEOofa-kCE_rpwkDChoHQ%7C1679658117%7C91361174e5a36c322ebcdf8c06f4316645139f0e804fa3db01e41d154995c6ec; passport_csrf_token=18cc42990eee1bfe86924b16e55bddf1; passport_csrf_token_default=18cc42990eee1bfe86924b16e55bddf1; s_v_web_id=verify_lflzxgmo_ZGYZwYa7_rHkq_4laW_BZ7u_Ooj719O2pU4P; xgplayer_user_id=295352157608; pwa2=%223%7C0%22; download_guide=%223%2F20230507%22; __ac_nonce=0646077350040fab2831c; __ac_signature=_02B4Z6wo00f01K3IqkgAAIDBvZZPcVNWNbit6K7AAE8XeVBcDAO-s1c1w4CMvfO6J.i7i.RmoCXC3-DMQ5DsjIspI69eooizvceFOPgmJZc0M2IY23AZwmd72QWRMVEJyhoY4PApw2AYnl9M72; douyin.com; strategyABtestKey=%221684043574.82%22; VIDEO_FILTER_MEMO_SELECT=%7B%22expireTime%22%3A1684648374831%2C%22type%22%3A1%7D; csrf_session_id=9e4805e3154a3ebc1e3c7737c533f2d6; tt_scid=OrOeB1qYNNIVw1GQfhLH6Nb87zdxWZoQJbqUR-1S7BCfsWPDD5JxuGP99NA3f5z0eaba; bd_ticket_guard_client_data=eyJiZC10aWNrZXQtZ3VhcmQtdmVyc2lvbiI6MiwiYmQtdGlja2V0LWd1YXJkLWl0ZXJhdGlvbi12ZXJzaW9uIjoxLCJiZC10aWNrZXQtZ3VhcmQtY2xpZW50LWNzciI6Ii0tLS0tQkVHSU4gQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tXHJcbk1JSUJEVENCdFFJQkFEQW5NUXN3Q1FZRFZRUUdFd0pEVGpFWU1CWUdBMVVFQXd3UFltUmZkR2xqYTJWMFgyZDFcclxuWVhKa01Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRXJwUVQyTm5rYkJQeGNteXk2NTZxU29PRFxyXG5LTk51WkdTamIxR0lJWjcrZXo0NFBnQ3B0R1FsUlBSSTBGTTZ1Y1FpZmd6ZlFLRm9SaEtCSC9oSUR6VDJ5NkFzXHJcbk1Db0dDU3FHU0liM0RRRUpEakVkTUJzd0dRWURWUjBSQkJJd0VJSU9kM2QzTG1SdmRYbHBiaTVqYjIwd0NnWUlcclxuS29aSXpqMEVBd0lEUndBd1JBSWdGY2NhSittdTQyT0tsNVlNUDA2bGtYR3MwaDFLUi96VElDRGhhY01GQUhzQ1xyXG5JQ2sxWXVXc08wemlwTlRuVi9Vc3A5cDI3Wnc2QlNJRmFUcGlFYjhRK2RrVVxyXG4tLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS1cclxuIn0=; msToken=fuXeZhpRb65tedIa3jLQjkn6ZM2lxDVsB5bEkq5qGoJvql34gdEr_DduzpxOxacWDnV3YqPJn5bitDYs7LMevFoq6Tt7jWVGPimBbTKjrYC_a5rB7ccP2NthLP6smKI=; home_can_add_dy_2_desktop=%221%22; msToken=PJkiDtDJPv_gkTg0Lxkoo2yQfJOQP_CFLO0s0MwZdVj5Bt9p8ye7i22qXvlAjX3fMeQC053OkJR8ukzJZB1TS1yeIWIskE0SPtE02jE4YcYlpwqMhLxXVV6488dR6s8=") req.Header.Set("referer", "https://www.douyin.com/user/MS4wLjABAAAACX0jKoZARKXONKiywn2yeDglLJ8oKoU5E_iPLflzKqM") req.Header.Set("sec-ch-ua", "Microsoft Edge";v="113", "Chromium";v="113", "Not-A.Brand";v="24") req.Header.Set("sec-ch-ua-mobile", "?0") req.Header.Set("sec-ch-ua-platform", "Windows") req.Header.Set("sec-fetch-dest", "empty") req.Header.Set("sec-fetch-mode", "cors") req.Header.Set("sec-fetch-site", "same-origin") req.Header.Set("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.42") resp, err := client.Do(req) if err != nil { log.Fatal(err) } defer resp.Body.Close() bodyText, _ := io.ReadAll(resp.Body) fmt.Printf("%s\n", bodyText) } func main() { TestRequest() } `

jqqjj commented 1 year ago

另外,没有做到io复用,虽然代码看到有写

meesii commented 11 months ago

最新版也不行,这个网址可以测试 https://tools.scrapfly.io/api/fp/akamai?extended=1 会返回“Bad HTTP version - only for HTTP/2.0”