Open TDK1969 opened 1 year ago
经过测试,访问该接口需要在cookies中携带一个合法的bvuid3
。
我的爬虫代码是先访问主站,然后携带cookie去访问该接口,但现在不可行了,可能该接口需要额外的验证环节?
之前都是无登录的情况下进行访问
curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" -e "https://space.bilibili.com/2/dynamic" "https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=2&offset="
我这边不带bvuid3
没有出现-352
问题
curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" -e "https://space.bilibili.com/2/dynamic" "https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=2&offset="
我这边不带
bvuid3
没有出现-352
问题
很奇怪,我换成你的UA之后也能不带bvuid3访问了,我懵逼了
curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" -e "https://space.bilibili.com/2/dynamic" "https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=2&offset="
我这边不带
bvuid3
没有出现-352
问题
现在我这里这个命令又不行了,我不懂了
@HashLiver 执行结果如图,还是不太懂
不行了,现在这个接口必须携带cookie才能访问了,不需要登录但是必须要cookie,否则返回-352 并且即使加上cookie 也不是一定可以成功访问 还是有一定几率返回-352,比如当你频繁访问的时候
我测试的跟ccokie没关系。跟ua有关系,有些ua可能是被拉黑了,所以352了。换个正常的ua应该就好了。(目前我观测的结果是这样,至于有没有其他方面的影响就不知道了
这下真的麻了,换什么UA都没用了 :(
@lovetingyuan 不登录的cookie怎么获得呀😢
import uuid
import random
import requests
headers = {
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
"referer": "https://www.bilibili.com",
}
cookies = {
"buvid3": "{}{:05d}infoc".format(uuid.uuid4(), random.randint(1, 99999))
}
r = requests.get("https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=2", headers=headers, cookies=cookies)
print(r.text)
@lovetingyuan 不登录的cookie怎么获得呀😢
不登录访问b站 仍然会注入一些cookie的 可以打开浏览器看看 document.cookie
@lovetingyuan 不登录的cookie怎么获得呀😢
不登录访问b站 仍然会注入一些cookie的 可以打开浏览器看看 document.cookie
我以前的做法是先访问主站再访问这个接口,还是寄
import uuid import random import requests headers = { "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36", "referer": "https://www.bilibili.com", } cookies = { "buvid3": "{}{:05d}infoc".format(uuid.uuid4(), random.randint(1, 99999)) } r = requests.get("https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=2", headers=headers, cookies=cookies) print(r.text)
这个意思是可以自己生成bvuid3吗?有意思,回去试一下
r = requests.get("https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=2", headers=headers, cookies=cookies)
我试了一下,是可以的,但不知道能管多久
我尝试了下, 好像只需要在cookie里面的DedeUserID为动态所属用户id就行了
我尝试了下, 好像只需要在cookie里面的DedeUserID为动态所属用户id就行了
太秀了哥,这是怎么发现的
太秀了哥,这是怎么发现的
把cookie里面的参数一个个删, 发现把DedeUserID
删了就开始报-352
了
没用,这个请求多了还是会-352
def get_user_space(user_id: int):
r1 = httpx.get(
"https://space.bilibili.com/1133258171/dynamic",
headers={
"Host": "space.bilibili.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
},
)
print(r1.cookies)
url = "https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi"
payload ='' # payload 是浏览器指纹,自己去抓一下用,浏览器无痕模式进入某个动态页面即可抓到
headers = {
"authority": "api.bilibili.com",
"accept": "*/*",
"accept-language": "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6",
"content-type": "application/json;charset=UTF-8",
"dnt": "1",
"origin": "https://space.bilibili.com",
"referer": "https://space.bilibili.com/1133258171/dynamic",
"sec-ch-ua": '"Not.A/Brand";v="8", "Chromium";v="114", "Microsoft Edge";v="114"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-site",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58",
}
response = httpx.post(url, headers=headers, data=payload, cookies=r1.cookies)
print(response.json)
params = {
"host_mid": user_id,
}
r = httpx.get(
"https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space",
params=params,
headers={
"Host": "api.bilibili.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
},
cookies=r1.cookies,
)
res = r.json()
print(res)
return res
get_user_space(1133258171)
payload参数注意需要自己抓一下包,应该是浏览器指纹,我就不放出来了。
浏览器测试的话需要打开无痕模式,第一次打开页面会带着从/dynamic
获取的cookie
去请求 /x/internal/gaia-gateway/ExClimbWuzhi
接口,但注意这个接口会被广告屏蔽软件如ublock屏蔽,导出跳出验证码。
如果你尝试在无痕模式清除cookie
后请求页面,会触发验证码
@renmu123 试了下你这个,请求都对,但最后的请求还是会-352 另外今天开始好像这个接口变的更严了,只传DedeUserID没效果了。隐私模式访问动态页面会直接弹验证码,过了验证码才刷出来 @SocialSisterYi 考虑下重开issue吧
@renmu123 试了下你这个,请求都对,但最后的请求还是会-352 另外今天开始好像这个接口变的更严了,只传DedeUserID没效果了。隐私模式访问动态页面会直接弹验证码,过了验证码才刷出来 @SocialSisterYi 考虑下重开issue吧
目前我还是可以使用的,payload
参数注意需要自己抓一下包,应该是浏览器指纹,我就不放出来了。
浏览器测试的话 /x/internal/gaia-gateway/ExClimbWuzhi
接口会被广告屏蔽软件如ublock
屏蔽,导出跳出验证码
之前使用已登录的cookie时是可以正常访问的,但今天收紧之后也不太好使了。蹲一个解决方法
收紧+1 有时候刷好几次才刷出来 刚试了下 必须要有buvid3这个cookie,并且这个buvid3并不是随便取的,试过通过接口取或者通过代码生成但是还是不行,只有通过访问官网然后拿官网种的cookie才有效
失效+1 蹲一个方法
尝试使用selenium获取cookie但好像没什么用
没找到解决方法,登录不是必须,但是cookie是必要,很奇怪浏览器点点点可以访问,搞后端就报-352,中间浏览器应该发生了什么以至于可以访问
我测试了一下,感觉只要把 SESSDATA 加在Cookie很像就可以了
const axios = require("axios");
const fs = require("fs");
const YOUR_SESSDATA = fs.readFileSync("./session_data.txt",{encoding:"utf8"});
main();
async function main() {
let channel_id = 1;
while(true) {
var options = {
method:'get',
url: `https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?offset=&host_mid=${channel_id}&timezone_offset=-480&features=itemOpusStyle`,
headers: {
'Cookie':`SESSDATA=${YOUR_SESSDATA}`,
},
};
await axios.request(options).then((result) => {
console.log(result.data);
}).catch((err) => {
console.log(err);
});
console.log(channel_id);
channel_id++;
}
}
但是有时候却会返回空的数据(例如:13,14 ) 坐等大佬解决空数据的问题,因为我感觉空数据很像是随机触发的
之前用的DedeUserID,现在失效了,好像强制必须有buvid3了,或者登录状态下的SESSDATA,但用自己号的相关信息还是太不安全了,被当成机器人封号血亏
renmu123贴的是可以用的,但他正文带几个错别字,我也看了半天才看懂。 首先关掉广告屏蔽扩展,浏览器无痕打开某个动态(或者清除cookie刷新),第一次打开会有验证码,这时不继续操作,直接刷新,刷新后你会发现这次和以后就没有验证码了,从中你可以发现些端倪,就是renmu123所说的。(如果你首次打开时没关闭广告屏蔽扩展,就会一直有验证码,即使你后面中途关闭扩展也会弹验证码)
log-reporter.js里搜payload可以看到payload怎么来的,里面基本上就是浏览器指纹加个timestamp和_uuid,_uuid是本地生成的,也是在这个文件里搜infoc就能搜到,不知道是否任意生成皆可。
带dynamic返回的cookie(包含buvid3)和payload请求过一次ExClimbWuzhi后,后续对dynamic的请求用相同的buvid3+任意UA,就不会-352了,有效期好像还挺长的。相对的,如果你是用手动验证码后的buvid3,请求不需要附带UA,但只有半小时有效期(半小时后带UA是否仍有效我没测试)
@UnluckyNinja 感谢提醒,用SESSDATA的确有风险。我也测试了一下 @renmu123 的方法,的确是可以用的。
以下是我用nodejs实现的方法(感谢 @renmu123 提供的代码,我只是翻译成nodejs而已)
const axios = require("axios");
const fs = require("fs");
main();
async function test() {
var payload = fs.readFileSync("./payload.txt", {encoding:"utf8"});
var cookie_buvid3;
var options = {
"method":"post",
"url":"https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi",
"headers": {
"Content-Type":"application/json",
},
"data": payload,
};
await axios.request(`https://space.bilibili.com/1/dynamic`).then((res) => {
var all_headers = res.headers["set-cookie"][0];
cookie_buvid3 = all_headers.split(" ");
cookie_buvid3 = cookie_buvid3[0];
}).catch((err) => {
console.log(err);
});
options.headers["Cookie"] = cookie_buvid3;
await axios.request(options).then((res) => {
console.log(res.data);
}).catch((err) => {
console.log(err);
});
return cookie_buvid3;
}
async function main() {
var buvid3 = await test();
console.log(buvid3);
var base_id = 1;
var options = {
"method":"get",
"url":`https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?offset=&host_mid=${base_id}&timezone_offset=-480&features=itemOpusSty`,
"headers": {
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
"Cookie": buvid3,
},
};
console.log(options);
while(true) {
await axios.request(options).then((res) => {
console.log(res.data);
}).catch((err) => {
console.log(err);
});
console.log(base_id);
base_id++;
};
}
然后我想补充一下抓payload的方法,https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi
这个接口只有在第一次访问bilibili才会触发。(以下图片给不会抓payload的人知道怎样抓,因为我自己也是卡了很久 :D )
刷新页面之后就会不见了
在我尝试研究SESSDATA如何生成时,发现了SESSDATA似乎是可以乱写的。只要是User-Agent放Mozilla/5.0 就可以了。
const axios = require("axios");
main();
async function main() {
let channel_id = 1;
while(true) {
var options = {
method:'get',
url: `https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?offset=&host_mid=${channel_id}&timezone_offset=-480&features=itemOpusStyle`,
headers: {
'Cookie':'SESSDATA=xie_xie_bilibili.com',
'User-Agent':'Mozilla/5.0',
},
};
await axios.request(options).then((result) => {
console.log(result.data);
}).catch((err) => {
console.log(err);
});
console.log(channel_id);
channel_id++;
}
}
我不确定是不是只有我这样 😂
在我尝试研究SESSDATA如何生成时,发现了SESSDATA似乎是可以乱写的。只要是User-Agent放Mozilla/5.0 就可以了。
const axios = require("axios"); main(); async function main() { let channel_id = 1; while(true) { var options = { method:'get', url: `https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?offset=&host_mid=${channel_id}&timezone_offset=-480&features=itemOpusStyle`, headers: { 'Cookie':'SESSDATA=xie_xie_bilibili.com', 'User-Agent':'Mozilla/5.0', }, }; await axios.request(options).then((result) => { console.log(result.data); }).catch((err) => { console.log(err); }); console.log(channel_id); channel_id++; } }
我不确定是不是只有我这样 😂
666,估计能用个两三周。话说我看了下之前的space_history老接口又能匿名访问了,建议有稳定性需求的可以 payload 和老接口都当作备用方式
UA为Mozilla/5.0
时不需要校验, 可直接请求https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space
curl -H "User-Agent: Mozilla/5.0" https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=1
上面的命令可以直接返回{"code":0,"message":"0","ttl":1,"data":{...}}
对于UA有额外参数的情况, payload也只需要3064, 39c8, 3c43即可, 3c43中也只要adca, 其他都可以删除, 详见下面的python示例
headers = {
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0',
# 'referer': 'https://www.bilibili.com/',
# 'accept-language': 'zh-CN,zh;q=0.9',
}
cookies = {
'_uuid': f'{str(uuid.uuid4()).upper()}{random.randint(0, 99999):05d}infoc'
}
response = requests.get(
'https://space.bilibili.com/1/dynamic',
headers=headers,
cookies=cookies
)
print(response.cookies)
spm_prefix = re.search(r'<meta name="spm_prefix" content="([^"]+?)">', response.text).group(1)
cookies.update(response.cookies.get_dict())
data = {
'3064': 1,
'39c8': f'{spm_prefix}.fp.risk',
'3c43': {
'adca': 'Win32' if 'Windows' in headers.get('user-agent', '') else 'Linux',
},
# 'df35': cookies['_uuid']
}
json_data = {
'payload': json.dumps(data, separators=(',', ':'))
}
response = requests.post(
'https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi',
cookies=cookies,
headers=headers,
json=json_data,
)
print(response.text)
print(response.cookies)
params = {
'offset': '',
'host_mid': '1',
'timezone_offset': '-480',
}
response = requests.get(
'https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space',
params=params,
cookies=cookies,
headers=headers,
)
print(response.text)
UA为
Mozilla/5.0
时不需要校验, 可直接请求https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space
curl -H "User-Agent: Mozilla/5.0" https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space?host_mid=1
上面的命令可以直接返回
{"code":0,"message":"0","ttl":1,"data":{...}}
对于UA有额外参数的情况, payload也只需要3064, 39c8, 3c43即可, 3c43中也只要adca, 其他都可以删除, 详见下面的python示例
headers = { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0', # 'referer': 'https://www.bilibili.com/', # 'accept-language': 'zh-CN,zh;q=0.9', } cookies = { '_uuid': f'{str(uuid.uuid4()).upper()}{random.randint(0, 99999):05d}infoc' } response = requests.get( 'https://space.bilibili.com/1/dynamic', headers=headers, cookies=cookies ) print(response.cookies) spm_prefix = re.search(r'<meta name="spm_prefix" content="([^"]+?)">', response.text).group(1) cookies.update(response.cookies.get_dict()) data = { '3064': 1, '39c8': f'{spm_prefix}.fp.risk', '3c43': { 'adca': 'Win32' if 'Windows' in headers.get('user-agent', '') else 'Linux', }, # 'df35': cookies['_uuid'] } json_data = { 'payload': json.dumps(data, separators=(',', ':')) } response = requests.post( 'https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi', cookies=cookies, headers=headers, json=json_data, ) print(response.text) print(response.cookies) params = { 'offset': '', 'host_mid': '1', 'timezone_offset': '-480', } response = requests.get( 'https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space', params=params, cookies=cookies, headers=headers, ) print(response.text)
这个方法已经-352了
@kaosaa 我试了一下,以上方法还可以使用。我觉得应该是你的问题
不清楚 我用自己账号的cookie可以 照抄以上的方法返回的是352 风控好像是用来让正常用户看不见被阿瓦隆系统屏蔽的动态 但是自己是可以看见的
------------------ 原始邮件 ------------------ 发件人: 00randomize00 @.> 发送时间: 2023年11月13日 15:43 收件人: SocialSisterYi/bilibili-API-collect @.> 抄送: kaosaa @.>, Mention @.> 主题: Re: [SocialSisterYi/bilibili-API-collect] 使用/x/polymer/web-dynamic/v1/feed/space接口获取动态时返回-352错误 (Issue #686)
那个python demo我这边也失效了....
现在还需要校验bfe9
, 可能是webgl画的一个png图片
headers = {
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0',
# 'referer': 'https://www.bilibili.com/',
# 'accept-language': 'zh-CN,zh;q=0.9',
}
cookies = {
'_uuid': f'{str(uuid.uuid4()).upper()}{random.randint(0, 99999):05d}infoc'
}
response = requests.get(
'https://space.bilibili.com/1/dynamic',
headers=headers,
cookies=cookies
)
print(response.cookies)
spm_prefix = re.search(r'<meta name="spm_prefix" content="([^"]+?)">', response.text).group(1)
cookies.update(response.cookies.get_dict())
rand_png_end = (random.randbytes(32) + bytes(4) + b'IEND' + random.randbytes(4)) # PNG结束部分
data = {
'3064': 1,
'39c8': f'{spm_prefix}.fp.risk',
'3c43': {
'adca': 'Win32' if 'Windows' in headers.get('user-agent', '') else 'Linux',
'bfe9': base64.b64encode(rand_png_end).decode()[-50:],
},
# 'df35': cookies['_uuid']
}
json_data = {
'payload': json.dumps(data, separators=(',', ':'))
}
response = requests.post(
'https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi',
cookies=cookies,
headers=headers,
json=json_data,
)
print(response.text)
print(response.cookies)
params = {
'offset': '',
'host_mid': '1',
'timezone_offset': '-480',
}
response = requests.get(
'https://api.bilibili.com/x/polymer/web-dynamic/v1/feed/space',
params=params,
cookies=cookies,
headers=headers,
)
print(response.text)
https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi 的作用是什么有人知道吗?实在没看懂
在这里https://github.com/SocialSisterYi/bilibili-API-collect/issues/868 获取最新的讨论
https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi 的作用是什么有人知道吗?实在没看懂
用来激活cookie,如果不调用那么获得的cookie无法生效
api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi 的作用是什么有人知道吗?实在没看懂
用来激活cookie,如果不调用那么获得的cookie无法生效
激活 HTML 那边拿到的 cookie?spi 拿到的 buvid3/4 也需要这样激活吗?
昨天早上写的爬虫还能无忧无虑的爬取,今天接口返回就是: { "code": -352, "message": "-352", "ttl": 1 } 然后我发现,校验的参数是 cookie 中的 buvid3 参数,这个参数从浏览器那边复制过来,在接口测试中的单个请求是能够使用的,可以拿到正常的返回值;(这个也不能做批量使用,应为在第一个请求中正确响应后,如果后续的请求cookie 还设置这个,仍然是会报错。。。就很奇怪)
另外从接口获取的是无效的,模拟请求主站,从主站的响应中提取的 cookie 同样无效; 刚看到 ExClimbWuzhi 这个接口的激活机制,等下试下再反馈;
api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi 的作用是什么有人知道吗?实在没看懂
用来激活cookie,如果不调用那么获得的cookie无法生效
激活 HTML 那边拿到的 cookie?spi 拿到的 buvid3/4 也需要这样激活吗?
spi接口好像除了能拿到符合规范的buvid3,别的没什么用,我所知道的无需登录却需要buvid3
参数cookie的接口,都未对buvid3进行校验,也就是说你随便填一个都行。ps: 但是如果你长时间使用同一个buvid3,貌似会被风控。
api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi 的作用是什么有人知道吗?实在没看懂
用来激活cookie,如果不调用那么获得的cookie无法生效
激活 HTML 那边拿到的 cookie?spi 拿到的 buvid3/4 也需要这样激活吗?
spi接口好像除了能拿到符合规范的buvid3,别的没什么用,我所知道的无需登录却需要
buvid3
参数cookie的接口,都未对buvid3进行校验,也就是说你随便填一个都行。ps: 但是如果你长时间使用同一个buvid3,貌似会被风控。
我看好像 spi 拿到的 buvid3 也会接着激活一次,emmm
简单修改了一下爬虫,按照上面的讨论内容,构建一下机制:
Additional info for registering buvid(?):
**Attention: DO NOT JUST COPY THE FOLLOWING EXAMPLES!!!***
https://api.bilibili.com/x/frontend/finger/spi
{
"code":0,
"data":{
"b_3":"6DEAB968-BFFF-82D2-42FC-6A148AF1A91A14121infoc", // buvid3
"b_4":"950387F6-81BA-FA85-1C1C-084C0C68537E14121-024011008-mqHh6TGQlnVgXazy647zgAx1XBsd+g7y946DRln50pkrLmYNRsOEtIPaLiatNJt0" // buvid4
},
"message":"ok"
}
POST https://api.bilibili.com/x/internal/gaia-gateway/ExClimbWuzhi
innersign=0; buvid3=6DEAB968-BFFF-82D2-42FC-6A148AF1A91A14121infoc; b_nut=1704873471; i-wanna-go-back=-1; b_ut=7; b_lsid=9910433CB_18CF260AB89; _uuid=312C2F31-1D48-E108C-4232-D7E96B104A8D1070864infoc; enable_web_push=DISABLE; header_theme_version=undefined; home_feed_column=4; browser_resolution=839-959; buvid4=950387F6-81BA-FA85-1C1C-084C0C68537E14121-024011008-mqHh6TGQlnVgXazy647zgAx1XBsd+g7y946DRln50pkrLmYNRsOEtIPaLiatNJt0; buvid_fp=946265982c5f2c530cfed6f97df5cf65
Related algorithms generating _uuid
or buvid_fp
are still unknown and need more investigation.
payload:"{\"3064\":1,\"5062\":\"1704873471951\",\"03bf\":\"https%3A%2F%2Fwww.bilibili.com%2F\",\"39c8\":\"333.1007.fp.risk\",\"34f1\":\"\",\"d402\":\"\",\"654a\":\"\",\"6e7c\":\"839x959\",\"3c43\":{\"2673\":1,\"5766\":24,\"6527\":0,\"7003\":1,\"807e\":1,\"b8ce\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\",\"641c\":0,\"07a4\":\"zh-CN\",\"1c57\":8,\"0bd0\":16,\"748e\":[2560,1080],\"d61f\":[2467,1091],\"fc9d\":480,\"6aa9\":\"Asia/Hong_Kong\",\"75b8\":1,\"3b21\":1,\"8a1c\":0,\"d52f\":\"not available\",\"adca\":\"Win32\",\"80c9\":[[\"PDF Viewer\",\"Portable Document Format\",[[\"application/pdf\",\"pdf\"],[\"text/pdf\",\"pdf\"]]],[\"Chrome PDF Viewer\",\"Portable Document Format\",[[\"application/pdf\",\"pdf\"],[\"text/pdf\",\"pdf\"]]],[\"Chromium PDF Viewer\",\"Portable Document Format\",[[\"application/pdf\",\"pdf\"],[\"text/pdf\",\"pdf\"]]],[\"Microsoft Edge PDF Viewer\",\"Portable Document Format\",[[\"application/pdf\",\"pdf\"],[\"text/pdf\",\"pdf\"]]],[\"WebKit built-in PDF\",\"Portable Document Format\",[[\"application/pdf\",\"pdf\"],[\"text/pdf\",\"pdf\"]]]],\"13ab\":\"mTUAAAAASUVORK5CYII=\",\"bfe9\":\"aTot0S1jJ7Ws0JC6QkvAL/A4H1PbV+/QA3AAAAAElFTkSuQmCC\",\"a3c1\":[\"extensions:ANGLE_instanced_arrays;EXT_blend_minmax;EXT_color_buffer_half_float;EXT_disjoint_timer_query;EXT_float_blend;EXT_frag_depth;EXT_shader_texture_lod;EXT_texture_compression_bptc;EXT_texture_compression_rgtc;EXT_texture_filter_anisotropic;EXT_sRGB;KHR_parallel_shader_compile;OES_element_index_uint;OES_fbo_render_mipmap;OES_standard_derivatives;OES_texture_float;OES_texture_float_linear;OES_texture_half_float;OES_texture_half_float_linear;OES_vertex_array_object;WEBGL_color_buffer_float;WEBGL_compressed_texture_s3tc;WEBGL_compressed_texture_s3tc_srgb;WEBGL_debug_renderer_info;WEBGL_debug_shaders;WEBGL_depth_texture;WEBGL_draw_buffers;WEBGL_lose_context;WEBGL_multi_draw\",\"webgl aliased line width range:[1, 1]\",\"webgl aliased point size range:[1, 1024]\",\"webgl alpha bits:8\",\"webgl antialiasing:yes\",\"webgl blue bits:8\",\"webgl depth bits:24\",\"webgl green bits:8\",\"webgl max anisotropy:16\",\"webgl max combined texture image units:32\",\"webgl max cube map texture size:16384\",\"webgl max fragment uniform vectors:1024\",\"webgl max render buffer size:16384\",\"webgl max texture image units:16\",\"webgl max texture size:16384\",\"webgl max varying vectors:30\",\"webgl max vertex attribs:16\",\"webgl max vertex texture image units:16\",\"webgl max vertex uniform vectors:4095\",\"webgl max viewport dims:[32767, 32767]\",\"webgl red bits:8\",\"webgl renderer:WebKit WebGL\",\"webgl shading language version:WebGL GLSL ES 1.0 (OpenGL ES GLSL ES 1.0 Chromium)\",\"webgl stencil bits:0\",\"webgl vendor:WebKit\",\"webgl version:WebGL 1.0 (OpenGL ES 2.0 Chromium)\",\"webgl unmasked vendor:Google Inc. (NVIDIA) #X3fQVPgERx\",\"webgl unmasked renderer:ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 Laptop GPU (0x00002560) Direct3D11 vs_5_0 ps_5_0, D3D11) #X3fQVPgERx\",\"webgl vertex shader high float precision:23\",\"webgl vertex shader high float precision rangeMin:127\",\"webgl vertex shader high float precision rangeMax:127\",\"webgl vertex shader medium float precision:23\",\"webgl vertex shader medium float precision rangeMin:127\",\"webgl vertex shader medium float precision rangeMax:127\",\"webgl vertex shader low float precision:23\",\"webgl vertex shader low float precision rangeMin:127\",\"webgl vertex shader low float precision rangeMax:127\",\"webgl fragment shader high float precision:23\",\"webgl fragment shader high float precision rangeMin:127\",\"webgl fragment shader high float precision rangeMax:127\",\"webgl fragment shader medium float precision:23\",\"webgl fragment shader medium float precision rangeMin:127\",\"webgl fragment shader medium float precision rangeMax:127\",\"webgl fragment shader low float precision:23\",\"webgl fragment shader low float precision rangeMin:127\",\"webgl fragment shader low float precision rangeMax:127\",\"webgl vertex shader high int precision:0\",\"webgl vertex shader high int precision rangeMin:31\",\"webgl vertex shader high int precision rangeMax:30\",\"webgl vertex shader medium int precision:0\",\"webgl vertex shader medium int precision rangeMin:31\",\"webgl vertex shader medium int precision rangeMax:30\",\"webgl vertex shader low int precision:0\",\"webgl vertex shader low int precision rangeMin:31\",\"webgl vertex shader low int precision rangeMax:30\",\"webgl fragment shader high int precision:0\",\"webgl fragment shader high int precision rangeMin:31\",\"webgl fragment shader high int precision rangeMax:30\",\"webgl fragment shader medium int precision:0\",\"webgl fragment shader medium int precision rangeMin:31\",\"webgl fragment shader medium int precision rangeMax:30\",\"webgl fragment shader low int precision:0\",\"webgl fragment shader low int precision rangeMin:31\",\"webgl fragment shader low int precision rangeMax:30\"],\"6bc5\":\"Google Inc. (NVIDIA) #X3fQVPgERx~ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 Laptop GPU (0x00002560) Direct3D11 vs_5_0 ps_5_0, D3D11) #X3fQVPgERx\",\"ed31\":0,\"72bd\":0,\"097b\":0,\"52cd\":[0,0,0],\"a658\":[\"Arial\",\"Arial Black\",\"Arial Narrow\",\"Book Antiqua\",\"Bookman Old Style\",\"Calibri\",\"Cambria\",\"Cambria Math\",\"Century\",\"Century Gothic\",\"Century Schoolbook\",\"Comic Sans MS\",\"Consolas\",\"Courier\",\"Courier New\",\"Georgia\",\"Helvetica\",\"Helvetica Neue\",\"Impact\",\"Lucida Bright\",\"Lucida Calligraphy\",\"Lucida Console\",\"Lucida Fax\",\"Lucida Handwriting\",\"Lucida Sans\",\"Lucida Sans Typewriter\",\"Lucida Sans Unicode\",\"Microsoft Sans Serif\",\"Monotype Corsiva\",\"MS Gothic\",\"MS PGothic\",\"MS Reference Sans Serif\",\"MS Sans Serif\",\"MS Serif\",\"Palatino Linotype\",\"Segoe Print\",\"Segoe Script\",\"Segoe UI\",\"Segoe UI Light\",\"Segoe UI Semibold\",\"Segoe UI Symbol\",\"Tahoma\",\"Times\",\"Times New Roman\",\"Trebuchet MS\",\"Verdana\",\"Wingdings\",\"Wingdings 2\",\"Wingdings 3\"],\"d02f\":\"124.04347527516074\"},\"54ef\":\"{\\\"b_ut\\\":\\\"7\\\",\\\"home_version\\\":\\\"V8\\\",\\\"i-wanna-go-back\\\":\\\"-1\\\",\\\"in_new_ab\\\":true,\\\"ab_version\\\":{\\\"for_ai_home_version\\\":\\\"V8\\\",\\\"tianma_banner_inline\\\":\\\"CONTROL\\\",\\\"enable_web_push\\\":\\\"DISABLE\\\"},\\\"ab_split_num\\\":{\\\"for_ai_home_version\\\":54,\\\"tianma_banner_inline\\\":54,\\\"enable_web_push\\\":10}}\",\"8b94\":\"\",\"df35\":\"312C2F31-1D48-E108C-4232-D7E96B104A8D1070864infoc\",\"07a4\":\"zh-CN\",\"5f45\":null,\"db46\":0}"
{
"3064":1, // ptype, mobile => 2, others => 1
"5062":"1704873471951", // timestamp
"03bf":"https%3A%2F%2Fwww.bilibili.com%2F", // url accessed
"39c8":"333.1007.fp.risk", // spm_id,
"34f1":"", // target_url, default empty now
"d402":"", // screenx, default empty
"654a":"", // screeny, default empty
"6e7c":"839x959", // browser_resolution, window.innerWidth || document.body && document.body.clientWidth + "x" + window.innerHeight || document.body && document.body.clientHeight
"3c43":{ // 3c43 => msg
"2673":1, // hasLiedResolution, window.screen.width < window.screen.availWidth || window.screen.height < window.screen.availHeight
"5766":24, // colorDepth, window.screen.colorDepth
"6527":0, // addBehavior, !!window.HTMLElement.prototype.addBehavior, html5 api
"7003":1, // indexedDb, !!window.indexedDB, html5 api
"807e":1, // cookieEnabled, navigator.cookieEnabled
"b8ce":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", // ua
"641c":0, // webdriver, navigator.webdriver, like Selenium
"07a4":"zh-CN", // language
"1c57":8, // deviceMemory in GB, navigator.deviceMemory
"0bd0":16, // hardwareConcurrency, navigator.hardwareConcurrency
"748e":[
2560, // window.screen.width
1080 // window.screen.height
], // screenResolution
"d61f":[
2467, // window.screen.availWidth
1091 // window.screen.availHeight
], // availableScreenResolution
"fc9d":480, // timezoneOffset, (new Date).getTimezoneOffset()
"6aa9":"Asia/Hong_Kong", // timezone, (new window.Intl.DateTimeFormat).resolvedOptions().timeZone
"75b8":1, // sessionStorage, window.sessionStorage, html5 api
"3b21":1, // localStorage, window.localStorage, html5 api
"8a1c":0, // openDatabase, window.openDatabase, html5 api
"d52f":"not available", // cpuClass, navigator.cpuClass
"adca":"Win32", // platform, navigator.platform
"80c9":[
[
"PDF Viewer",
"Portable Document Format",
[
[
"application/pdf",
"pdf"
],
[
"text/pdf",
"pdf"
]
]
],
[
"Chrome PDF Viewer",
"Portable Document Format",
[
[
"application/pdf",
"pdf"
],
[
"text/pdf",
"pdf"
]
]
],
[
"Chromium PDF Viewer",
"Portable Document Format",
[
[
"application/pdf",
"pdf"
],
[
"text/pdf",
"pdf"
]
]
],
[
"Microsoft Edge PDF Viewer",
"Portable Document Format",
[
[
"application/pdf",
"pdf"
],
[
"text/pdf",
"pdf"
]
]
],
[
"WebKit built-in PDF",
"Portable Document Format",
[
[
"application/pdf",
"pdf"
],
[
"text/pdf",
"pdf"
]
]
]
], // plugins
"13ab":"mTUAAAAASUVORK5CYII=", // canvas fingerprint
"bfe9":"aTot0S1jJ7Ws0JC6QkvAL/A4H1PbV+/QA3AAAAAElFTkSuQmCC", // webgl_str
"a3c1":[
"extensions:ANGLE_instanced_arrays;EXT_blend_minmax;EXT_color_buffer_half_float;EXT_disjoint_timer_query;EXT_float_blend;EXT_frag_depth;EXT_shader_texture_lod;EXT_texture_compression_bptc;EXT_texture_compression_rgtc;EXT_texture_filter_anisotropic;EXT_sRGB;KHR_parallel_shader_compile;OES_element_index_uint;OES_fbo_render_mipmap;OES_standard_derivatives;OES_texture_float;OES_texture_float_linear;OES_texture_half_float;OES_texture_half_float_linear;OES_vertex_array_object;WEBGL_color_buffer_float;WEBGL_compressed_texture_s3tc;WEBGL_compressed_texture_s3tc_srgb;WEBGL_debug_renderer_info;WEBGL_debug_shaders;WEBGL_depth_texture;WEBGL_draw_buffers;WEBGL_lose_context;WEBGL_multi_draw",
"webgl aliased line width range:[1, 1]",
"webgl aliased point size range:[1, 1024]",
"webgl alpha bits:8",
"webgl antialiasing:yes",
"webgl blue bits:8",
"webgl depth bits:24",
"webgl green bits:8",
"webgl max anisotropy:16",
"webgl max combined texture image units:32",
"webgl max cube map texture size:16384",
"webgl max fragment uniform vectors:1024",
"webgl max render buffer size:16384",
"webgl max texture image units:16",
"webgl max texture size:16384",
"webgl max varying vectors:30",
"webgl max vertex attribs:16",
"webgl max vertex texture image units:16",
"webgl max vertex uniform vectors:4095",
"webgl max viewport dims:[32767, 32767]",
"webgl red bits:8",
"webgl renderer:WebKit WebGL",
"webgl shading language version:WebGL GLSL ES 1.0 (OpenGL ES GLSL ES 1.0 Chromium)",
"webgl stencil bits:0",
"webgl vendor:WebKit",
"webgl version:WebGL 1.0 (OpenGL ES 2.0 Chromium)",
"webgl unmasked vendor:Google Inc. (NVIDIA) #X3fQVPgERx",
"webgl unmasked renderer:ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 Laptop GPU (0x00002560) Direct3D11 vs_5_0 ps_5_0, D3D11) #X3fQVPgERx",
"webgl vertex shader high float precision:23",
"webgl vertex shader high float precision rangeMin:127",
"webgl vertex shader high float precision rangeMax:127",
"webgl vertex shader medium float precision:23",
"webgl vertex shader medium float precision rangeMin:127",
"webgl vertex shader medium float precision rangeMax:127",
"webgl vertex shader low float precision:23",
"webgl vertex shader low float precision rangeMin:127",
"webgl vertex shader low float precision rangeMax:127",
"webgl fragment shader high float precision:23",
"webgl fragment shader high float precision rangeMin:127",
"webgl fragment shader high float precision rangeMax:127",
"webgl fragment shader medium float precision:23",
"webgl fragment shader medium float precision rangeMin:127",
"webgl fragment shader medium float precision rangeMax:127",
"webgl fragment shader low float precision:23",
"webgl fragment shader low float precision rangeMin:127",
"webgl fragment shader low float precision rangeMax:127",
"webgl vertex shader high int precision:0",
"webgl vertex shader high int precision rangeMin:31",
"webgl vertex shader high int precision rangeMax:30",
"webgl vertex shader medium int precision:0",
"webgl vertex shader medium int precision rangeMin:31",
"webgl vertex shader medium int precision rangeMax:30",
"webgl vertex shader low int precision:0",
"webgl vertex shader low int precision rangeMin:31",
"webgl vertex shader low int precision rangeMax:30",
"webgl fragment shader high int precision:0",
"webgl fragment shader high int precision rangeMin:31",
"webgl fragment shader high int precision rangeMax:30",
"webgl fragment shader medium int precision:0",
"webgl fragment shader medium int precision rangeMin:31",
"webgl fragment shader medium int precision rangeMax:30",
"webgl fragment shader low int precision:0",
"webgl fragment shader low int precision rangeMin:31",
"webgl fragment shader low int precision rangeMax:30"
], // webgl_params, cab be set to [] if webgl is not supported
"6bc5":"Google Inc. (NVIDIA) #X3fQVPgERx~ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 Laptop GPU (0x00002560) Direct3D11 vs_5_0 ps_5_0, D3D11) #X3fQVPgERx", // webglVendorAndRenderer
"ed31":0, // hasLiedLanguages
"72bd":0, // hasLiedOs
"097b":0, // hasLiedBrowser
"52cd":[
0, // void 0 !== navigator.maxTouchPoints ? t = navigator.maxTouchPoints : void 0 !== navigator.msMaxTouchPoints && (t = navigator.msMaxTouchPoints);
0, // document.createEvent("TouchEvent"), if succeed 1 else 0
0 // "ontouchstart" in window ? 1 : 0
], // touch support
"a658":[
"Arial",
"Arial Black",
"Arial Narrow",
"Book Antiqua",
"Bookman Old Style",
"Calibri",
"Cambria",
"Cambria Math",
"Century",
"Century Gothic",
"Century Schoolbook",
"Comic Sans MS",
"Consolas",
"Courier",
"Courier New",
"Georgia",
"Helvetica",
"Helvetica Neue",
"Impact",
"Lucida Bright",
"Lucida Calligraphy",
"Lucida Console",
"Lucida Fax",
"Lucida Handwriting",
"Lucida Sans",
"Lucida Sans Typewriter",
"Lucida Sans Unicode",
"Microsoft Sans Serif",
"Monotype Corsiva",
"MS Gothic",
"MS PGothic",
"MS Reference Sans Serif",
"MS Sans Serif",
"MS Serif",
"Palatino Linotype",
"Segoe Print",
"Segoe Script",
"Segoe UI",
"Segoe UI Light",
"Segoe UI Semibold",
"Segoe UI Symbol",
"Tahoma",
"Times",
"Times New Roman",
"Trebuchet MS",
"Verdana",
"Wingdings",
"Wingdings 2",
"Wingdings 3"
], // font details. see https://github.com/fingerprintjs/fingerprintjs for implementation details
"d02f":"124.04347527516074" // audio fingerprint. see https://github.com/fingerprintjs/fingerprintjs for implementation details
},
"54ef":"{\"b_ut\":\"7\",\"home_version\":\"V8\",\"i-wanna-go-back\":\"-1\",\"in_new_ab\":true,\"ab_version\":{\"for_ai_home_version\":\"V8\",\"tianma_banner_inline\":\"CONTROL\",\"enable_web_push\":\"DISABLE\"},\"ab_split_num\":{\"for_ai_home_version\":54,\"tianma_banner_inline\":54,\"enable_web_push\":10}}", // abtest info, embedded in html
"8b94":"", // refer_url, document.referrer ? encodeURIComponent(document.referrer).substr(0, 1e3) : ""
"df35":"312C2F31-1D48-E108C-4232-D7E96B104A8D1070864infoc", // _uuid, set from cookie, generated by client side(algorithm remains unknown)
"07a4":"zh-CN", // language
"5f45":null, // laboratory, set from cookie, null if empty, source remains unknown
"db46":0 // is_selfdef, default 0
}
简单修改了一下爬虫,按照上面的讨论内容,构建一下机制:
- 调用 https://api.bilibili.com/x/frontend/finger/spi 获得 cookie 值 bvuid3
- 使用 ExClimbWuzhi 激活 cookie 值 bvuid3
- 后续调用 动态接口时,带上这个 cookie 值 bvuid3 和 referer(设置主站地址即可) 然后就正常了,测试爬取的是 极客湾 的所有动态数据,没有遇到接口报错的情况
想问一下,使用ExClimbWuzhi这个接口时的payload和需要激活的bvuid3有关系吗?还是直接用浏览器爬下来的就行?
简单修改了一下爬虫,按照上面的讨论内容,构建一下机制:
- 调用 https://api.bilibili.com/x/frontend/finger/spi 获得 cookie 值 bvuid3
- 使用 ExClimbWuzhi 激活 cookie 值 bvuid3
- 后续调用 动态接口时,带上这个 cookie 值 bvuid3 和 referer(设置主站地址即可) 然后就正常了,测试爬取的是 极客湾 的所有动态数据,没有遇到接口报错的情况
想问一下,使用ExClimbWuzhi这个接口时的payload和需要激活的bvuid3有关系吗?还是直接用浏览器爬下来的就行?
你是一点没看到我发的payload解析啊
/x/polymer/web-dynamic/v1/feed/space
接口用于获取指定用户的动态列表 以前该接口使用正常,自2023.5.24下午5点20左右开始,在不带cookie下访问该接口会返回错误-352,如下: