DIYgod / RSSHub

🧡 Everything is RSSible
https://docs.rsshub.app
MIT License
32.5k stars 7.24k forks source link

微博图片403 Forbidden #11096

Open KawausoJyou opened 1 year ago

KawausoJyou commented 1 year ago

路由地址

/weibo/user/:uid/:routeParams?

完整路由地址

/weibo/user/6048193311/readable=1&showEmojiForRetweet=1&addLinkForPics=1&showTimestampInDescription=1&displayVideo=0&showEmojiInDescription=0&showLinkIconInDescription=0

相关文档

https://docs.rsshub.app/social-media.html#wei-bo-bo-zhu

预期是什么?

正常显示图片

实际发生了什么?

图片无法正常显示,且图片链接显示403 Forbidden Snipaste_2022-10-15_10-22-29 Snipaste_2022-10-15_10-22-38

部署

自建

部署相关信息

OS: Linux, Node: v12.22.12, Docker: v20.10.18

额外信息

<item>
<title>
<![CDATA[ 本周的宝可梦四格漫画来啰!这次的主角是卡蒂狗!卡蒂狗是不是很可爱呢!你觉得它努力忍耐了多久才输给了食欲?#宝可梦# [图片] ]]>
</title>
<description>
<![CDATA[ 本周的宝可梦四格漫画来啰!<br>这次的主角是卡蒂狗!<br><br>卡蒂狗是不是很可爱呢!<br>你觉得它努力忍耐了多久才输给了食欲?<br><a href="https://m.weibo.cn/search?containerid=231522type%3D1%26t%3D10%26q%3D%23%E5%AE%9D%E5%8F%AF%E6%A2%A6%23&amp;isnewpage=1" data-hide=""><span class="surl-text">#宝可梦#</span></a> <br clear="both"><div style="clear: both"></div><a href="https://wx3.sinaimg.cn/large/006BjBn1ly1h75p83ei0sj30u02jgn5g.jpg"><img vspace="8" hspace="4" style="" src="https://wx3.sinaimg.cn/large/006BjBn1ly1h75p83ei0sj30u02jgn5g.jpg" referrerpolicy="no-referrer"></a> ]]>
</description>
<pubDate>Sat, 15 Oct 2022 01:07:34 GMT</pubDate>
<guid isPermaLink="false">https://weibo.com/6048193311/Mao8CfRUO</guid>
<link>https://weibo.com/6048193311/Mao8CfRUO</link>
<author>
<![CDATA[ 知任CHS ]]>
</author>
</item>

这不是重复的 issue

github-actions[bot] commented 1 year ago
Searching for maintainers:

To maintainers: if you are not willing to be disturbed, list your username in scripts/workflow/test-issue/call-maintainer.js. In this way, your username will be wrapped in an inline code block when tagged so you will not be notified.

如果有任何路由无法匹配,issue 将会被自动关闭。如果 issue 和路由无关,请使用 NOROUTE 关键词,或者留下评论。我们会重新审核。 If there is any route not found, the issue will be closed automatically. Please use NOROUTE for a route-irrelevant issue or leave a comment if it is a mistake.

TonyRL commented 1 year ago

Inoreader?

KawausoJyou commented 1 year ago

Inoreader?

FreshRSS

TonyRL commented 1 year ago
Screenshot ![image](https://user-images.githubusercontent.com/11386903/195984876-e62cd82f-0b8b-4c37-b03b-806cce617330.png)

It works fine on a new installation of FreshRSS

KawausoJyou commented 1 year ago

Screenshot image

It works fine on a new installation of FreshRSS

What is your version of FreshRSS and RSS Hub? My FreshRSS version is 1.20.0, and my RSSHub version is https://github.com/DIYgod/RSSHub/commit/ed08cefe307ee4863e18c695abf347cad4b5c56c. Did Weibo add my IP to the blacklist?

Rongronggg9 commented 1 year ago

For self-hosted instances, refer to #10874 for more available workarounds.

TonyRL commented 1 year ago

Screenshot

It works fine on a new installation of FreshRSS

What is your version of FreshRSS and RSS Hub? My FreshRSS version is 1.20.0, and my RSSHub version is ed08cef. Did Weibo add my IP to the blacklist?

FreshRSS 1.20.0. To be specific, it's https://hub.docker.com/r/linuxserver/freshrss RSS feed is fetched from the demo instance

KawausoJyou commented 1 year ago

For self-hosted instances, refer to #10874 for more available workarounds.

Sorry, I can't fully understand how to use "image_hotlink_template". Could you please explain it in detail?

Rongronggg9 commented 1 year ago

For self-hosted instances, refer to #10874 for more available workarounds.

Sorry, I can't fully understand how to use "image_hotlink_template". Could you please explain it in detail?

Have you read the beginning of https://docs.rsshub.app/en/parameter.html?

KawausoJyou commented 1 year ago

For self-hosted instances, refer to #10874 for more available workarounds.

Sorry, I can't fully understand how to use "image_hotlink_template". Could you please explain it in detail?

Have you read the beginning of https://docs.rsshub.app/en/parameter.html?

I have already read the link you sent. I found that when the referer is empty, the picture cannot be displayed. When the referer is "weibo. com", the picture can be displayed normally. However, I don't know how to write "image_hotlink_template" to add a referer.

Snipaste_2022-10-15_21-29-00 Snipaste_2022-10-15_21-28-53
Rongronggg9 commented 1 year ago

Ohh, I see. sinaimg.cn has been upgraded, it bans requests from browsers without referer set to https://weibo.com. However, the limit is only applied when the request is from Mainland China. In other regions, zero referer is permitted, that's why @TonyRL can load the pic.

Rongronggg9 commented 1 year ago

If you deploy https://github.com/Rongronggg9/rsstt-img-relay on Cloudflare, a referer is unnecessary as Cloudflare has no edges in Mainland China.

KawausoJyou commented 1 year ago

If you deploy https://github.com/Rongronggg9/rsstt-img-relay on Cloudflare, a referer is unnecessary as Cloudflare has no edges in Mainland China.

Thank you very much for your patience. I have successfully deployed rsst-img-relay on Cloudflare and configured the "HOTLINK_TEMPLATE" variable. Now the image can be displayed normally.

lingnand commented 1 year ago

I'm observing the same issue, and my IP is not within mainland China. Has weibo gone stricter on this?

leoneoo commented 1 year ago

https://zmingcx.com/sina-micro-album-address.html 参考这篇文章,应该是新浪为了防止图片外链被盗用,更改了外链地址吧!

KawausoJyou commented 1 year ago

https://zmingcx.com/sina-micro-album-address.html 参考这篇文章,应该是新浪为了防止图片外链被盗用,更改了外链地址吧!

感谢回复,请问如何将文章中的修改应用到RSSHub呢🥲

lingnand commented 1 year ago

https://zmingcx.com/sina-micro-album-address.html 参考这篇文章,应该是新浪为了防止图片外链被盗用,更改了外链地址吧!

能用 👍🏻 @Rongronggg9 能否把这个整合到 weibo 的 router 里?

或者如果 HOTLINK_TEMPLATE 可以支持 regex 替换的话,那对于这类情况还挺有用的——只需要 replace 一下 subdomain 就好了。

zydou commented 1 year ago

If you deploy https://github.com/Rongronggg9/rsstt-img-relay on Cloudflare, a referer is unnecessary as Cloudflare has no edges in Mainland China.

Just tested it, not worked any more. :(

KwToPA commented 1 year ago

又挂了

有些能显示,有些裂图

xmt328 commented 1 year ago

If you deploy https://github.com/Rongronggg9/rsstt-img-relay on Cloudflare, a referer is unnecessary as Cloudflare has no edges in Mainland China.

这个方法也失效了

Rongronggg9 commented 1 year ago

If you deploy https://github.com/Rongronggg9/rsstt-img-relay on Cloudflare, a referer is unnecessary as Cloudflare has no edges in Mainland China.

这个方法也失效了

https://github.com/DIYgod/RSSHub/issues/10874#issuecomment-1369647937

i-iooi-i commented 1 year ago

What are the new ways to solve this problem?

zizhengwu commented 1 year ago

我不太懂反盗链/反反盗链,请问如果我用 chrome 网页访问 inoreader,用 no-referrer 的 userscript 有用吗?例子:

https://greasyfork.org/en/scripts/376884-%E6%98%BE%E7%A4%BA%E9%98%B2%E7%9B%97%E9%93%BE%E5%9B%BE%E7%89%87-for-inoreader/code

KwToPA commented 1 year ago

现在全部图片都裂了 右键Access Denied Reference # 一串数字

zizhengwu commented 1 year ago

是不是可能应该 reopen 一下这个 issue,因为问题似乎依然存在,而这个 issue 是 Closed 的状态。微博可能是用的人比较多的路由,所以解决这个问题会帮助很多人。

ttxhxz commented 1 year ago

是不是可能应该 reopen 一下这个 issue,因为问题似乎依然存在,而这个 issue 是 Closed 的状态。微博可能是用的人比较多的路由,所以解决这个问题会帮助很多人。

的确,是否能支持一下解决这个问题。

fightroad commented 1 year ago

我测试使用rsshub默认参数启动 netnewswire的mac客户端可以打开微博图片。但是ios端不行。不知道原因在哪。

linzzzzllll commented 1 year ago

现在怎么解决吗微博图片问题

zizhengwu commented 1 year ago

目前我用了一个 workaround,适用于电脑 chrome 浏览器浏览 inoreader。可以用插件,给图片加 referer。详情见下面帖子的 4 楼

https://www.v2ex.com/t/904989#;

移动端或者其他客户端有没有解决方式我也想知道。

KwToPA commented 1 year ago

依然无法工作,有时候右键新图标打开都不行,图片抬头wx1改成wx2就报错403

只能用这个插件,large改成original再调转一次可以看原图

https://greasyfork.org/en/scripts/2312-resize-image-on-open-image-in-new-tab

eaglu commented 11 months ago

在iOS/iPadOS上的一个方法是使用支持mitm和rewrite的代理软件先对singimg.cn进行解密然后再添加header,在loon中可以这么写: 在mitm中添加域名.sinaimg.cn,然后在复写中使用如下正则表达式https://..sinaimg.cn/.*,接着添加referer即可

junfengP commented 7 months ago

我提供一个代码段,结合image_hotlink_template参数可以解决referer问题。需要依赖自建RSSHub和一段python程序。

原理是通过替换image_hotlink_template替换中链接地址,由自定义python程序发起图片URL请求并返回给RSS阅读器。

假设下面python程序对外提供服务url为 http://my-image-proxy.com:8080/image

  1. 打开HOTLINK功能,RSSHub环境变量ALLOW_USER_HOTLINK_TEMPLATE: 'true'
  2. 订阅的路由修改/weibo/user/:uid/:routeParams? -> /weibo/user/:uid/:routeParams?image_hotlink_template=http://my-image-proxy.com:8080/image?url=${href_ue}%26referer=https://weibo.com

其中需要注意image_hotlink_template中对于&需要使用URL编码的%26,否则后续的参数无法被rsshub解析到image_hotlink_template

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import requests
import os
import urllib.parse
from cachetools import LRUCache, TTLCache
from typing import Any
import httpx

# 创建一个LRU缓存实例,例如:缓存100个最近使用的条目,并且每个条目在5分钟内有效
cache = TTLCache(maxsize=100, ttl=5 * 60)
app = FastAPI()

@app.get('/image')
async def proxy_request(request: Request):
    query_string = request.url.query
    # 解码原始查询字符串以正确处理转义的 &amp;
    query_params = urllib.parse.parse_qs(query_string)

    url_param = query_params.get('url', [''])[0]
    # 尝试从缓存中获取结果
    cached_response = cache.get(url_param)
    if cached_response is not None:
        headers, content = cached_response
        return StreamingResponse(iter([content]), status_code=200, headers=headers)

    referer_env = os.environ.get('DEFAULT_REFERER', '')
    referer_param = query_params.get('referer', [referer_env])[0]

    print(f"Proxying request to URL: {url_param}")
    print(f"Referer: {referer_param}")

    user_agent_env = os.environ.get('USER_AGENT_HEADER')
    if user_agent_env is None:
        user_agent_header = request.headers.get('user-agent', '')
    else:
        user_agent_header = user_agent_env

    proxy_uri = os.environ.get('PROXY_URI', None)

    async with httpx.AsyncClient(proxy=proxy_uri) as client:
        response = await client.get(url_param, headers={'referer': referer_param, 'user-agent': user_agent_header})
        # 确保完整读取response.content
        response_content = response.content
        content_length = len(response_content)

        headers = response.headers.copy()
        headers['Content-Length'] = str(content_length)  # 更新Content-Length为实际长度
        # 只缓存HTTP状态码为200的响应
        if response.status_code == 200:
            cache[url_param] = (headers, response_content)

        return StreamingResponse(
            iter([response_content]),
            status_code=response.status_code,
            headers=headers,
        )

if __name__ == '__main__':
    import uvicorn
    PORT = 8080
    uvicorn.run(app, host='0.0.0.0', port=PORT)

requirements.txt

requests
fastapi
uvicorn
cachetools
httpx
ghost commented 5 months ago

@junfengP 这段程序也可以应用在multimedia_hotlink_template里吗

Ch1llNoodle commented 2 months ago

services: rsshub: ... environment: ... HOTLINK_TEMPLATE: 'https://image.baidu.com/search/down?url=$${href_ue}' HOTLINK_INCLUDE_PATHS: /weibo depends_on: In the process of deploying with Docker Compose, you can convert Weibo external links like this.