search

HenryQW / Awesome-TTRSS

🐋 Awesome TTRSS, a powerful Dockerised all-in-one RSS solution.
http://ttrss.henry.wang
MIT License
2.44k stars 494 forks source link

Mercury 不能通过代理？[BUG] #149

Closed MisteryMonster closed 4 years ago

MisteryMonster commented 4 years ago

Bug 描述 尝试获取 hk01、 BBC 文章的时候后台 log 显示 200 37 - 10003.812 ms，并且提示Unable to extract fulltext for this article。 这应该是超时了，但是文章是可以正常通过 TTRSS 获取的。

获取DW中文网的时候是200 70 - 3.415 ms，这个就不清楚是怎么回事了，也许是不支持？

使用国内的源可以正常执行动作，不会提示Unable to extract fulltext for this article。

你谷歌/百度了吗？

部署方法 docker-compose [latest]

部署环境

复现步骤 复现该 bug 的详细步骤

  1. 打开不用代理无法访问的文章
  2. 发现错误提示Unable to extract fulltext for this article。

预期结果 获取全文

截屏 image

错误日志 [13-May-2020 10:16:27] NOTICE: fpm is running, pid 21 [13-May-2020 10:16:27] NOTICE: ready to handle connections s6-svscan: warning: unable to exec finish script .s6-svscan/finish: No such file or directory s6-svscan: warning: executing into .s6-svscan/crash s6-svscan: fatal: unable to exec .s6-svscan/crash: No such file or directory [13-May-2020 10:18:48] NOTICE: fpm is running, pid 19 [13-May-2020 10:18:48] NOTICE: ready to handle connections s6-svscan: warning: unable to exec finish script .s6-svscan/finish: No such file or directory s6-svscan: warning: executing into .s6-svscan/crash s6-svscan: fatal: unable to exec .s6-svscan/crash: No such file or directory [13-May-2020 10:24:19] NOTICE: fpm is running, pid 20 [13-May-2020 10:24:19] NOTICE: ready to handle connections s6-svscan: warning: unable to exec finish script .s6-svscan/finish: No such file or directory s6-svscan: warning: executing into .s6-svscan/crash s6-svscan: fatal: unable to exec .s6-svscan/crash: No such file or directory [13-May-2020 10:27:12] NOTICE: fpm is running, pid 21 [13-May-2020 10:27:12] NOTICE: ready to handle connections s6-svscan: warning: unable to exec finish script .s6-svscan/finish: No such file or directory s6-svscan: warning: executing into .s6-svscan/crash s6-svscan: fatal: unable to exec .s6-svscan/crash: No such file or directory PDOException: SQLSTATE[08006] [7] FATAL: the database system is starting up in /configure-db.php:101 Stack trace:

0 /configure-db.php(101): PDO->__construct('pgsql:dbname=po...', 'postgres', 'ttrss')

1 /configure-db.php(109): dbconnect(Array)

2 /configure-db.php(31): dbcheckconn(Array)

3 {main}[13-May-2020 10:36:32] NOTICE: fpm is running, pid 20

[13-May-2020 10:36:32] NOTICE: ready to handle connections

其他 TTRSS因为设置了代理所以可以获取文章，但是全文获取就失败。

HenryQW commented 4 years ago

mercury的确没有支持代理，有空给它添加代理支持。

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity in 14 days. It will be closed if no further activity occurs in 7 days. Thank you for your contributions.

HenryQW commented 4 years ago

https://github.com/postlight/mercury-parser/issues/424

底层依赖目前不支持代理，所以 API 目前不能支持代理。

HenryQW commented 4 years ago

@MisteryMonster mercury 用的是 postman-request，可以尝试一下在 mercury API docker 设置环境变量看看是否有效：

https://github.com/postmanlabs/postman-request#controlling-proxy-behaviour-using-environment-variables

MisteryMonster commented 4 years ago

我记得之前我在环境变量里添加过，ttrss 能但是 mercury 好像不行（会无法连接）。不过已经是一个月前了所以记忆模糊哈哈，我考完试再试试！

MisteryMonster commented 4 years ago 
service.mercury: # set Mercury Parser API endpoint to `service.mercury:3000` on TTRSS plugin setting page
    image: wangqiru/mercury-parser-api:latest
    container_name: mercury
    environment: 
      - HTTP_PROXY=192.168.144.1:7890
      - HTTPS_PROXY=192.168.144.1:7890
    expose:
      - 3000
    restart: always

我这样设置后就没办法用 mercury 了（Unable to extract fulltext for this article），docker logs mercury 同样也有请求：

添加环境变量前，可以获取文章：

GET /parser?url=http%3A%2F%2Fepaper.bjnews.com.cn%2Fhtml%2F2020-06%2F17%2Fnode_26.htm 200 1722 - 402.885 ms

这是添加环境变量后的，显示Unable...：

GET /parser?url=http%3A%2F%2Fepaper.bjnews.com.cn%2Fhtml%2F2020-06%2F17%2Fnode_26.htm 200 60 - 0.736 ms
HenryQW commented 4 years ago

192.168.144.1 是宿主 IP，在容器内是无法解析到的。

MisteryMonster commented 4 years ago 
  service.rss:
    image: wangqiru/ttrss:latest
    container_name: ttrss
    ports:
      - 181:80
    environment:
      - SELF_URL_PATH=http://192.168.1.234:181/ # please change to your own domain
      - DB_HOST=database.postgres
      - DB_PORT=5432
      - DB_NAME=ttrss
      - DB_USER=postgres
      - DB_PASS=ttrss # please change the password
      - ENABLE_PLUGINS=auth_internal,fever # auth_internal is required. Plugins enabled here will be enabled for all users as system plugins
      - FEED_LOG_QUIET=true
      - HTTP_PROXY=192.168.144.1:7890
      - SINGLE_USER_MODE=true
    stdin_open: true
    tty: true
    restart: always
    command: sh -c 'sh /wait-for.sh $$DB_HOST:$$DB_PORT -- php /configure-db.php && exec s6-svscan /etc/s6/'

但是在 service.rss 的环境变量指定的是工作的（可以通过代理获取到源），是不是有些奇怪……

HenryQW commented 4 years ago

如果没有对 docker network 进行设置，容器内是无法访问使用局域网 IP 访问宿主机器上的任何服务的。 可以试试 docker exec -it ttrss ping 192.168.144.1 看看能不能 ping 通。

MisteryMonster commented 4 years ago

可以 ping 通。原本我以为和我设置的环境变量有关，我曾经在/etc/systemd/system/docker.service.d/http-proxy.conf里面添加过环境变量HTTP_PROXY，但是回过头检查的时候我已经把它删掉了（用systemctl show --property=Environment docker看是空的了）。

HenryQW commented 4 years ago

那就不清楚是什么原因了。。我对这块也不是很了解

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity in 14 days. It will be closed if no further activity occurs in 7 days. Thank you for your contributions.

kaaass commented 3 years ago

实测是可以的，主要 @MisteryMonster 的配置漏了协议，比如这样的配置就可以：

  service.mercury: # set Mercury Parser API endpoint to `service.mercury:3000` on TTRSS plugin setting page
    image: wangqiru/mercury-parser-api:latest
    container_name: mercury
    ports:
      - 9000:3000
    environment:
      - HTTP_PROXY=http://10.1.1.3:8888
      - HTTPS_PROXY=http://10.1.1.3:8888
    networks:
      - public_access
      - service_only
    restart: always

另外，DW 的抓取失败还有 mercury_fulltext 插件本身 URL 编码的问题，需要先对 url encode 一回（总共两次）。有空我发个 pr。