search

PBH-BTN / PeerBanHelper

BT 反吸血工具 - 自动封禁不受欢迎、吸血和异常的 BT 客户端，并支持自定义规则。PeerId黑名单/UserAgent黑名单/IP CIDR/假进度/超量下载/进度回退/多播追猎/连锁封禁/伪装检测 支持 qBittorrent/Transmission/Deluge/BiglyBT/Vuze(Azureus)/BitComet
GNU General Public License v3.0
2.4k stars 79 forks source link

[Feature] 支持 qBittorrent 4.x 下排除私有 torrent #472

Closed ccloli closed 1 month ago

ccloli commented 1 month ago

此问题与 #399 相似，但后者仅提到 qBittorrent 5.x，且目前 WebUI 也提示仅支持 qBittorrent 5.x。

对于 qBittorrent 4.x（例如 v4.5.5）可以使用 /api/v2/torrents/properties 获取到 torrent 的详细信息，其中包含 is_private 字段可用于判断是否是私有 torrent。

image

Update: 原 issue 提到了 #22 的问题，是否考虑在服务端添加 LRU 缓存，记录 torrent hash 与是否私有的映射，来减少活跃种子的检测请求？

Ghost-chu commented 1 month ago

出于 #22 的性能考虑，明确拒绝 5.x 以下版本添加私有种子支持。

ccloli commented 1 month ago

不好意思，我在看原 issue 时才看到 #22 相关的问题。不知是否可以使用缓存来记录映射关系？我理解活跃 torrent 仅是所有 torrent 的一小部分，且仅需发起一次检查即可，应该不会发起过多请求。此外，我个人认为不太会出现启动后瞬间会有 100 个活跃 torrent 的情况，理论上活跃 torrent 是逐步增加的。

可以考虑使用串行请求 + 间隔延迟的方式减少请求负载，而不是一次性并发获取所有的结果，相关结果也可以存储在本地缓存，服务端重启也可复用之前的缓存。

Ghost-chu commented 1 month ago

不好意思，我在看原 issue 时才看到 #22 相关的问题。不知是否可以使用缓存来记录映射关系？我理解活跃 torrent 仅是所有 torrent 的一小部分，且仅需发起一次检查即可，应该不会发起过多请求。此外，我个人认为不太会出现启动后瞬间会有 100 个活跃 torrent 的情况，理论上活跃 torrent 是逐步增加的。

可以考虑使用串行请求 + 间隔延迟的方式减少请求负载，而不是一次性并发获取所有的结果，相关结果也可以存储在本地缓存，服务端重启也可复用之前的缓存。

即使是缓存也需要定期从下载器更新数据（还需要考虑更新失败的情况），极大的提升了复杂性，暂时没有添加的计划。

PR is welcome

ccloli commented 1 month ago

Hi，不好意思再打扰一下。

我按照上面的逻辑试着编写了一部分代码，但是由于我本地并没有完整的 Java 开发环境，且没有看到相关的开发文档，所以不太清楚该如何配置开发环境。我看到项目内有一个 Dockerfile，所以想直接使用 Docker 构建镜像后再 docker cp 出产物。

但是在安装依赖的容器内，mvn 提示了如下错误：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.8.0:go-offline (default-cli) on project peerbanhelper ``` 1067.3 [WARNING] Could not transfer metadata com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml from/to reposilite-repository-releases (https://maven.sergeybochkov.com/releases): maven.sergeybochkov.com: Try again 1067.3 [WARNING] Could not transfer metadata com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml from/to codemc-releases (https://repo.codemc.io/repository/maven-releases/): repo.codemc.io 1067.3 [WARNING] Could not transfer metadata com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml from/to codemc-snapshots (https://repo.codemc.io/repository/maven-snapshots/): repo.codemc.io 1067.3 [WARNING] Could not transfer metadata com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml from/to maven-snapshots (https://s01.oss.sonatype.org/content/repositories/snapshots/): s01.oss.sonatype.org 1067.3 [WARNING] com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml failed to transfer from https://maven.sergeybochkov.com/releases during a previous attempt. This failure was cached in the local repository and resolution will not be reattempted until the update interval of reposilite-repository-releases has elapsed or updates are forced. Original error: Could not transfer metadata com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml from/to reposilite-repository-releases (https://maven.sergeybochkov.com/releases): maven.sergeybochkov.com: Try again 1067.3 [WARNING] com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml failed to transfer from https://repo.codemc.io/repository/maven-releases/ during a previous attempt. This failure was cached in the local repository and resolution will not be reattempted until the update interval of codemc-releases has elapsed or updates are forced. Original error: Could not transfer metadata com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml from/to codemc-releases (https://repo.codemc.io/repository/maven-releases/): repo.codemc.io 1067.3 [WARNING] com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml failed to transfer from https://repo.codemc.io/repository/maven-snapshots/ during a previous attempt. This failure was cached in the local repository and resolution will not be reattempted until the update interval of codemc-snapshots has elapsed or updates are forced. Original error: Could not transfer metadata com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml from/to codemc-snapshots (https://repo.codemc.io/repository/maven-snapshots/): repo.codemc.io 1067.3 [WARNING] com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml failed to transfer from https://s01.oss.sonatype.org/content/repositories/snapshots/ during a previous attempt. This failure was cached in the local repository and resolution will not be reattempted until the update interval of maven-snapshots has elapsed or updates are forced. Original error: Could not transfer metadata com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:2.0.2-SNAPSHOT/maven-metadata.xml from/to maven-snapshots (https://s01.oss.sonatype.org/content/repositories/snapshots/): s01.oss.sonatype.org 1067.3 [INFO] Downloading from reposilite-repository-releases: https://maven.sergeybochkov.com/releases/com/ghostchu/lib/unofficial/com/alessiodp/libby/libby-standalone/2.0.2-SNAPSHOT/libby-standalone-2.0.2-SNAPSHOT.pom 1067.3 [INFO] Downloading from Jitpack: https://jitpack.io/com/ghostchu/lib/unofficial/com/alessiodp/libby/libby-standalone/2.0.2-SNAPSHOT/libby-standalone-2.0.2-SNAPSHOT.pom 1067.5 [INFO] Downloading from codemc-releases: https://repo.codemc.io/repository/maven-releases/com/ghostchu/lib/unofficial/com/alessiodp/libby/libby-standalone/2.0.2-SNAPSHOT/libby-standalone-2.0.2-SNAPSHOT.pom 1072.5 [INFO] Downloading from codemc-snapshots: https://repo.codemc.io/repository/maven-snapshots/com/ghostchu/lib/unofficial/com/alessiodp/libby/libby-standalone/2.0.2-SNAPSHOT/libby-standalone-2.0.2-SNAPSHOT.pom 1072.5 [INFO] Downloading from maven-snapshots: https://s01.oss.sonatype.org/content/repositories/snapshots/com/ghostchu/lib/unofficial/com/alessiodp/libby/libby-standalone/2.0.2-SNAPSHOT/libby-standalone-2.0.2-SNAPSHOT.pom 1077.6 [INFO] ------------------------------------------------------------------------ 1077.6 [INFO] BUILD FAILURE 1077.6 [INFO] ------------------------------------------------------------------------ 1077.6 [INFO] Total time: 17:55 min (Wall Clock) 1077.6 [INFO] Finished at: 2024-09-14T17:39:05Z 1077.6 [INFO] ------------------------------------------------------------------------ 1077.6 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.8.0:go-offline (default-cli) on project peerbanhelper: org.eclipse.aether.resolution.DependencyResolutionException: Failed to read artifact descriptor for com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:jar:2.0.2-SNAPSHOT: The following artifacts could not be resolved: com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:pom:2.0.2-SNAPSHOT (absent): Could not transfer artifact com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:pom:2.0.2-SNAPSHOT from/to reposilite-repository-releases (https://maven.sergeybochkov.com/releases): maven.sergeybochkov.com -> [Help 1] 1077.6 [ERROR] 1077.6 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 1077.6 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 1077.6 [ERROR] 1077.6 [ERROR] For more information about the errors and possible solutions, please read the following articles: 1077.6 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException ------ 1 warning found (use --debug to expand): - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 10) Dockerfile:14 -------------------- 12 | WORKDIR /build 13 | # fetch all dependencies 14 | >>> RUN mvn dependency:go-offline -B -T 1.5C -Daether.dependencyCollector.impl=bf -Dmaven.artifact.threads=32 15 | 16 | # 构建后端 -------------------- ERROR: failed to solve: process "/bin/sh -c mvn dependency:go-offline -B -T 1.5C -Daether.dependencyCollector.impl=bf -Dmaven.artifact.threads=32" did not complete successfully: exit code: 1 ```

我试着修改 Dockerfile，将指令换成 mvn install 也是相似的错误，只是上面的 WARNING 变成了 ERROR：

[ERROR] Failed to execute goal on project peerbanhelper: Could not collect dependencies for project com.ghostchu.peerbanhelper:peerbanhelper:takari-jar:6.2.1 ``` 1891.7 [INFO] ------------------------------------------------------------------------ 1891.7 [INFO] BUILD FAILURE 1891.7 [INFO] ------------------------------------------------------------------------ 1891.7 [INFO] Total time: 31:30 min 1891.7 [INFO] Finished at: 2024-09-14T18:11:03Z 1891.7 [INFO] ------------------------------------------------------------------------ 1891.7 [ERROR] Failed to execute goal on project peerbanhelper: Could not collect dependencies for project com.ghostchu.peerbanhelper:peerbanhelper:takari-jar:6.2.1 1891.7 [ERROR] Failed to read artifact descriptor for com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-core:jar:2.0.2-SNAPSHOT 1891.7 [ERROR] Caused by: The following artifacts could not be resolved: com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-core:pom:2.0.2-SNAPSHOT (absent): Could not transfer artifact com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-core:pom:2.0.2-SNAPSHOT from/to reposilite-repository-releases (https://maven.sergeybochkov.com/releases): maven.sergeybochkov.com 1891.7 [ERROR] Failed to read artifact descriptor for com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:jar:2.0.2-SNAPSHOT 1891.7 [ERROR] Caused by: The following artifacts could not be resolved: com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:pom:2.0.2-SNAPSHOT (absent): Could not transfer artifact com.ghostchu.lib.unofficial.com.alessiodp.libby:libby-standalone:pom:2.0.2-SNAPSHOT from/to reposilite-repository-releases (https://maven.sergeybochkov.com/releases): maven.sergeybochkov.com 1891.7 [ERROR] 1891.7 [ERROR] -> [Help 1] 1891.7 [ERROR] 1891.7 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 1891.7 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 1891.7 [ERROR] 1891.7 [ERROR] For more information about the errors and possible solutions, please read the following articles: 1891.7 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException ------ 1 warning found (use --debug to expand): - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 10) Dockerfile:14 -------------------- 12 | WORKDIR /build 13 | # fetch all dependencies 14 | >>> RUN mvn install # dependency:go-offline -B -T 1.5C -Daether.dependencyCollector.impl=bf -Dmaven.artifact.threads=32 15 | 16 | # 构建后端 -------------------- ERROR: failed to solve: process "/bin/sh -c mvn install # dependency:go-offline -B -T 1.5C -Daether.dependencyCollector.impl=bf -Dmaven.artifact.threads=32" did not complete successfully: exit code: 1 ```

此问题应该与网络无关，因为其他的依赖均可正常拉取。已经尝试过 docker build . 多次，均是相同问题。

我理解 Docker 镜像构建的环境应该是比较稳定、不会有太大变数的，可以避免本地与构建环境相关的问题，所以使用相同的 Dockerfile 构建应该不会有问题。不知是否是我的配置有问题，或者构建环境不对？还望不吝赐教，若有打扰还望谅解，感谢。

Updated: GitHub Actions 工作正常，先暂时使用 GitHub Actions 构建测试，目前还在验证功能中。

ccloli commented 1 month ago

测试了下，在修改获取所有 torrent 列表，最终列表为 ~800 个 torrent 的情况下，并发 5 个线程调用请求，逐一获取 is_private 的部分耗时大约 3s。在 Ryzen 5600 4x vCPU+16G RAM 的情况下，qBittorrent CPU 闲时占用 ~2%，在 PBH 定时拉取列表时占用 ~5%，再加上私有 torrent 的并发请求后首次启动占用 ~8%，感觉性能尚可。后续执行均命中缓存，获取 is_private 的耗时基本在 1ms 左右。

image

试着调整为并发 10 / 15 个线程，耗时与资源占用并没有明显变化。怀疑是输出抢占资源，故去掉了调试输出。

去掉后，5 线程耗时 ~1.1s，峰值占用 ~15%；10 线程耗时 ~1.2s，峰值占用 ~15%；15 线程耗时大约 1s，峰值占用 ~17%。感觉再堆线程边际效应会递减，因此 5 线程应该已经足够。且上述测试发现吞吐量已经达到瓶颈，增加线程数并没有意义。

image

此外，吞吐率也有可能是 PBH 自身相关，因为 PBH 启动时会占用大约 80% 左右的 CPU，留给 qBittorrent 的资源相对有限。一般来说请求数较多的情况也仅出现在 PBH 启动的时候，因此感觉应该可以忽略不计。

上述 torrent 中有大约 350 条是私有 torrent，本 issue 相关功能实现后，过滤得到实际 torrent 数为 450 条，需要检查的 torrent 数降低了，后续定时检查的耗时也从 ~670ms 降低到了 ~450ms，其中 /info 这一 API 在浏览器开发者工具里显示本身大约会耗时 ~120ms，所以排除掉获取列表的时间，其他操作时间从 ~550ms 降低到 ~330ms，减少了大约 2/5 的时间。因此对于私有 torrent 比例较高的客户端来说，整体性能其实是更好的。

image

感兴趣的话可以获取上面测试的 jar 产物试验下，本地 qBittorrent 的 torrent 数量不是很多，所以可能不能直接暴露性能问题。不过一般来说客户端都会有最大连接数限制，如果没有主动修改过的话，默认的全局最大连接数也就是 500，因此极端情况下最多对应 500 个活跃 torrent。

产物 jar 文件 构建产物: https://github.com/ccloli/PeerBanHelper/actions/runs/10869505871 （在现在的 master 分支上补充 issue 相关功能） 测试产物: https://github.com/ccloli/PeerBanHelper/actions/runs/10865805597 （torrent 列表取全量数据，而不仅仅是当前活跃 torrent，用于压测在大量活跃 torrent 场景下的性能问题） ****** Updated: 调整了构建产物的 Actions 地址，之前误在 master 分支上执行了（大概这就是通宵的代价）