search

opennaslab / kubespider

A global resource download orchestration system, build your home download center.
https://kubespider.netlify.app/
Apache License 2.0
2.06k stars 118 forks source link

使用订阅方式自动下载云盘中的指定目录时，会持续报错 Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying #541

Open xtay573269555 opened 3 months ago

xtay573269555 commented 3 months ago

使用订阅方式自动下载云盘中的指定目录时，会持续报错 Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying

kubespider版本：v0.7.0 部署方式：docker

docker 启动脚本：

[Systemd] ❯ cat run_kubespider.sh 
#!/bin/bash

mkdir -p kubespider

docker run -d --restart unless-stopped \
  --name kubespider \
  -p 0.0.0.0:3080:3080 \
  -e TZ=Asia/Shanghai \
  -v "$PWD"/kubespider:/app/.config \
  cesign/kubespider:v0.7.0

kubespider/source_provider.yaml 中的配置

alist_source_provider:
  enable: true
  type: alist_source_provider
  downloader:
    - aria2
  host: http://172.16.20.23:5244/
  watch_dirs:
    - /aliyun_disk_1/push_to_kubespider
    - /aliyun_disk_2/push_to_kubespider
    - /baiduyun_disk_1/push_to_kubespider

日志错误：


[Systemd] ❯ docker logs -f kubespider 
2024-07-31 09:47:12,989-INFO: File handler start running...

 _          _                     _     _
| | ___   _| |__   ___  ___ _ __ (_) __| | ___ _ __
| |/ / | | | '_ \ / _ \/ __| '_ \| |/ _` |/ _ \ '__|
|   <| |_| | |_) |  __/\__ \ |_) | | (_| |  __/ |
|_|\_\\__,_|_.__/ \___||___/ .__/|_|\__,_|\___|_|
                           |_|                    

KubeSpider - A global resource download orchestration system
Build Tag: ${GITHUB_SHA}
2024-07-31 09:47:13,021-INFO: Source Provider:bilibili_source_provider enabled...
2024-07-31 09:47:13,031-INFO: Source Provider:alist_source_provider enabled...
2024-07-31 09:47:13,038-INFO: Download Provider:aria2 enabled...
2024-07-31 09:47:13,047-INFO: Period Server Queue handler start running...
2024-07-31 09:47:13,047-INFO: PT Server start running...
2024-07-31 09:47:13,047-INFO: Notification Server Queue handler start running...
2024-07-31 09:47:13,048-INFO: Period Server producer start running...
2024-07-31 09:47:13,048-INFO: Download trigger job start running...
2024-07-31 09:47:13,048-WARNING: Failed to setup telegram bot token, telegram server not started
2024-07-31 09:47:13,048-INFO: Webhook Server start running...
2024-07-31 09:47:13,049-INFO: Serving on http://0.0.0.0:3080
2024-07-31 09:47:14,179-INFO: Find new resource:alist_source_provider/http://172.16.20.23:5244/d/aliyun_disk_1...
2024-07-31 09:47:14,179-INFO: filter downloader type:None, names:['aria2']
2024-07-31 09:47:14,179-INFO: download link type general, with provider size 1
2024-07-31 09:47:14,179-INFO: Download general file with downloader(aria2)
2024-07-31 09:47:14,181-INFO: Start general file download:http://172.16.20.23:5244/d/aliyun_disk_1/push_to_kubespider/test.doc?modified=2024-07-31T01:43:06.946Z&sign=
2024-07-31 09:47:14,191-INFO: Create download task result:[<aria2p.downloads.Download object at 0x7febf2dcdf90>]
2024-07-31 09:47:14,191-INFO: Find new resource:alist_source_provider/http://172.16.20.23:5244/d/aliyun_disk_1...
2024-07-31 09:47:14,191-INFO: filter downloader type:None, names:['aria2']
2024-07-31 09:47:14,191-INFO: download link type general, with provider size 1
2024-07-31 09:47:14,191-INFO: Download general file with downloader(aria2)
2024-07-31 09:47:14,193-INFO: Start general file download:http://172.16.20.23:5244/d/aliyun_disk_1/push_to_kubespider/test.jpg?modified=2024-07-31T01:45:36.415Z&sign=
2024-07-31 09:47:14,195-INFO: Create download task result:[<aria2p.downloads.Download object at 0x7febf28c6ad0>]
2024-07-31 09:47:14,195-INFO: Find new resource:alist_source_provider/http://172.16.20.23:5244/d/aliyun_disk_1...
2024-07-31 09:47:14,195-INFO: filter downloader type:None, names:['aria2']
2024-07-31 09:47:14,195-INFO: download link type general, with provider size 1
2024-07-31 09:47:14,195-INFO: Download general file with downloader(aria2)
2024-07-31 09:47:14,196-INFO: Start general file download:http://172.16.20.23:5244/d/aliyun_disk_2/push_to_kubespider/test.jpg?modified=2024-07-31T01:45:55.87Z&sign=
2024-07-31 09:47:14,198-INFO: Create download task result:[<aria2p.downloads.Download object at 0x7febf28bdf10>]
2024-07-31 09:47:14,198-INFO: Find new resource:alist_source_provider/http://172.16.20.23:5244/d/baiduyun_disk...
2024-07-31 09:47:14,198-INFO: filter downloader type:None, names:['aria2']
2024-07-31 09:47:14,198-INFO: download link type general, with provider size 1
2024-07-31 09:47:14,198-INFO: Download general file with downloader(aria2)
2024-07-31 09:47:14,200-INFO: Start general file download:http://172.16.20.23:5244/d/baiduyun_disk_1/push_to_kubespider/test.jpg?modified=2024-07-31T01:45:36Z&sign=
2024-07-31 09:47:14,202-INFO: Create download task result:[<aria2p.downloads.Download object at 0x7febf2dce750>]
2024-07-31 10:47:13,050-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-07-31 10:47:14,051-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-07-31 10:47:15,052-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
...
...
...
2024-08-12 12:47:21,123-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
usermod: no changes
2024-08-12 12:56:33,024-INFO: File handler start running...

 _          _                     _     _
| | ___   _| |__   ___  ___ _ __ (_) __| | ___ _ __
| |/ / | | | '_ \ / _ \/ __| '_ \| |/ _` |/ _ \ '__|
|   <| |_| | |_) |  __/\__ \ |_) | | (_| |  __/ |
|_|\_\\__,_|_.__/ \___||___/ .__/|_|\__,_|\___|_|
                           |_|                    

KubeSpider - A global resource download orchestration system
Build Tag: ${GITHUB_SHA}
2024-08-12 12:56:33,275-INFO: Source Provider:bilibili_source_provider enabled...
2024-08-12 12:56:33,339-INFO: Source Provider:alist_source_provider enabled...
2024-08-12 12:56:33,373-INFO: Download Provider:aria2 enabled...
2024-08-12 12:56:33,435-INFO: PT Server start running...
2024-08-12 12:56:33,435-INFO: Download trigger job start running...
2024-08-12 12:56:33,435-INFO: Notification Server Queue handler start running...
2024-08-12 12:56:33,435-WARNING: Failed to setup telegram bot token, telegram server not started
2024-08-12 12:56:33,436-INFO: Webhook Server start running...
2024-08-12 12:56:33,437-INFO: Period Server producer start running...
2024-08-12 12:56:33,437-INFO: Period Server Queue handler start running...
2024-08-12 12:56:33,437-INFO: Serving on http://0.0.0.0:3080
2024-08-12 12:56:34,448-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-12 12:56:35,449-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-12 12:56:36,451-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-12 12:56:37,453-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
...
...
...
2024-08-20 14:56:41,483-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:05,898-INFO: /app/.config/source_provider.yaml file has be changed, the kubespider will reboot
2024-08-20 15:53:08,959-INFO: Source Provider:bilibili_source_provider enabled...
2024-08-20 15:53:08,976-INFO: Download Provider:aria2 enabled...
2024-08-20 15:53:08,985-INFO: Period Server Queue handler start running...
2024-08-20 15:53:08,986-INFO: PT Server start running...
2024-08-20 15:53:08,986-INFO: Period Server producer start running...
2024-08-20 15:53:08,986-INFO: Download trigger job start running...
2024-08-20 15:53:08,986-INFO: Notification Server Queue handler start running...
2024-08-20 15:53:08,986-WARNING: Failed to setup telegram bot token, telegram server not started
2024-08-20 15:53:08,987-INFO: Webhook Server start running...
2024-08-20 15:53:08,989-INFO: Serving on http://0.0.0.0:3080
2024-08-20 15:53:15,156-INFO: /app/.config/source_provider.yaml file has be changed, the kubespider will reboot
2024-08-20 15:53:18,189-INFO: Source Provider:bilibili_source_provider enabled...
2024-08-20 15:53:18,198-INFO: Source Provider:alist_source_provider enabled...
2024-08-20 15:53:18,206-INFO: Download Provider:aria2 enabled...
2024-08-20 15:53:18,215-INFO: Period Server Queue handler start running...
2024-08-20 15:53:18,216-INFO: PT Server start running...
2024-08-20 15:53:18,216-INFO: Period Server producer start running...
2024-08-20 15:53:18,216-INFO: Download trigger job start running...
2024-08-20 15:53:18,216-INFO: Notification Server Queue handler start running...
2024-08-20 15:53:18,216-WARNING: Failed to setup telegram bot token, telegram server not started
2024-08-20 15:53:18,217-INFO: Webhook Server start running...
2024-08-20 15:53:18,217-INFO: Serving on http://0.0.0.0:3080
2024-08-20 15:53:19,227-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:20,228-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:21,230-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:22,231-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:23,233-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:24,234-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:25,236-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:26,237-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-20 15:53:27,239-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
miRemid commented 3 months ago

感谢试用KS，麻烦提供一下Alist的版本号以及Alist的日志情况

xtay573269555 commented 3 months ago

alist版本：xhofe/alist:v3.36.0

alist全部日志

[Systemd] ❯ docker logs alist 
INFO[2024-07-31 01:50:30] reading config file: data/config.json        
INFO[2024-07-31 01:50:30] load config from env with prefix:            
INFO[2024-07-31 01:50:30] init logrus...                               
INFO[2024-07-31 01:50:30] init tool aria2 success: aria2 version: 1.37.0 
INFO[2024-07-31 01:50:30] init tool SimpleHttp success: ok             
WARN[2024-07-31 01:50:40] init tool qBittorrent failed: Post "http://172.16.20.26:8080/api/v2/auth/login": dial tcp 172.16.20.26:8080: connect: no route to host 
INFO[2024-07-31 01:50:40] init tool pikpak success: ok                 
INFO[2024-07-31 01:50:40] start HTTP server @ 0.0.0.0:5244             
INFO[2024-07-31 01:50:50] success load storage: [/aliyun_disk_1], driver: [AliyundriveOpen], order: [0] 
INFO[2024-07-31 01:50:50] success load storage: [/aliyun_disk_2], driver: [AliyundriveOpen], order: [2] 
INFO[2024-07-31 01:50:55] success load storage: [/quark_disk_1], driver: [Quark], order: [3] 
INFO[2024-07-31 01:51:01] success load storage: [/baiduyun_disk_1], driver: [BaiduNetdisk], order: [4] 
INFO[2024-08-12 04:56:06] Shutdown server...                           
INFO[2024-08-12 04:56:07] Server exit                                  
INFO[2024-08-12 04:56:27] reading config file: data/config.json        
INFO[2024-08-12 04:56:27] load config from env with prefix:            
INFO[2024-08-12 04:56:27] init logrus...                               
INFO[2024-08-12 04:56:27] init tool pikpak success: ok                 
INFO[2024-08-12 04:56:27] init tool aria2 success: aria2 version: 1.37.0 
INFO[2024-08-12 04:56:27] init tool SimpleHttp success: ok             
INFO[2024-08-12 04:56:27] init tool qBittorrent success: ok            
INFO[2024-08-12 04:56:27] start HTTP server @ 0.0.0.0:5244             
INFO[2024-08-12 04:56:39] success load storage: [/aliyun_disk_1], driver: [AliyundriveOpen], order: [0] 
INFO[2024-08-12 04:56:40] success load storage: [/aliyun_disk_2], driver: [AliyundriveOpen], order: [2] 
INFO[2024-08-12 04:56:45] success load storage: [/quark_disk_1], driver: [Quark], order: [3] 
INFO[2024-08-12 04:56:50] success load storage: [/baiduyun_disk_1], driver: [BaiduNetdisk], order: [4] 
INFO[2024-08-12 04:57:06] success load storage: [/ctyun_disk_1], driver: [189CloudPC], order: [5]
miRemid commented 3 months ago

从KS日志来看，KS在开始是能正常工作的，但后续可能触发Alist的某种机制导致无法获取对应的列表资源

2024-07-31 09:47:14,193-INFO: Start general file download:http://172.16.20.23:5244/d/aliyun_disk_1/push_to_kubespider/test.jpg?modified=2024-07-31T01:45:36.415Z&sign=
2024-07-31 09:47:14,195-INFO: Create download task result:[<aria2p.downloads.Download object at 0x7febf28c6ad0>]
2024-07-31 09:47:14,195-INFO: Find new resource:alist_source_provider/http://172.16.20.23:5244/d/aliyun_disk_1...
2024-07-31 09:47:14,195-INFO: filter downloader type:None, names:['aria2']
2024-07-31 09:47:14,195-INFO: download link type general, with provider size 1
2024-07-31 09:47:14,195-INFO: Download general file with downloader(aria2)
2024-07-31 09:47:14,196-INFO: Start general file download:http://172.16.20.23:5244/d/aliyun_disk_2/push_to_kubespider/test.jpg?modified=2024-07-31T01:45:55.87Z&sign=
2024-07-31 09:47:14,198-INFO: Create download task result:[<aria2p.downloads.Download object at 0x7febf28bdf10>]
2024-07-31 09:47:14,198-INFO: Find new resource:alist_source_provider/http://172.16.20.23:5244/d/baiduyun_disk...
2024-07-31 09:47:14,198-INFO: filter downloader type:None, names:['aria2']
2024-07-31 09:47:14,198-INFO: download link type general, with provider size 1
2024-07-31 09:47:14,198-INFO: Download general file with downloader(aria2)
2024-07-31 09:47:14,200-INFO: Start general file download:http://172.16.20.23:5244/d/baiduyun_disk_1/push_to_kubespider/test.jpg?modified=2024-07-31T01:45:36Z&sign=
2024-07-31 09:47:14,202-INFO: Create download task result:[<aria2p.downloads.Download object at 0x7febf2dce750>]
2024-07-31 10:47:13,050-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-07-31 10:47:14,051-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-07-31 10:47:15,052-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...

能否提供debug模式下的Alist日志情况？目前Alist默认日志信息太少，无法确定问题

xtay573269555 commented 3 months ago

这3个盘的 push_to_kubespider/test.jpg 文件，是我特地放进去的，目的是为了避免监听的目录（push_to_kubespider）为空时导致 KS 报错（获取不到文件会抛出异常）。而且这3个 test.jpg 在 2024-07-31 都正常下载下来了，我当时认为能正常下载就没管这个 func: fs_list 的报错。

但是昨天我将一个包含文件的目录挪到 push_to_kubespider 中后（如下图），发现没有出现下载任务，去看KS日志发现一直报这个错误，且没有触发向 alist 发送下载链接的日志。

2024-08-21_09-53-02

xtay573269555 commented 3 months ago

会不会是这个原因： kubespider/source_provider.yaml 中的配置监听目录只有3个，而 kubespider/state.yaml 文件中的 alist_source_provider 有4个对象，多出来的那个路径导致 fs_list 函数遇到了 NoneType ？

[Systemd] ❯ cat kubespider/state.yaml 
meijutt_source_provider: []
alist_source_provider:
- 244768e3f995161e3321544edb4fc859
- 6d811cb5d70a4f91e827ec45d1200b7a
- 88c2bdc7833d8b56bd10ccbabb7d56b0
- 450b9ef4225790c1ca6deebe21ef96c9
xtay573269555 commented 3 months ago

alist debug信息如下：

2024/08/21 10:26:49 /home/runner/work/alist/alist/internal/db/settingitem.go:20
[1.330ms] [rows:1] SELECT * FROM `x_setting_items` WHERE `key` = "token" ORDER BY `x_setting_items`.`key` LIMIT 1
[GIN] 2024/08/21 - 10:26:49 | 200 |    8.862691ms |      172.17.0.1 | POST     "/api/fs/list"
[GIN] 2024/08/21 - 10:26:50 | 200 |       32.06µs |      172.17.0.1 | POST     "/api/fs/list"
[GIN] 2024/08/21 - 10:26:51 | 200 |      35.756µs |      172.17.0.1 | POST     "/api/fs/list"
[GIN] 2024/08/21 - 10:26:52 | 200 |      33.022µs |      172.17.0.1 | POST     "/api/fs/list"
[GIN] 2024/08/21 - 10:26:53 | 200 |      34.494µs |      172.17.0.1 | POST     "/api/fs/list"
[GIN] 2024/08/21 - 10:26:54 | 200 |      37.038µs |      172.17.0.1 | POST     "/api/fs/list"
[GIN] 2024/08/21 - 10:26:55 | 200 |      32.671µs |      172.17.0.1 | POST     "/api/fs/list"
[GIN] 2024/08/21 - 10:26:56 | 200 |      36.177µs |      172.17.0.1 | POST     "/api/fs/list"
[GIN] 2024/08/21 - 10:26:57 | 200 |      33.512µs |      172.17.0.1 | POST     "/api/fs/list"

对应的KS日志：

2024-08-21 10:25:44,697-INFO: File handler start running...

 _          _                     _     _
| | ___   _| |__   ___  ___ _ __ (_) __| | ___ _ __
| |/ / | | | '_ \ / _ \/ __| '_ \| |/ _` |/ _ \ '__|
|   <| |_| | |_) |  __/\__ \ |_) | | (_| |  __/ |
|_|\_\\__,_|_.__/ \___||___/ .__/|_|\__,_|\___|_|
                           |_|                    

KubeSpider - A global resource download orchestration system
Build Tag: ${GITHUB_SHA}
2024-08-21 10:25:44,774-INFO: Download Provider:aria2 enabled...
2024-08-21 10:25:44,783-INFO: Download trigger job start running...
2024-08-21 10:25:44,784-INFO: Notification Server Queue handler start running...
2024-08-21 10:25:44,784-INFO: Period Server producer start running...
2024-08-21 10:25:44,784-INFO: Period Server Queue handler start running...
2024-08-21 10:25:44,784-INFO: PT Server start running...
2024-08-21 10:25:44,784-WARNING: Failed to setup telegram bot token, telegram server not started
2024-08-21 10:25:44,785-INFO: Webhook Server start running...
2024-08-21 10:25:44,786-INFO: Serving on http://0.0.0.0:3080

2024-08-21 10:26:45,178-INFO: /app/.config/source_provider.yaml file has be changed, the kubespider will reboot
2024-08-21 10:26:48,218-INFO: Source Provider:alist_source_provider enabled...
2024-08-21 10:26:48,225-INFO: Download Provider:aria2 enabled...
2024-08-21 10:26:48,235-INFO: Period Server producer start running...
2024-08-21 10:26:48,235-INFO: Download trigger job start running...
2024-08-21 10:26:48,235-INFO: Notification Server Queue handler start running...
2024-08-21 10:26:48,235-INFO: Period Server Queue handler start running...
2024-08-21 10:26:48,236-INFO: PT Server start running...
2024-08-21 10:26:48,236-INFO: Webhook Server start running...
2024-08-21 10:26:48,237-WARNING: Failed to setup telegram bot token, telegram server not started
2024-08-21 10:26:48,237-INFO: Serving on http://0.0.0.0:3080
2024-08-21 10:26:49,249-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-21 10:26:50,251-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-21 10:26:51,252-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-21 10:26:52,254-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-21 10:26:53,255-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-21 10:26:54,256-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-21 10:26:55,257-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-21 10:26:56,259-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
2024-08-21 10:26:57,260-ERROR: Error happened, func: fs_list, err: 'NoneType' object has no attribute 'get', retrying...
xtay573269555 commented 3 months ago

会不会是这个原因： kubespider/source_provider.yaml 中的配置监听目录只有3个，而 kubespider/state.yaml 文件中的 alist_source_provider 有4个对象，多出来的那个路径导致 fs_list 函数遇到了 NoneType ？

[Systemd] ❯ cat kubespider/state.yaml 
meijutt_source_provider: []
alist_source_provider:
- 244768e3f995161e3321544edb4fc859
- 6d811cb5d70a4f91e827ec45d1200b7a
- 88c2bdc7833d8b56bd10ccbabb7d56b0
- 450b9ef4225790c1ca6deebe21ef96c9

删除 kubespider/state.yaml 文件后 ，再重启 KS，这个文件压根没生成

miRemid commented 3 months ago

state.yaml文件是用于下载去重的，没有下载任务完成该文件不会被更新

能否提供下你现有KS中有关alist的配置，因为根据你上面的描述在测试过程中是完全可行的

miRemid commented 3 months ago

@evell1992 大佬也帮忙看看，我对alist这块不太熟悉

xtay573269555 commented 3 months ago

这是配置目录的所有文件，密码已经清理，其它没动

kubespider.zip

leijuly commented 2 weeks ago

同样的问题，部署后稳定运行了一段时间。 突然开始报错。 后面删除state.yaml，重启docker容器，修改监听目录无效。 最后通过停止容器，再启动恢复了。很奇怪的问题。。。

qingchoulove commented 2 weeks ago

@leijuly 感谢报告，我来跟踪下这个问题，能否辛苦提供下alist版本，我搭建下验证环境。