search

ludoux / ngapost2md

艾泽拉斯国家地理论坛/NGA玩家社区/NGA单帖爬虫/牛国安一键存贴，不怕冲水
MIT License
106 stars 9 forks source link

worker exits from panic: runtime error: invalid memory address or nil pointer dereference #93

Open RichardGuyNotFavMHY opened 8 months ago

RichardGuyNotFavMHY commented 8 months ago

在保存帖子时出现“ worker exits from panic: runtime error: invalid memory address or nil pointer dereference”错误 例子：保存 39607365

ludoux commented 8 months ago 
PS D:\PortableApps\ngapost2md-NEO> .\ngapost2md.exe 39607365
ngapost2md (c) ludoux [ GitHub: https://github.com/ludoux/ngapost2md/tree/neo ]
Version: 1.5.2     2023-10-30T19:40:59+08:00
2024/03/30 12:38:32 下载第 01 页
2024/03/30 12:38:32 下载所有页面总耗时: 0ms
2024/03/30 12:38:32 开始修正第 01 楼层
2024/03/30 12:38:32 开始修正第 00 楼层
2024/03/30 12:38:32 下载图片: 1_1a9611fp.jpg
2024/03/30 12:38:32 下载图片: 0_078b4d96.png
2024/03/30 12:38:32 开始修正第 02 楼层
2024/03/30 12:38:32 开始修正第 03 楼层
2024/03/30 12:38:32 开始修正第 04 楼层
2024/03/30 12:38:32 开始修正第 05 楼层
2024/03/30 12:38:32 修正楼层总耗时: 548ms
2024/03/30 12:38:32 本次任务结束。

今天测试没有问题，可能帖子状态更新了😂这个issue先关闭，后面假如还有类似的请重新开一个，谢谢 :)

RichardGuyNotFavMHY commented 8 months ago

请再试试 39711776 好像超过5楼才会出现问题😂 下面是命令行输出 output.txt

ludoux commented 8 months ago

我还没有测试，但是看输出对照代码应该是获取IP地址相关的函数出了问题。在修复之前可以尝试修改配置文件，把这个功能给关掉。

话说方便贴一下config.ini文件么，涉及隐私的请注意删除，主要是下面的配置：

; 网络并发数，提高理论上可以增加下载速度。仅支持1、2、3。若开启enhance_ori_reply，请将此值设定为1。默认值2。
thread=2
; [#56]每次下载限制新下载的大约页数。到上限后需要重新运行程序再追加下载，如此直至全部下载成功。允许范围-1（含）至100（含）。值为0或-1时则不限制。默认值100（约100页）。
page_download_limit=100

[post]
; [#44]是否查询用户基于IP的地理位置？若启用则会导致至高20倍于未启用的网络请求。默认值False（不启用）。
get_ip_location=False
; [#35]将被回复的楼层内容补充完整。开启此功能要求同步将thread线程数设置为1，否则可能会补充到未format的文本。默认值False（不启用）。
enhance_ori_reply=False
RichardGuyNotFavMHY commented 8 months ago

好的 域名是178那个 ; 网络并发数，提高理论上可以增加下载速度。仅支持1、2、3。若开启enhance_ori_reply，请将此值设定为1。默认值2。 thread=1 ; [#56]每次下载限制新下载的大约页数。到上限后需要重新运行程序再追加下载，如此直至全部下载成功。允许范围-1（含）至100（含）。值为0或-1时则不限制。默认值100（约100页）。 page_download_limit=10

[post] ; [#44]是否查询用户基于IP的地理位置？若启用则会导致至高20倍于未启用的网络请求。默认值False（不启用）。 get_ip_location=True ; [#35]将被回复的楼层内容补充完整。开启此功能要求同步将thread线程数设置为1，否则可能会补充到未format的文本。默认值False（不启用）。 enhance_ori_reply=True ; [#58]是否使用本地表情图片资源而不是引用在线资源。默认值False（不启用）。 use_local_smile_pic=False ; [#58]本地表情图片资源路径。支持绝对路径与相对路径。尾部需要以 / 结尾。 local_smile_pic_path=../smile/ ; [#21]文件夹名是否包含标题。默认值False。修改后仅对全新拉取的tid生效。 use_title_as_folder_name=True ; [#21]Markdown 文件名是否为标题。默认值False。修改后仅对全新拉取的tid生效。 use_title_as_md_file_name=False

RichardGuyNotFavMHY commented 8 months ago

好像确认了，关闭IP地址记录功能可以暂时性的解决这个bug。

oarinv commented 8 months ago

api地址

Post("nuke.php?lib=ucp&act=get&__output=11")

if GET_IP_LOCATION {
            Client := req.NewClient()
            Client.SetCommonHeader("cookie", COOKIE)
            Client.SetBaseURL(BASE_URL)
            Client.SetUserAgent(UA)
            resp, err := Client.R().SetFormData(map[string]string{
                "uid": cast.ToString(floor.UserId),
            }).Post("nuke.php?__lib=ucp&__act=get&__output=11")
            if err != nil {
                log.Println(err.Error())
            } else {
                println(cast.ToString(resp.Bytes()))
                value_str, err := jsonparser.GetString(resp.Bytes(), "data", "0", "ipLoc")
                if err != nil {
                    log.Println("获取用户IP位置失败: " + err.Error())
                } else {
                    floor.IpLocation = cast.ToString(value_str)
                }
            }
        }
2024/03/30 22:26:48 下载第 01 页
2024/03/30 22:26:49 下载第 02 页
2024/03/30 22:26:49 下载第 03 页
2024/03/30 22:26:49 下载第 04 页
2024/03/30 22:26:49 下载第 05 页
2024/03/30 22:26:49 下载所有页面总耗时: 832ms
2024/03/30 22:26:49 开始修正第 00 楼层
{"data":[{"uid":65356758,"username":"UID65356758","gid":42,"groupid":42,"memberid":42,"group":"\u8b66\u544a\u7b49\u7ea71","title":"","honor":"","bit":135266817,"regdate":1696764883,"muteTime":0,"medal":"0","avatar":"","sign":"","clientChat":0,"umengChat":0,"lastpost":0,"lastvisit":0,"thisvisit":0,"more_info":[{"type":8,"data":1602},{"type":9,"data":1}],"ipLoc":"\u6c5f\u897f","userForum":false,"adfree":0,"follow":0,"follow_by_num":8,"faction":0,"money":0,"fame":0,"rvrc":0,"posts":0,"adminForums":null,"_admin":0,"_super":0,"_greater":0,"_lesser":0}],"time":1711808809}
2024/03/30 22:26:49 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 下载图片: 0_e4750dlc.jpg
2024/03/30 22:26:50 开始修正第 01 楼层
{"data":[{"uid":988779,"username":"UID988779","gid":39,"groupid":39,"memberid":39,"group":"\u5b66\u5f92","title":"","honor":"","bit":169148929,"regdate":1211780962,"muteTime":0,"medal":"425","avatar":"","sign":"","clientChat":0,"umengChat":0,"lastpost":0,"lastvisit":0,"thisvisit":0,"more_info":[{"type":8,"data":304049},{"type":9,"data":31}],"ipLoc":"\u5c71\u4e1c","userForum":false,"adfree":0,"follow":0,"follow_by_num":26,"faction":0,"money":0,"fame":0,"rvrc":0,"posts":0,"adminForums":null,"_admin":0,"_super":0,"_greater":0,"_lesser":0}],"time":1711808810}
2024/03/30 22:26:50 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 开始修正第 02 楼层
{"data":[{"uid":43336262,"username":"UID43336262","gid":39,"groupid":39,"memberid":39,"group":"\u5b66\u5f92","title":"","honor":"","bit":169082880,"regdate":1537185777,"muteTime":0,"medal":"0","avatar":"","sign":"","clientChat":0,"umengChat":0,"lastpost":0,"lastvisit":0,"thisvisit":0,"more_info":[{"type":8,"data":14793}],"ipLoc":"\u6c5f\u82cf","userForum":false,"remark":true,"adfree":0,"follow":0,"faction":0,"money":0,"fame":0,"rvrc":0,"posts":0,"adminForums":null,"_admin":0,"_super":0,"_greater":0,"_lesser":0}],"time":1711808810}
2024/03/30 22:26:50 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 开始修正第 03 楼层
{"data":[{"uid":61015774,"username":"UID61015774","gid":39,"groupid":39,"memberid":39,"group":"\u5b66\u5f92","title":"","honor":"","bit":135266304,"regdate":1569135146,"muteTime":0,"medal":"0","avatar":"","sign":"","clientChat":0,"umengChat":0,"lastpost":0,"lastvisit":0,"thisvisit":0,"more_info":[{"type":8,"data":519}],"ipLoc":"\u4e2d\u56fd","userForum":false,"remark":true,"adfree":0,"follow":0,"faction":0,"money":0,"fame":0,"rvrc":0,"posts":0,"adminForums":null,"_admin":0,"_super":0,"_greater":0,"_lesser":0}],"time":1711808810}
2024/03/30 22:26:50 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 开始修正第 04 楼层

2024/03/30 22:26:50 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 开始修正第 05 楼层

2024/03/30 22:26:51 获取用户IP位置失败: Key path not found
2024/03/30 22:26:51 开始修正第 06 楼层

2024/03/30 22:26:51 获取用户IP位置失败: Key path not found
2024/03/30 22:26:51 开始修正第 07 楼层

获取api时，到第四层时获取不到ip地址数据，但是通过api工具bruno可以正常获取所有的数据

ludoux commented 8 months ago

api地址

Post("nuke.php?lib=ucp&act=get&__output=11")

if GET_IP_LOCATION {
          Client := req.NewClient()
          Client.SetCommonHeader("cookie", COOKIE)
          Client.SetBaseURL(BASE_URL)
          Client.SetUserAgent(UA)
          resp, err := Client.R().SetFormData(map[string]string{
              "uid": cast.ToString(floor.UserId),
          }).Post("nuke.php?__lib=ucp&__act=get&__output=11")
          if err != nil {
              log.Println(err.Error())
          } else {
              println(cast.ToString(resp.Bytes()))
              value_str, err := jsonparser.GetString(resp.Bytes(), "data", "0", "ipLoc")
              if err != nil {
                  log.Println("获取用户IP位置失败: " + err.Error())
              } else {
                  floor.IpLocation = cast.ToString(value_str)
              }
          }
      }
2024/03/30 22:26:48 下载第 01 页
2024/03/30 22:26:49 下载第 02 页
2024/03/30 22:26:49 下载第 03 页
2024/03/30 22:26:49 下载第 04 页
2024/03/30 22:26:49 下载第 05 页
2024/03/30 22:26:49 下载所有页面总耗时: 832ms
2024/03/30 22:26:49 开始修正第 00 楼层
{"data":[{"uid":65356758,"username":"UID65356758","gid":42,"groupid":42,"memberid":42,"group":"\u8b66\u544a\u7b49\u7ea71","title":"","honor":"","bit":135266817,"regdate":1696764883,"muteTime":0,"medal":"0","avatar":"","sign":"","clientChat":0,"umengChat":0,"lastpost":0,"lastvisit":0,"thisvisit":0,"more_info":[{"type":8,"data":1602},{"type":9,"data":1}],"ipLoc":"\u6c5f\u897f","userForum":false,"adfree":0,"follow":0,"follow_by_num":8,"faction":0,"money":0,"fame":0,"rvrc":0,"posts":0,"adminForums":null,"_admin":0,"_super":0,"_greater":0,"_lesser":0}],"time":1711808809}
2024/03/30 22:26:49 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 下载图片: 0_e4750dlc.jpg
2024/03/30 22:26:50 开始修正第 01 楼层
{"data":[{"uid":988779,"username":"UID988779","gid":39,"groupid":39,"memberid":39,"group":"\u5b66\u5f92","title":"","honor":"","bit":169148929,"regdate":1211780962,"muteTime":0,"medal":"425","avatar":"","sign":"","clientChat":0,"umengChat":0,"lastpost":0,"lastvisit":0,"thisvisit":0,"more_info":[{"type":8,"data":304049},{"type":9,"data":31}],"ipLoc":"\u5c71\u4e1c","userForum":false,"adfree":0,"follow":0,"follow_by_num":26,"faction":0,"money":0,"fame":0,"rvrc":0,"posts":0,"adminForums":null,"_admin":0,"_super":0,"_greater":0,"_lesser":0}],"time":1711808810}
2024/03/30 22:26:50 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 开始修正第 02 楼层
{"data":[{"uid":43336262,"username":"UID43336262","gid":39,"groupid":39,"memberid":39,"group":"\u5b66\u5f92","title":"","honor":"","bit":169082880,"regdate":1537185777,"muteTime":0,"medal":"0","avatar":"","sign":"","clientChat":0,"umengChat":0,"lastpost":0,"lastvisit":0,"thisvisit":0,"more_info":[{"type":8,"data":14793}],"ipLoc":"\u6c5f\u82cf","userForum":false,"remark":true,"adfree":0,"follow":0,"faction":0,"money":0,"fame":0,"rvrc":0,"posts":0,"adminForums":null,"_admin":0,"_super":0,"_greater":0,"_lesser":0}],"time":1711808810}
2024/03/30 22:26:50 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 开始修正第 03 楼层
{"data":[{"uid":61015774,"username":"UID61015774","gid":39,"groupid":39,"memberid":39,"group":"\u5b66\u5f92","title":"","honor":"","bit":135266304,"regdate":1569135146,"muteTime":0,"medal":"0","avatar":"","sign":"","clientChat":0,"umengChat":0,"lastpost":0,"lastvisit":0,"thisvisit":0,"more_info":[{"type":8,"data":519}],"ipLoc":"\u4e2d\u56fd","userForum":false,"remark":true,"adfree":0,"follow":0,"faction":0,"money":0,"fame":0,"rvrc":0,"posts":0,"adminForums":null,"_admin":0,"_super":0,"_greater":0,"_lesser":0}],"time":1711808810}
2024/03/30 22:26:50 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 开始修正第 04 楼层

2024/03/30 22:26:50 获取用户IP位置失败: Key path not found
2024/03/30 22:26:50 开始修正第 05 楼层

2024/03/30 22:26:51 获取用户IP位置失败: Key path not found
2024/03/30 22:26:51 开始修正第 06 楼层

2024/03/30 22:26:51 获取用户IP位置失败: Key path not found
2024/03/30 22:26:51 开始修正第 07 楼层

获取api时，到第四层时获取不到ip地址数据，但是通过api工具bruno可以正常获取所有的数据

感觉是被流控了，服务端返回了一些报错的信息，软件没有识别到

oarinv commented 8 months ago

通过python的脚本测试，也会经常返回失败，所以api被限制了

RichardGuyNotFavMHY commented 8 months ago

也许可以参照下 https://github.com/kisshang1993/NGA-BBS-Script 他的脚本不知是如何获取ip地址信息的 貌似没有登录这个脚本也能显示ip信息 编辑： 参考 https://greasyfork.org/zh-CN/scripts/416741-nga-likes-support 的信息 登录情况下调用这个接口会有限制 没登录的情况下反而没有限制