white-shiro-bai / ghost_sa

open_server for sensorsdata ghost_sa(鬼策)的用途是接收 神策SDK 上报的埋点数据,移动广告监测,站外阅读监测,短链创建与解析,反爬,接入控制与管理,用户分群与召回等功能
325 stars 78 forks source link

上传报错 #48

Open shenhongbin7854 opened 1 year ago

shenhongbin7854 commented 1 year ago

File "/data/ghost-server/component/db_op.py", line 64, in do_tidb_exe results, result_count , lastest_id = exe_tidb(sql=sql, args=args,presql=presql) File "/data/ghost-server/component/db_op.py", line 31, in exe_tidb result_count = cur.execute(query=sql, args=args) File "/data/python3/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/data/python3/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/data/python3/lib/python3.10/site-packages/pymysql/connections.py", line 546, in query sql = sql.encode(self.encoding, "surrogateescape") UnicodeEncodeError: 'utf-8' codec can't encode characters in position 1780-1781: surrogates not allowed

我们切了一些流量到鬼策: "$url": "pages-shop/assistant/index?item={\"id\":6497,\"spuId\":136767,\"content\":\"2.5mm超薄饼胚\\n咔滋咔滋<U+2728>满口酥脆不粘牙\\n咬上一口,满满的芝士香气喷涌而出\\n让人意犹未尽☀\\n下单\ud83d\udc49https://dwz.cn/SFaxkQWt\",\"friendContent\":\"\\n下单\ud83d\udc49https://dwz.cn/SFaxkQWt\",\"video\":\"\",\"videoCover\":\"\",\"imageList\":[\"https://cdn.huoduomei.com/middle-other/web/img/bb094650-59c5-11ed-b58f-8b3b94211c81.jpeg\\",\\"https://cdn.huoduomei.com/middle-other/web/img/ce7c37b0-59c5-11ed-b58f-8b3b94211c81.jpeg\\",\\"https://cdn.huoduomei.com/middle-other/web/img/bb099470-59c5-11ed-b58f-8b3b9421",

这个是不是因为emoj 表情的问题

shenhongbin7854 commented 1 year ago

更新地址来源 None 回调DSP None 2023-08-23 17:49:36.511911,get_datas,Traceback (most recent call last): File "/data/ghost-server/component/api.py", line 139, in get_datas return get_data() File "/data/ghost-server/component/api.py", line 112, in get_data if get_url_params('data'): File "/data/ghost-server/component/url_tools.py", line 45, in get_url_params v = got_json[params] TypeError: string indices must be integers

还有个这个报错

shenhongbin7854 commented 1 year ago

component-->db_op.py-->exe_tidb: if args is not None: for key in args.keys: if type(args[key]) == str: args[key] = args[key].encode('utf-8',"ignore").decode('utf-8') 对参数进行 对表情进行强行处理,解决了报错

white-shiro-bai commented 1 year ago

抱歉,最近比较忙才看到,是可以这样解的。非常感谢您提供的用例。我验证一下,没问题的话会合到代码里。贡献列表里会加上您的Github主页。 可以加我微信"Ben_Xiaobai"

white-shiro-bai commented 11 months ago

更新地址来源 None 回调DSP None 2023-08-23 17:49:36.511911,get_datas,Traceback (most recent call last): File "/data/ghost-server/component/api.py", line 139, in get_datas return get_data() File "/data/ghost-server/component/api.py", line 112, in get_data if get_url_params('data'): File "/data/ghost-server/component/url_tools.py", line 45, in get_url_params v = got_json[params] TypeError: string indices must be integers

还有个这个报错

感谢大佬的反馈,请问这个是具体什么端SDK上报的?我用小程序批量上报和Web单条尝试暂时没能复现出这个场景。

white-shiro-bai commented 11 months ago

component-->db_op.py-->exe_tidb: if args is not None: for key in args.keys: if type(args[key]) == str: args[key] = args[key].encode('utf-8',"ignore").decode('utf-8') 对参数进行 对表情进行强行处理,解决了报错

非常感谢大佬的代码。在这里做了修复。已加贡献列表。

https://github.com/white-shiro-bai/ghost_sa/commit/c56226ba767d87d223f1e6e6b0bc225b3a1fa130

shenhongbin7854 commented 11 months ago

获取参数的时候,拿到json的参数,但是类型是str的,所以加了层处理,现在没有了: 核心是request.json 拿到不是一个字典,是一个str。try: got_json = request.json except: # noqa got_json = None v = None if got_json:

判断这个json的类型

if type(got_json) == str:
    tmpList = got_json.split("=")
    if len(tmpList) &gt;= 2:
        if params == tmpList[0]:
            # 处理字符串
            v = tmpList[1]
    else:
        。。。。。

对着镜子说加油 @.***

 

------------------ 原始邮件 ------------------ 发件人: "white-shiro-bai/ghost_sa" @.>; 发送时间: 2023年10月29日(星期天) 下午4:48 @.>; @.**@.>; 主题: Re: [white-shiro-bai/ghost_sa] 上传报错 (Issue #48)

更新地址来源 None 回调DSP None 2023-08-23 17:49:36.511911,get_datas,Traceback (most recent call last): File "/data/ghost-server/component/api.py", line 139, in get_datas return get_data() File "/data/ghost-server/component/api.py", line 112, in get_data if get_url_params('data'): File "/data/ghost-server/component/url_tools.py", line 45, in get_url_params v = got_json[params] TypeError: string indices must be integers

还有个这个报错

感谢大佬的反馈,请问这个是具体什么端SDK上报的?我用小程序批量上报和Web单条尝试暂时没能复现出这个场景。

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

white-shiro-bai commented 11 months ago

@shenhongbin7854 我找到了一个更好的办法。可以把emoji保留下来了。 https://github.com/white-shiro-bai/ghost_sa/commit/a23ae0242fd6e9d7edd794d1913ef0ac17f0bdf2