search

puppylpg / oddish

Crawl csgo skin info from `buff.163.com` and steam, then find the most suitable one to buy from the former and to sell to the latter.
https://puppylpg.github.io/2019/12/07/python-crawler-buff-optimaze/
GNU General Public License v3.0
321 stars 80 forks source link

爬取完buff数据时 在爬取steam市场时 数据类型报错 #16

Closed metathrone closed 3 years ago

metathrone commented 4 years ago

爬取完buff的数据之后，抛出异常 2020-08-31 15:07:15,795 [INFO ] Total 142 items to get history price. 2020-08-31 15:07:15,795 [INFO ] GET steam history price 1/142 for (AK-47（StatTrak™） | 混沌点阵 (久经沙场)): https://steamcommunity.com/market/pricehistory/?country=CN&currency=1&appid=730&market_hash_name=StatTrak%E2%84%A2%20AK-47%20%7C%20Point%20Disarray%20%28Field-Tested%29 2020-08-31 15:07:15,795 [INFO ] sleep 3s at 2020-08-31 15:07:15.795036 Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\oddish\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\ProgramData\Anaconda3\envs\oddish\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:/Users/ayane/Downloads/Compressed/python_project/oddish/src__main__.py", line 13, in table = item_crawler.crawl() File "C:\Users\ayane\Downloads\Compressed\python_project\oddish\src\crawl\item_crawler.py", line 129, in crawl table = crawl_website() File "C:\Users\ayane\Downloads\Compressed\python_project\oddish\src\crawl\item_crawler.py", line 76, in crawl_website enrich_item_with_price_history(csgo_items) File "C:\Users\ayane\Downloads\Compressed\python_project\oddish\src\crawl\item_crawler.py", line 57, in enrich_item_with_price_history history_price_crawler.crawl_history_price(csgo_items) File "C:\Users\ayane\Downloads\Compressed\python_project\oddish\src\crawl\history_price_crawler.py", line 41, in crawl_history_price crawl_item_history_price(index, item, total_price_number) File "C:\Users\ayane\Downloads\Compressed\python_project\oddish\src\crawl\history_price_crawler.py", line 17, in crawl_item_history_price raw_price_history = steam_history_prices['prices'] TypeError: list indices must be integers or slices, not str

Process finished with exit code 1

puppylpg commented 4 years ago

好的今晚回去我看一下，可能是steam返回的数据结构变了=。=

puppylpg commented 4 years ago

这个确实很奇怪，我仔细核对了一下，steam返回的数据结构没有变，可能是在很特殊的情况下才会出现这种情况，我代码里处理一下，碰到这种罕见错误就忽略了，继续爬

puppylpg commented 4 years ago

https://github.com/puppylpg/oddish/releases/tag/v3.6.1

10935336 commented 3 years ago

@puppylpg 遇到了同样的问题，已经是3.6.1版了。 全都是这样的错误，一个也没爬到，最后的结果也都是NaN

2020-09-25 17:32:55,063 [INFO ]  GET steam history price 3977/3978 for (印花 | 喷射火焰（全息）): https://steamcommunity.com/market/pricehistory/?country=CN&currency=1&appid=730&market_hash_name=Sticker%20%7C%20Firestarter%20%28Holo%29
2020-09-25 17:32:55,063 [INFO ]  sleep 3s at 2020-09-25 17:32:55.063777
2020-09-25 17:32:58,514 [ERROR]  Traceback (most recent call last):
  File "C:\Users\a1093\Desktop\oddish-3.6.1\oddish-3.6.1\src\crawl\history_price_crawler.py", line 43, in crawl_history_price
    crawl_item_history_price(index, item, total_price_number)
  File "C:\Users\a1093\Desktop\oddish-3.6.1\oddish-3.6.1\src\crawl\history_price_crawler.py", line 18, in crawl_item_history_price
    raw_price_history = steam_history_prices['prices']
TypeError: list indices must be integers or slices, not str

2020-09-25 17:32:58,516 [INFO ]  GET steam history price 3978/3978 for (印花 | 招财猫（闪亮）): https://steamcommunity.com/market/pricehistory/?country=CN&currency=1&appid=730&market_hash_name=Sticker%20%7C%20Lucky%20Cat%20%28Foil%29
2020-09-25 17:32:58,516 [INFO ]  sleep 4s at 2020-09-25 17:32:58.516990
2020-09-25 17:33:02,966 [ERROR]  Traceback (most recent call last):
  File "C:\Users\a1093\Desktop\oddish-3.6.1\oddish-3.6.1\src\crawl\history_price_crawler.py", line 43, in crawl_history_price
    crawl_item_history_price(index, item, total_price_number)
  File "C:\Users\a1093\Desktop\oddish-3.6.1\oddish-3.6.1\src\crawl\history_price_crawler.py", line 18, in crawl_item_history_price
    raw_price_history = steam_history_prices['prices']
TypeError: list indices must be integers or slices, not str
buff买steam卖：

单位价钱收益最大——

After threshold(history_sold >= 70) filtered:
       price  sell_num  steam_predict_price  buy_max_price  gap  gap_percent  history_sold  history_days  average_sold_price  average_sold_price_after_tax
count    0.0       0.0                  0.0            0.0  0.0          0.0           0.0           0.0                 0.0                           0.0
mean     NaN       NaN                  NaN            NaN  NaN          NaN           NaN           NaN                 NaN                           NaN
std      NaN       NaN                  NaN            NaN  NaN          NaN           NaN           NaN                 NaN                           NaN
min      NaN       NaN                  NaN            NaN  NaN          NaN           NaN           NaN                 NaN                           NaN
25%      NaN       NaN                  NaN            NaN  NaN          NaN           NaN           NaN                 NaN                           NaN
50%      NaN       NaN                  NaN            NaN  NaN          NaN           NaN           NaN                 NaN                           NaN
75%      NaN       NaN                  NaN            NaN  NaN          NaN           NaN           NaN                 NaN                           NaN
max      NaN       NaN                  NaN            NaN  NaN          NaN           NaN           NaN
ccinv commented 3 years ago

检查一下 steam_cookie 有没有正确设置或者过期 @metathrone

hooohoooh commented 3 years ago

steam cookie填错了，我也遇到了这个情况，重新填steam cookie就好了@10935336