search

platonai / exotic-amazon

A complete solution to crawl amazon at scale completely and accurately.
143 stars 46 forks source link

java.lang.IllegalArgumentException: Illegal sql, no url found #44

Open ws4435700 opened 3 months ago

ws4435700 commented 3 months ago

https://www.woolworths.co.za/prod/Women/New-In/Knitwear/Woolworths-Cashmere-Jumper/A-507948144?isFromPLP=true https://superbalist.com/women/beauty/nutricosmetics/glow-from-within-single-serving-sachets-10g/1016223?clickRef=catalogue https://superbalist.com/women/beauty/nutricosmetics/glow-from-within-pure-hydrolyzed-collagen-250g/925744?clickRef=catalogue

不知道为什么，这些url在爬取时会报no url found，但是别的url不会

platonai commented 3 months ago

如果是个别错误，就不要管它。

ws4435700 commented 3 months ago

一开始是个别错误，现在发现[https://www.woolworths.co.za]这个网站的所有asin页都有这个问题

ws4435700 commented 3 months ago

想问下这个大概是什么原因，后期有什么方法规避

ws4435700 commented 3 months ago

刚才又试了几次发现了，把url最后面?isFromPLP=true给删掉就不会报错了。似乎加请求参数会让sql这边获取不到url?