archroad
commented
9 years ago omg, Just realized the copy-paste of the HTML code is terrible. I tried to attach a text file here but failed. Looks like no way to show my debug results?
Closed archroad closed 9 years ago
archroad
commented
9 years ago omg, Just realized the copy-paste of the HTML code is terrible. I tried to attach a text file here but failed. Looks like no way to show my debug results?
archroad
commented
9 years ago I posted this question on Google groups and got the answer there. The answer is that this is a strategy from website to block crawlers. How strange the strategy is! :(
Hi Scrapy experts, I am not sure if this is an existing issue -- I searched in Google, github and didn't find anything similar. Here is my question. I wrote a crawler to crawl products on mango.com. I set CONCURRENT_REQUESTS = 16 in settings.py. However, I found sometimes the response passed to my parse function is ANOTHER page. I used inspect_response(response, self) to stop in my parse function to debug. I found response.url is correct. But if I type view(response), it is a completely different page from response.url. Later if I set CONCURRENT_REQUESTS = 1 and reran the crawler, the problem seems gone.
I pasted my debug results in this thread. The response.url is http://shop.mango.com/US/p0/men/accessories/wool-fedora-hat/?id=33053675_G5&ident=0_accesorio112_0_1415864699123&n=1&s=accesorios_he.sombreros_he&ts=1415864699123. However, I also printed out the response.boy (sorry for the mass page content). You will find it's completely different from the actual url content.
Did I miss anything here?
Thanks, Jun
Debug results ======================================
2014-11-13 00:06:06-0800 [spider_mango] DEBUG: File (uptodate): Downloaded image from <GET http://st.mngbcn.com/rcs/pics/static/T3/fotos/S6/31067584_02.jpg> referred in
2014-11-13 00:06:06-0800 [spider_mango] DEBUG: File (uptodate): Downloaded image from <GET http://st.mngbcn.com/rcs/pics/static/T3/fotos/S6/31067584_02_R.jpg> referred in
2014-11-13 00:06:06-0800 [spider_mango] DEBUG: File (uptodate): Downloaded image from <GET http://st.mngbcn.com/rcs/pics/static/T3/fotos/S6/31067584_02_D1.jpg> referred in
2014-11-13 00:06:06-0800 [spider_mango] DEBUG: File (uptodate): Downloaded image from <GET http://st.mngbcn.com/rcs/pics/static/T3/fotos/outfit/S6/31067584_02-33035589_02-33013683_02-33000307_02-33090327_93.jpg> referred in
2014-11-13 00:06:06-0800 [spider_mango] DEBUG: Now parsing http://shop.mango.com/US/p0/men/accessories/wool-fedora-hat/?id=33053675_G5&ident=0_accesorio112_0_1415864699123&n=1&s=accesorios_he.sombreros_he&ts=1415864699123
[s] Available Scrapy objects:
[s] crawler <scrapy.crawler.Crawler object at 0x10a7ea9d0>
[s] item {}
[s] request <GET http://shop.mango.com/US/p0/men/accessories/wool-fedora-hat/?id=33053675_G5&ident=0_accesorio112_0_1415864699123&n=1&s=accesorios_he.sombreros_he&ts=1415864699123>
[s] response <200 http://shop.mango.com/US/p0/men/accessories/wool-fedora-hat/?id=33053675_G5&ident=0_accesorio112_0_1415864699123&n=1&s=accesorios_he.sombreros_he&ts=1415864699123>
[s] settings <scrapy.settings.Settings object at 0x109c33a50>
[s] spider <SpiderMango 'spider_mango' at 0x10a7f82d0>
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] view(response) View response in a browser
In [1]: response.xpath('//div[@itemprop="name"]/h1/text()').extract()) File "", line 1
response.xpath('//div[@itemprop="name"]/h1/text()').extract())
^
SyntaxError: invalid syntax
In [2]: response.xpath('//div[@itemprop="name"]/h1/text()').extract() Out[2]: [u'\n\t\t\t\t\t\t\t\tRibbed wool beanie\n\t\t\t\t\t\t\t']
In [3]: response.xpath('//div[@itemprop="name"]/h1/text()').extract() Out[3]: [u'\n\t\t\t\t\t\t\t\tRibbed wool beanie\n\t\t\t\t\t\t\t']
In [4]: view(response) Out[4]: True
In [5]: response.url Out[5]: 'http://shop.mango.com/US/p0/men/accessories/wool-fedora-hat/?id=33053675_G5&ident=0_accesorio112_0_1415864699123&n=1&s=accesorios_he.sombreros_he&ts=1415864699123'
In [6]: view(response) Out[6]: True
In [7]: response.xpath('//span[@itemprop="description"]/span/text()').extract() Out[7]: [u'Ribbed wool beanie with rolled-up edge. ']
In [8]: cat = crawl_util.get_substring(response.url, '/US/p0/', '/?id=', 0)
NameError Traceback (most recent call last)
\n\t\t\t\t\t\t\t- \n\t\t\t\tNEW\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tpromotion\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tOutlet\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tfeatured\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tPremium evening\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tRetro Sixties\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tOnline exclusive\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tCardigans and sweaters\n\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tCLOTHING\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tDresses\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tCoats\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJackets\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tLeather\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tCardigans and sweaters\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBlouses and shirts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tT-shirts and tops\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tTrousers\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJeans\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tSkirts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tShorts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJumpsuits\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tPremium\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tEssentials\n\t\t\t\t\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tAccessories\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tShoes\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBags\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJewellery\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tLeather goods\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBelts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tHats and caps\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tFoulards and scarves\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tGloves\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tSunglasses\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tOther accessories\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tCatalogue\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tWinter 2014\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tTouch 2014\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\t2014 Autumn\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tLookbooks\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tNovember\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tMango Premium\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tSeptember\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tTHE MAGAZINE\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tSOCIAL\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tTV\n\t\t\t\t\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tNEW\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tpromotion\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tOutlet\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tCASUAL\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tSUITS\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tBLACKSMITH\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tSPORT\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tfeatured\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tKnit o\'clock\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tLightweight down-feather jackets\n\t\t\t\t\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tCLOTHING\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tCoats\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJackets\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBlazers\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tLeather \n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tCardigans and sweaters\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tShirt\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tT-shirts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tPolo shirts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tTrousers\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJeans\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBermudes\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tPyjamas\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tUnderwear and socks\n\t\t\t\t\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tAccessories\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tShoes\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBags\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBelts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tLeather goods\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBracelets\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tGloves, scarves and foulards\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tTies\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tSunglasses\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tHats and caps\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tOther accessories\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tCatalogue\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tWinter 2014\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\t2014 Autumn\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tLookbooks\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\t2014 November\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\t2014 August\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tTHE POST\n\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tNEW\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tpromotion\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tfeatured\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tAutumn Weekend\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tWrap up in style\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tCasual dresses\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tKnitwear collection\n\t\t\t\t\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tCLOTHING\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tDresses\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tCoats\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJackets\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tCardigans and sweaters\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBlouses and shirts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tT-shirts and tops\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tTrousers\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJeans\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tSkirts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJumpsuits\n\t\t\t\t\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tAccessories\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tShoes\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBags\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tJewellery\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tLeather goods\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tBelts\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tHats and caps\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tFoulards and scarves\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tGloves\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\t\t- \n\t\t\t\tCatalogue\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\t2014 Autumn\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tLookbooks\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t- \n\t\t\t\tF/W 2014 Preview\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t- \n\t\t\t\tBROOKLYN GIRL\n\t\t\t\t\t\t\t
\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t