search

econchick / new-coder

New Coder tutorials
zlib License
597 stars 394 forks source link

wrong XPATHs Scrape tutorial #140

Closed caynan closed 9 years ago

caynan commented 9 years ago

All the XPATHs on livingsocial_spider.py is wrong. So because of that the scrapy doesn't generate any data.

caynan commented 9 years ago

I already corrected all the xpaths and the website text. The only problem now seems to be with the category field on which is not informed, on the product div, anymore. So what should I do about that?

caynan commented 9 years ago

Hello it's been quite a while, I was wondering if anyone has a position about this. Should I adapt the tutorial to the new design?

econchick commented 9 years ago

Hi @caynan ! I'm not at my computer right now to look- what items are not being scraped because of incorrect xpaths? Is it still just category?

caynan commented 9 years ago

Hi @econchick I'm going to take a look again, it's been a while that I have fixed the xpaths. I'll commit the changes for the variables that still exist. And leave open to discussion what to do with the ones that is not possible to get the info from the page.

caynan commented 9 years ago

@econchick just created push request with fixes, please take a look. Also do you still have plans in transforming the tutorials in iPython notebooks?

econchick commented 9 years ago

@caynan - wrt iPython notebooks - yes! during the long holiday weekend in two weeks, I have plans to "hack" all weekend on new coder (finally). If you'd like to participate with me - like via IRC or something - let me know! even if it's for a few hours. @merwok you interested at all?

caynan commented 9 years ago

@econchick sounds like a plan for me :smile: I'm caynan on freenode, just ping me. And about the category field, should I remove it from the code/tutorial?

econchick commented 9 years ago

Hey @caynan. A couple of things:

  1. I took your PR code and adapted it in - thank you so much! I also made a couple of changes (e.g. since there's no category of deals, I just removed it entirely, as well as added orig_price field and end_date field).
  2. About the iPython NB - I've decided that I want to create IPyNBs for all the tutorials (tutorial language & code) for the "Workshop Kit" portion of the site (newcoder.io/workshop). I am going to create a separate issue for that describing more in detail. If you want, you can assign it to yourself (the whole issue, or particular tutorials).

I'll push a site & code update a little bit later. Thanks!

caynan commented 9 years ago

Hey @econchick

I'd love to help with the IPyNB migration, should I just create a notebook containing the text and the codes as already in the tutorials? Or should we try to do something like leaving cells blank for the user to fulfill?

:smile: