search

scrapy / scrapely

A pure-python HTML screen-scraping library
1.86k stars 272 forks source link

Question: automate training #106

Open imduffy15 opened 7 years ago

imduffy15 commented 7 years ago

Hi,

I was wondering if it would be possible to do automating training using something like boilerpipe or goose to do title, content and date discovery?

I know by default those libraries don't supply the extracted nodes and just the values.... Figured it was better to ask before diving in to see if somebody has already done this.

ghost commented 7 years ago

Hi @imduffy15 - The question is interesting, I think, because it makes me ask "why used Scrapely at all in this case". :)

Scrapely will try to infer rules for extracting data, provided with a page and an example of the extracted data. Once "trained" it works the same on every page passed to it. Boilerpipe / etc, will use algorithms to decide which content should be extracted every time a page is passed to it.

If you pass boilerpipe output to Scrapely, and it's fully correct, then you might get Scrapely rules that work well in all cases, where Boilerpipe doesn't. So, that's nice. But if you pass Scrapely boilerpipe output that is wrong or incomplete, then you'll get rubbish rules or no rules at all.

If you trust the boilerpipe output then I guess you could set up a system to do this in a human-reviewed, semi-automated way. But, I don't think the time to develop a system like that would end up saving you time, it sounds more like the "worst of one, plus normal usage of another" rather than "best of both".

Perhaps I misinterpreted your meaning, though; I'm happy to help out or discuss further, either way. :)

AnAppAMonth commented 6 years ago

@cathalgarvey Currently we are creating wrappers manually for each website we scrape, we need relatively high accuracy so using Boilerpipe / etc directly isn't an option. I'm also thinking about using Boilerpipe + Scrapely to auto-generate wrappers, and then manually test them. If a decent percentage of the wrappers work, the manual work to create these wrappers can be saved. What do you think on this approach?