search

Sotera / DatawakeDepot

Loopback web application for administration of Datawake networks
Apache License 2.0
9 stars 7 forks source link

Panel: Extractions may occur from other page elements not visible #163

Open bmcdougald opened 8 years ago

bmcdougald commented 8 years ago

Looks like we are extracting everything on a given page if it's in HTML. I came across pulldown menus on a site that were just lists of anchor hrefs. The text for these pulldown menus got extracted too. When I added them to the domain, I didn't see them on the page text body itself, but when you would extend the pulldown menu on the page the entry you added would be highlighted as a domain item like any other. screen shot 2016-02-01 at 2 12 54 pm

justinlueders commented 8 years ago

Is this considered a bad thing?

On Feb 1, 2016, at 2:13 PM, Brodie McDougald notifications@github.com wrote:

Looks like we are extracting everything on a given page if it's in HTML. I came across pulldown menus on a site that were just lists of anchor hrefs. The text for these pulldown menus got extracted too. When I added them to the domain, I didn't see them on the page text body itself, but when you would extend the pulldown menu on the page the entry you added would be highlighted as a domain item like any other.

— Reply to this email directly or view it on GitHub.

bmcdougald commented 8 years ago

My personal opinion is that it is undesirable. Pulling terms from page elements like menus and jamming them into the trails just adds a bunch of noise. In most cases the content contained in stuff like menus is not directly related to page being researched/viewed rather the site as a whole. Add to this that menu items don't really have context and it's just more irrelevant garbage. Imagine looking at a website like a forum or something...you're interested in this guy Charlie who's been selling some illegal guns on this forum listing site (i.e. gunbroker). The site has pulldown menus that have "To contact our New York office", and maybe some other links that break down guns by category or something. None of that is relevant to the Charlie search really.

I think there's this general idea that "let's put everything under the moon in our trails" as being a good thing. Personally I disagree. When you start adding tons of irrelevant junk to the data set it becomes muddied and unusable, not to mention difficult for someone to follow up and manually prune the trailed info to get rid of extracted content they don't want that comes in from the viewed page. If we focus on gathering "all the stuff" without some gates for relevancy or context our tools become less focused and less useful in helping people find patterns or information in the data. If it becomes more work for them to sort out garbage so they are left with something useful, they will just stop using our tool altogether.

justinlueders commented 8 years ago

I don't think we are qualified to make that determination. If those items int the menu were types of guns or names of people you could hire you might want that information. I think the extractor should be trained to weed out things like that instead of our tool being specific to one domain or another. "To contact or New York Office" could just as easily be in the body of the page.

I think this is more a bug against the extractor no?

On Mon, Feb 1, 2016 at 2:58 PM, Brodie McDougald notifications@github.com wrote:

My personal opinion is that it is undesirable. Pulling terms from page elements like menus and jamming them into the trails just adds a bunch of noise. In most cases the content contained in stuff like menus is not directly related to page being researched/viewed rather the site as a whole. Add to this that menu items don't really have context and it's just more irrelevant garbage. Imagine looking at a website like a forum or something...you're interested in this guy Charlie who's been selling some illegal guns on this forum listing site (i.e. gunbroker). The site has pulldown menus that have "To contact our New York office", and maybe some other links that break down guns by category or something. None of that is relevant to the Charlie search really.

I think there's this general idea that "let's put everything under the moon in our trails" as being a good thing. Personally I disagree. When you start adding tons of irrelevant junk to the data set it becomes muddied and unusable, not to mention difficult for someone to follow up and manually prune the trailed info to get rid of extracted content they don't want that comes in from the viewed page. If we focus on gathering "all the stuff" without some gates for relevancy or context our tools become less focused and less useful in helping people find patterns or information in the data. If it becomes more work for them to sort out garbage so they are left with something useful, they will just stop using our tool altogether.

— Reply to this email directly or view it on GitHub https://github.com/Sotera/DatawakeDepot/issues/163#issuecomment-178186351 .

bmcdougald commented 8 years ago

We are best qualified to decide how a tool we are designing "should" work I'd think. Especially since we don't get concrete direction or requirements from the customer. I wasn't saying the tool was being confined by domain or training...more that it should focus on the content viewed in the page body...not menu structure or other extraneous unrelated garbage. To use your example, if the blurb of the New York office was in the page...great we're extracting that. Omitting menu garbage is totally unrelated to what we extract from the body.