dhowe
commented
7 years ago From @cqx931 on November 11, 2016 2:39
STEP 1 Getting a set of websites according to topic/demographic
(I will call this approach #1). From here we need to get to a set of websites. For some of these cases, we may be able to use ‘top * sites’, like ‘Top 100 Sports Sites’. Once we have a set of sites, then we can visit them and extract links to other sites (a normal crawling procedure). Ideally this process would continue indefinitely with more and more ads being found over time (there are some research projects that take a similar approach).
I had not thought of your idea of using ad-blocking lists (approach #2), but it is a good one. As I say above, the key is getting ads for a specific topic. Maybe we can find some topic-specific ad lists? Language is of course one very limited way of doing this…
Another way would be (#3) to start with a bunch of keywords, then search them on search engines and simply load the pages that the search engine returns (and possibly follow links from these pages as well).
A few thoughs about this:
- Topic is easier than demographic, as topic links to keyword directly and demographic are a set of possible interesting points. We can start with topic and find a way to combine them into a demographic profile later (personally I find demographic profile more interesting...).
1.Top site list + crawling In the crawling process, are we going to follow all the links to other websites, or the range of the links is constrained within the topic area? One idea I have about crawling is to use this method to go through more content pages from a website. As many ads are in the content pages rather than homepage. And this process is also more similar to the real process from a user. By doing this we could increase the ads found from one website(other approaches could be repeat the process or do the same test set on another day), then a shorter website list is needed for the same amount of ads.
2.Ad-Blocking List My idea to use this is more to the direction of getting a long list of websites with visual ads... I didn't find any topic categorized ad-blocking lists, but one interesting thing we could do with these sites is to get the website meta info/or feed to google to see the first search result. After that, we could do an analysis of these short info and match them to specific categories according to keyword. But apparently more efforts and time are needed for this method.
cqx931
janetthecoder
From @dhowe on November 10, 2016 17:35
Requirements:
Tasks:
Existing options:
References:
Copied from original issue: dhowe/AdNauseam#612