mneunomne commented 3 years ago

Documentation for collecting different types of google ads

`.GoogleActiveViewElement`

Examples:

Filters to allow:

@@||googleads.g.doubleclick.net^$subdocument
@@||c.statcounter.com^$ping
@@||secure.statcounter.com^$script

#@#.adsbygoogle
#@#.ads_1

#@#+js(no-floc)

mneunomne commented 3 years ago

@dhowe can you help me list website with collectable google ads (non-video ads)? I created this so I can document in one place a way to better collect them.

Some challenges collecting google ads:

Their structure can vary a lot, so I need to make querySelectors to collect title, body, url and image(if there are any).
As mentioned above, I need to create some exception for them to be rendered, and if we make these exceptions not domain-specific, it will require a lot of testing to see if we are not letting any ads appear.
They are very region specfic, the screenshot of the ads you showed me have a completely different structure of the ones I usually see. We need to figure out as many "patterns" as we can.

Perhaps one way we can document these ads is when we find them, we print their html structure here so we can know what we are dealing with ? It will be easier for me to create the inner queries for title and body later on.

For example: test-capture

<div class="ns-nof7x-e-0 x-layout GoogleActiveViewElement web-on-show" data-google-av-adk="3374222452" data-google-av-aid="0" data-google-av-btr="" data-google-av-cpmav="" data-google-av-cxn="https://pagead2.googlesyndication.com/pcs/activeview?xai=AKAOjsvxmJ8KDmHCLP_esu2KW64Kgol7--rYvp7Rw2cdwctB44BhNCkxxJ5jBBWCC6mYVx0lVeAgg3wljVlEtPHBVZT9rEmOpXnNt1KTpRntGr13CmPUR0NYrQ&amp;sai=AMfl-YQVxcyF-0C_2NBtMywoZbNQZFx7Pj0FICfxIRPbb3tUw9IpX5PQ6J6IpHo1uciJk7tW1QyFR2nymuZq78n4dfhh6G1HbYd-bzhnZq_MNgewgF4U2x_tUaUkH0LqtMa4Rywgbw&amp;sig=Cg0ArKJSzJ2khffXqc_4EAE" data-google-av-dm="2" data-google-av-flags="[&quot;x%278440'9efotm(&amp;753374%2bejvf/%27844>'9wuvb$&amp;56533>!=|vqc)!273794&amp;<qqvb/%<1735020!=nehu`/!364=5051!9abk{a($160210:3&amp;<cbotf+*0150034:%2bejvf/%72;17613!=efdwa*'76463;21$?ebkpb$&amp;0366717>*>bgipf+!3=712363%9aihwc)!7202<217'9efotm(&amp;20061;48&amp;>`dopb~&quot;]" data-google-av-immediate="true" data-google-av-itpl="22" data-google-av-metadata="la=1&amp;xdi=0&amp;" data-google-av-naid="1" data-google-av-override="-1" data-google-av-rs="2" data-google-av-slift="" data-ifc="[[[&quot;5,5,5,5&quot;,null,null,2]]]" id="banner-large-vanilla" lang="de" x-ns-nof7x-e="0" x-overflow-forbidden="xy" data-creative-load-listener=""><div class="ns-nof7x-e-1 column-container banner-large-board flex-filler" x-ns-nof7x-e="1" x-overflow-forbidden="xy" x-score="62"><div class="ns-nof7x-e-2 column-container background flex-filler" x-ns-nof7x-e="2" x-overflow-forbidden="xy"><div class="ns-nof7x-e-3 column-container canvas content-flex-filler center-justified" x-ns-nof7x-e="3" x-overflow-forbidden="xy"><div class="ns-nof7x-e-4 title-line1" title="Acronis.com" x-ns-nof7x-e="4" x-overflow-forbidden="xy"><a class="ns-nof7x-e-5" data-asoch-targets="ad0,titleClk" dir="auto" href="https://www.googleadservices.com/pagead/aclk?sa=L&amp;ai=CO5RmwH9IYcHOOpuu7_UPkP-NmAntmcqGY4uNlsOQDaTn8u2VAhABIILkjAVgleqOgpwHoAGP9vn2A8gBAagDAcgDywSqBLgBT9A7R-TfW7I2xo7Vi31v1aHLhm1H16Q5CIoi37q56YS7k1VdS6ko0y3RmJndKwz1qlSQaA44O7vR1BZ8AL9byuw4cVgGXyie1_BOrYq4k52ISrbh2BrS738hxgWDueoKGngcATVZim6u5s8aX-lf1HoAGlMpPXjBfw3RRK1uo56J1rtmJgm40IVEqUh-uaIZPGlOr53EPGYy7a3xlTDHjIjAeUhsNw4Ff_1c323JJF1tFt0sZJttRcAEj7ixvJ0DkAYBgAePtZwSiAcBkAcCqAfVyRuoB_DZG6gH8tkbqAeOzhuoB5PYG6gHugaoB-6WsQKoB6a-G6gH89EbqAeW2BuoB6qbsQKoB9-fsQLYBwHSCAkIgICAEBACGBqxCc88desaaBMygAoBmAsByAsBogwIKgYKBPmesQK4DAHYEw2IFAPQFQGAFwE&amp;ae=1&amp;num=1&amp;cid=CAMSeQClSFh3PyHEFBi9v_SCk6moSis2PcD8MH1U-t2u8jVDiwFlX6XYFAQ4cm7C78r5WmP9jqY9lyvpIVQOhRUQS6NtrM14hOiphlk-Brj0v2iEdTdURPJFLukarG5mjtp3WxNqFJ9nm6ltno4OH5VFb1y_4NCy-6bzo9E&amp;sig=AOD64_2iSUvIRw-huRd1ovPPO2igb3P-6A&amp;client=ca-pub-4185349146732778&amp;nb=0&amp;adurl=https://www.acronis.com/de-de/products/cloud/%3Futm_content%3Dsmart%26gclid%3DCjwKCAjw4qCKBhAVEiwAkTYsPPusl1uUyc4Cu6zFIxRA4rKO8tKPGfejD4Pp8v3QZjYAeKP2ca1jahoCZfcQAvD_BwE" target="_top" x-ns-nof7x-e="5" x-remove="false">Die #1 für Service Provider</a></div><div class="ns-nof7x-e-6 body" x-ns-nof7x-e="6" x-overflow-forbidden="xy"><a class="ns-nof7x-e-7" data-asoch-targets="ad0,bodyClk" dir="auto" href="https://www.googleadservices.com/pagead/aclk?sa=L&amp;ai=CO5RmwH9IYcHOOpuu7_UPkP-NmAntmcqGY4uNlsOQDaTn8u2VAhABIILkjAVgleqOgpwHoAGP9vn2A8gBAagDAcgDywSqBLgBT9A7R-TfW7I2xo7Vi31v1aHLhm1H16Q5CIoi37q56YS7k1VdS6ko0y3RmJndKwz1qlSQaA44O7vR1BZ8AL9byuw4cVgGXyie1_BOrYq4k52ISrbh2BrS738hxgWDueoKGngcATVZim6u5s8aX-lf1HoAGlMpPXjBfw3RRK1uo56J1rtmJgm40IVEqUh-uaIZPGlOr53EPGYy7a3xlTDHjIjAeUhsNw4Ff_1c323JJF1tFt0sZJttRcAEj7ixvJ0DkAYBgAePtZwSiAcBkAcCqAfVyRuoB_DZG6gH8tkbqAeOzhuoB5PYG6gHugaoB-6WsQKoB6a-G6gH89EbqAeW2BuoB6qbsQKoB9-fsQLYBwHSCAkIgICAEBACGBqxCc88desaaBMygAoBmAsByAsBogwIKgYKBPmesQK4DAHYEw2IFAPQFQGAFwE&amp;ae=1&amp;num=1&amp;cid=CAMSeQClSFh3PyHEFBi9v_SCk6moSis2PcD8MH1U-t2u8jVDiwFlX6XYFAQ4cm7C78r5WmP9jqY9lyvpIVQOhRUQS6NtrM14hOiphlk-Brj0v2iEdTdURPJFLukarG5mjtp3WxNqFJ9nm6ltno4OH5VFb1y_4NCy-6bzo9E&amp;sig=AOD64_2iSUvIRw-huRd1ovPPO2igb3P-6A&amp;client=ca-pub-4185349146732778&amp;nb=7&amp;adurl=https://www.acronis.com/de-de/products/cloud/%3Futm_content%3Dsmart%26gclid%3DCjwKCAjw4qCKBhAVEiwAkTYsPPusl1uUyc4Cu6zFIxRA4rKO8tKPGfejD4Pp8v3QZjYAeKP2ca1jahoCZfcQAvD_BwE" target="_top" x-ns-nof7x-e="7">Acronis® Cyber Cloud für Service Provider: Backup, Malware-, Ransomware-Abwehr, DraaS+Mehr</a></div></div><div class="ns-nof7x-e-8 column-container white-background flex-filler" x-ns-nof7x-e="8" x-overflow-forbidden="xy"><div class="ns-nof7x-e-9 row-container flex-filler" x-ns-nof7x-e="9" x-overflow-forbidden="xy"><div class="ns-nof7x-e-10 row-container banner-tall-bar" x-ns-nof7x-e="10" x-overflow-forbidden="xy"><div class="ns-nof7x-e-11 row-container flex-filler" x-ns-nof7x-e="11" x-overflow-forbidden="xy"><div class="ns-nof7x-e-12 url" title="Acronis.com" x-ns-nof7x-e="12" x-overflow-forbidden="xy"><a class="ns-nof7x-e-13" data-asoch-targets="ad0,urlClk" dir="auto" href="https://www.googleadservices.com/pagead/aclk?sa=L&amp;ai=CO5RmwH9IYcHOOpuu7_UPkP-NmAntmcqGY4uNlsOQDaTn8u2VAhABIILkjAVgleqOgpwHoAGP9vn2A8gBAagDAcgDywSqBLgBT9A7R-TfW7I2xo7Vi31v1aHLhm1H16Q5CIoi37q56YS7k1VdS6ko0y3RmJndKwz1qlSQaA44O7vR1BZ8AL9byuw4cVgGXyie1_BOrYq4k52ISrbh2BrS738hxgWDueoKGngcATVZim6u5s8aX-lf1HoAGlMpPXjBfw3RRK1uo56J1rtmJgm40IVEqUh-uaIZPGlOr53EPGYy7a3xlTDHjIjAeUhsNw4Ff_1c323JJF1tFt0sZJttRcAEj7ixvJ0DkAYBgAePtZwSiAcBkAcCqAfVyRuoB_DZG6gH8tkbqAeOzhuoB5PYG6gHugaoB-6WsQKoB6a-G6gH89EbqAeW2BuoB6qbsQKoB9-fsQLYBwHSCAkIgICAEBACGBqxCc88desaaBMygAoBmAsByAsBogwIKgYKBPmesQK4DAHYEw2IFAPQFQGAFwE&amp;ae=1&amp;num=1&amp;cid=CAMSeQClSFh3PyHEFBi9v_SCk6moSis2PcD8MH1U-t2u8jVDiwFlX6XYFAQ4cm7C78r5WmP9jqY9lyvpIVQOhRUQS6NtrM14hOiphlk-Brj0v2iEdTdURPJFLukarG5mjtp3WxNqFJ9nm6ltno4OH5VFb1y_4NCy-6bzo9E&amp;sig=AOD64_2iSUvIRw-huRd1ovPPO2igb3P-6A&amp;client=ca-pub-4185349146732778&amp;nb=1&amp;adurl=https://www.acronis.com/de-de/products/cloud/%3Futm_content%3Dsmart%26gclid%3DCjwKCAjw4qCKBhAVEiwAkTYsPPusl1uUyc4Cu6zFIxRA4rKO8tKPGfejD4Pp8v3QZjYAeKP2ca1jahoCZfcQAvD_BwE" target="_top" x-ns-nof7x-e="13">Acronis.com</a></div></div><div class="ns-nof7x-e-14 space-holder" x-ns-nof7x-e="14" x-overflow-forbidden="xy"></div><a class="ns-nof7x-e-15 cta-outer cta-outer-animated common-animation-pattern" data-asoch-targets="ad0,btnClk" href="https://www.googleadservices.com/pagead/aclk?sa=L&amp;ai=CO5RmwH9IYcHOOpuu7_UPkP-NmAntmcqGY4uNlsOQDaTn8u2VAhABIILkjAVgleqOgpwHoAGP9vn2A8gBAagDAcgDywSqBLgBT9A7R-TfW7I2xo7Vi31v1aHLhm1H16Q5CIoi37q56YS7k1VdS6ko0y3RmJndKwz1qlSQaA44O7vR1BZ8AL9byuw4cVgGXyie1_BOrYq4k52ISrbh2BrS738hxgWDueoKGngcATVZim6u5s8aX-lf1HoAGlMpPXjBfw3RRK1uo56J1rtmJgm40IVEqUh-uaIZPGlOr53EPGYy7a3xlTDHjIjAeUhsNw4Ff_1c323JJF1tFt0sZJttRcAEj7ixvJ0DkAYBgAePtZwSiAcBkAcCqAfVyRuoB_DZG6gH8tkbqAeOzhuoB5PYG6gHugaoB-6WsQKoB6a-G6gH89EbqAeW2BuoB6qbsQKoB9-fsQLYBwHSCAkIgICAEBACGBqxCc88desaaBMygAoBmAsByAsBogwIKgYKBPmesQK4DAHYEw2IFAPQFQGAFwE&amp;ae=1&amp;num=1&amp;cid=CAMSeQClSFh3PyHEFBi9v_SCk6moSis2PcD8MH1U-t2u8jVDiwFlX6XYFAQ4cm7C78r5WmP9jqY9lyvpIVQOhRUQS6NtrM14hOiphlk-Brj0v2iEdTdURPJFLukarG5mjtp3WxNqFJ9nm6ltno4OH5VFb1y_4NCy-6bzo9E&amp;sig=AOD64_2iSUvIRw-huRd1ovPPO2igb3P-6A&amp;client=ca-pub-4185349146732778&amp;nb=8&amp;adurl=https://www.acronis.com/de-de/products/cloud/%3Futm_content%3Dsmart%26gclid%3DCjwKCAjw4qCKBhAVEiwAkTYsPPusl1uUyc4Cu6zFIxRA4rKO8tKPGfejD4Pp8v3QZjYAeKP2ca1jahoCZfcQAvD_BwE" target="_top" title="Acronis.com" x-ns-nof7x-e="15" x-overflow-forbidden="xy" x-remove="false"><div class="ns-nof7x-e-16 cta-background-wrapper" x-ns-nof7x-e="16"><div class="ns-nof7x-e-17 cta-background cta-background-animated common-animation-pattern" x-ns-nof7x-e="17" x-overflow-forbidden="xy"></div></div><div class="ns-nof7x-e-18 cta-button animated" x-ns-nof7x-e="18" x-overflow-forbidden="xy"><div class="ns-nof7x-e-19 cta-text animated cta-text-animated common-animation-pattern" x-ns-nof7x-e="19" x-overflow-forbidden="xy"><span class="ns-nof7x-e-20" dir="auto" x-ns-nof7x-e="20" x-score="1">Herunterladen</span></div></div></a></div></div></div></div></div></div>

mneunomne commented 2 years ago

@dhowe I think this could be removed from milestone? Maybe we can put this in a "Documentation" segment, so we can easily access these "on going reference issues" ?

dhowe commented 2 years ago

You mean the info about the DOM elements? Whatever you prefer, but I'd like to get this into the next release -- its quite important.

mneunomne commented 2 years ago

Seems that some .GoogleActiveViewElement are still being collected but NOT all... I wish there was a page showing all the different types of ads one can create. Wonder if I can test create ads with google platform to test this.

dhowe commented 2 years ago

We do have a Google advertising account, so might be able to use it for this...

mneunomne commented 2 years ago

Uncollected Google ads with image sequence:

https://edition.cnn.com/business

Screenshot 2022-08-17 at 11 57 34