search

mozilla-extensions / firefox-voice

Firefox Voice is an experiment in a voice-controlled web user agent
Mozilla Public License 2.0
287 stars 122 forks source link

[Search] Use a heuristic to detect card #498

Closed ianb closed 4 years ago

ianb commented 4 years ago

There's a ton of different card types and the selectors might go on forever (or maybe not!)

We might be able to use a heuristic based on location in the DOM and getComputedStyle.

ianb commented 4 years ago

Here's a bunch of examples:

Each example has the first one or two elements, the "CSS Selector" (Firefox's attempt to come up with a simple unique selector), the "CSS Path" (all the elements that lead up to the element) and the XPath.

The sidebar is always inside an element #rhs

Each card has this style (sometimes across more than one class):

border: 1px solid #dfe1e5;
border-radius: 8px;

"2d6"

<div class="kp-blk Wnoohf OJXvsb"><div class="xpdopen rYczAc">
div.bkWMgd:nth-child(1) > div:nth-child(1) > div:nth-child(1)
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div.g.mnr-c.g-blk div.kp-blk.Wnoohf.OJXvsb
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/div[1]

Definition (main)

<div class="kp-blk"><div id="tsuid6" jsl="$t t-ORWJ5-27JLU;$x 0;" class="r-iGgjWNaF2TgM xpdbox xpdclose">
#dictionary-modules > div:nth-child(1)
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div.g.knavi.obcontainer.mod div.RQXSBc div div#dictionary-modules.dob-modules.y.yf div.kp-blk
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/div/div/div/div[1]

Definition (sidebar)

<div class="EyBRub kp-blk knowledge-panel Wnoohf OJXvsb" data-hveid="CBIQAQ" data-ved="2ahUKEwj-idzP8cblAhULRKwKHSayCMwQww0oAHoECBIQAQ"><div class="xpdopen"><div class="ifM9O">
.EyBRub
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div#rhscol.col div#rhs div#rhs_block.rhstc5 div.g.rhsvw.kno-kp.mnr-c.g-blk div.EyBRub.kp-blk.knowledge-panel.Wnoohf.OJXvsb
/html/body/div[6]/div[3]/div[10]/div[1]/div[3]/div/div/div/div[1]

Weather

<div class="vk_c card-section" id="wob_wc"><span aria-level="3" role="heading">
#wob_wc
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div.g.knavi.obcontainer.mod div div#wob_wc.vk_c.card-section
//*[@id="wob_wc"]

"Barack Obama" (sidebar)

<div class="EyBRub kp-blk knowledge-panel Wnoohf OJXvsb" data-hveid="CBoQAQ" data-ved="2ahUKEwj0u_bI8sblAhUNd6wKHchJCL8Qww0oAHoECBoQAQ"><div class="xpdopen"><div class="ifM9O">
.EyBRub
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div#rhscol.col div#rhs div#rhs_block.rhstc5 div.g.rhsvw.kno-kp.mnr-c.g-blk div.EyBRub.kp-blk.knowledge-panel.Wnoohf.OJXvsb
/html/body/div[6]/div[3]/div[10]/div[1]/div[3]/div/div/div/div[1]

"jobs hiring near me"

<div class="k9uN1c kfn9hb" jscontroller="BmlyBe" jsdata="WuaQje;;Axmunw" jsaction="D8I1:rTt2Qd;tb_ts:f20zue;rcuQ6b:npT2md"><div jscontroller="hAJB3c" jsaction="gwAh3e:rJEOTc;Aq4C5:noryc;rcuQ6b:npT2md"><div class="SBFvB" data-hveid="CAcQAg">
.k9uN1c
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div g-card.URhAHe div div.k9uN1c.kfn9hb
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/g-card/div/div

"funny cat videos" (video)

<div class="kp-blk knowledge-block__video-nav-block Wnoohf OJXvsb"><div class="xpdopen"><div class="ifM9O">
.knowledge-block__video-nav-block
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div.g.mnr-c.g-blk div.kp-blk.knowledge-block__video-nav-block.Wnoohf.OJXvsb
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/div

"who created breaking bad"

<div class="EyBRub kp-blk fm06If knowledge-panel Wnoohf OJXvsb" data-hveid="CAoQAQ" data-ved="2ahUKEwiW-ajj88blAhVLcq0KHYxlDJcQww0oAHoECAoQAQ"><div id="tsuid13" jsl="$t t-ORWJ5-27JLU;$x 0;" class="r-igGUOYaBqDVQ xpdbox xpdclose xpdns"><div class="ifM9O">
.EyBRub
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div.g.kno-kp.mnr-c.g-blk div.EyBRub.kp-blk.fm06If.knowledge-panel.Wnoohf.OJXvsb
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/div[1]

"What was the score of the world series game last night"

<div class="imso-loa imso-lhof QMSMXe Wrsj9b" role="link" tabindex="0" jsaction="r.mAecoUMOnoM" data-rtid="ivFM_nOXJ33U" jsl="$x 1;" style="background:#092668" data-ved="2ahUKEwj6gr-K9MblAhVKKKwKHUl_Au8QtUswAHoECAkQAg"><div class="i3wO0d SwsxUd"><g-img class="kno-fb-ctx t7NTy" data-dtype="d3sel" style="height:32px">
.imso-lhof
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div.g.mnr-c.g-blk div.kp-blk.Wnoohf.OJXvsb div.xpdopen.rYczAc div.ifM9O div div.NFQFxe.viOShc.EfDVh.mod div#imso-root div#sports-app.AaS7G.r-inFokbWnwEpg div.iDYYme.inFokbWnwEpg-WJhYXr2KTgo.OiT5Ue.r-ivFM_nOXJ33U div.imso-loa.imso-lhof.QMSMXe.Wrsj9b
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/div/div/div[1]/div/div/div/div/div/div[1]

"traffic in san francisco"

<div class="vk_c" style="position:relative;padding:0;margin:0 0 0 -16px;width:632px">
.vk_c
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div div.g.rQUFld div.vk_c
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/div/div

"how long will it take to get to mountain view"

<div class="vk_c IPbEu"><div class="iQtkylWKFp7w-7JGDfYsJFhk r-iAZTT42NQ_vg" jscontroller="" jsl="$t t-uZ7SuCtZt48$t-TK1mf72HIqg;$x 0;"><div class="GRc6Yb"><div class="PUWPMb">
.vk_c
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div.GguUuf div#lud-ed.y.yf div.g div.ULktNd.rQUFld.rrecc div#lud-dsu.r-iQtkylWKFp7w div.vk_c.IPbEu
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/div[2]/div/div/div/div

"pizza nearby"

<div class="AEprdc vk_c" data-hveid="CAoQAg" data-ved="2ahUKEwj8-ZqP9cblAhUSmuAKHcvWA4EQ34oBegQIChAC">
.AEprdc
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div div.r-iw3YyfjQciCo div.AEprdc.vk_c
/html/body/div[6]/div[3]/div[10]/div[1]/div[2]/div/div[2]/div[2]/div/div/div[1]/div/div/div[2]

"10 + 10"

<div class="vk_c card-section" style="min-height:72px;line-height:1;padding:0" id="cwmcwd" jsaction="mouseover:Q6qFEf;mouseout:lpHYw"><div jscontroller="bTaGX" jsdata="ZBX0ee;;BR4/10" jsaction="i60oqe:Z0NzZd;ISFFIe:YzSkFd;cGuLC:Z2Ekqd;KeDSPe:w07Nsb;NdWHob:Mk4Mke;prs_btn:A2jXUd;prs_drc:IGArIc;prs_eqb:bTMBwc;prs_invb:FJTJHe;prs_dltb:xyppdc;flp_sbsbs_clrs:Qntjtc;dlt_md:x8Eglb">
#cwmcwd
html body#gsr.srp.tbo.vasq div#main div#cnt.big div.mw div#rcnt div.col div#center_col div#res.med div#search div div#rso div.bkWMgd div.g.knavi.obcontainer.NFQFxe.mod div div div#cwmcwd.vk_c.card-section
//*[@id="cwmcwd"]