Closed RReverser closed 6 years ago
Updated to group output, it's less noisy this way.
Not sure if I should attempt to remove these automatically or going manually one by one will be easier...
For some of them, conceptually, it kind of makes sense to be that way. For example, UserEcho.com.xml
lists "actual" subdomains first and then the wildcard is used to cover customer subdomains.
What I like about it is that the day custom subdomains support is removed, we are just one line away to fix this ruleset, we don't have to find and check every "regular" subdomain again.
A similar alternative would be:
<target host="userecho.com"/>
<target host="*.userecho.com"/>
<!-- All regular subdomains -->
<test url="http://www.userecho.com/"/>
<test url="http://blog.userecho.com/"/>
<test url="http://feedback.userecho.com/"/>
<!-- Customer subdomains examples -->
<test url="http://imgur.userecho.com/"/>
<test url="http://unchecky.userecho.com/"/>
But I have no idea how big an impact this overlapping targets have on performance.
That's interesting, looking at https://github.com/EFForg/https-everywhere/pull/12320/commits/cdc90c422a7a0addfefa37a744286bef7333227d, it looks like most of the overlaps are due to the fact that the ruleset author didn't account for the fact that left wildcards affect more than first-level subdomains.
Well, I noticed them as part of hunt for wildcard-in-the-middle regexps, and thought that it makes sense to remove overlaps completely as they're somewhat confusing.
On the other hand, I see your point too, even though 1) *.somehost
hosts are better for both CPU and memory performance than long lists of precise subdomains (especially with reverse-trie approach) and 2) it's future-proof for cases when domain gets new subdomains that also need http: -> https: rewrite and so I'd rather wish to keep these than long lists of specific targets.
But I know not everyone in HTTPS Everywhere team agrees with this 😄
Anyway, in cases where they overlap, I think using <test url />
makes more sense if it's just for documentation purposes, although by the time custom subdomains are gone, ruleset will likely have to update list of targets anyway as new subdomains might appear or host might get HSTS etc.
That's interesting, looking at cdc90c4, it looks like most of the overlaps are due to the fact that the ruleset author didn't account for the fact that left wildcards affect more than first-level subdomains.
Yup, I got the same impression.
But I have no idea how big an impact this overlapping targets have on performance.
AFAIK, the lookup performance will be improved as the complicated operations in rules.js#L347-L354
can be avoid.
That's interesting, looking at cdc90c4, it looks like most of the overlaps are due to the fact that the ruleset author didn't account for the fact that left wildcards affect more than first-level subdomains.
I do not know this until recently when I create #11998 looking at rules.js. Maybe we should document this behavior.
It is documented at https://www.eff.org/https-everywhere/rulesets like some other details, but yes, looks like that information is out of sync with contributing guidelines in the repo.
Automated fix was merged, so closing this as resolved.
https://github.com/EFForg/https-everywhere/pull/12320 helped to find a list of rulesets where targets overlap, and so unnecessarily duplicate each other. Reducing number of those might help with #12232:
*.16163.com
also coverswww.16163.com
,app.16163.com
,bbs.16163.com
,m.16163.com
*.3min.de
also coverswww.3min.de
*.acenet-inc.net
also covers*.esupport.acenet-inc.net
*.ace.advertising.com
also coverssecure.ace.advertising.com
*.afrinic.net
also covers*.meeting.afrinic.net
*.airtricity.com
also coverswww.airtricity.com
*.alice-dsl.de
also coverswww.alice-dsl.de
*.american.edu
also covers*.wcl.american.edu
*.asmp.org
also coverswww.admin.asmp.org
*.anl.gov
also coverswww.*.anl.gov
*.argos.co.uk
also coversimage.email.argos.co.uk
*.audiko.net
also coverscss.cdn.audiko.net
,jpg.st.audiko.net
*.uni-weimar.de
also covers*.webmail.uni-weimar.de
*.bennetts.co.uk
also covers*.quotes.bennetts.co.uk
*.brenet.de
also covers*.webmail.brenet.de
*.caller.com
also coverswww.caller.com
,login.caller.com
*.cfda.gov
also covers*.www.cfda.gov
*.champssports.com
also covers*.e.champssports.com
,*.www.champssports.com
*.city.ac.uk
also coverswww.soi.city.ac.uk
,*.www.city.ac.uk
*.clara.net
also coverswebmail.bln.de.clara.net
,nswebmail.uk.clara.net
,portal.uk.clara.net
*.cloudhexa.com
also covers*.www.cloudhexa.com
*.hr.columbia.edu
also covers*.managers.hr.columbia.edu
*.compendiumblog.com
also coverscdn2.content.compendiumblog.com
,cdn.content.compendiumblog.com
,global.content.compendiumblog.com
*.cfr.org
also coverssecure.www.cfr.org
*.cox.com
also covers*.store.cox.com
*.cox.net
also coversidm.east.cox.net
*.adage.com
also coverswww.amiga.adage.com
*.dailymotion.com
also coversak2.static.dailymotion.com
*.digia.com
also coversblog.qt.digia.com
*.img-dpreview.com
also covers*.static.img-dpreview.com
*.insightmgr.com
also covers*.digi.insightmgr.com
,*.www.insightmgr.com
*.dotcomhost.com
also covers*.webmail.dotcomhost.com
*.dpfile.com
also coversqcloud.dpfile.com
,www.dpfile.com
*.eastbay.com
also covers*.teamsales.eastbay.com
,*.www.eastbay.com
*.epi.org
also covers*.secure.epi.org
*.thesims3.com
also covers*.store.thesims3.com
*.epson.com
also coversglobal.latin.epson.com
*.eyeviewads.com
also coverstrack.eyeviewads.com
*.familie-redlich.de
also coverswww.systeme.familie-redlich.de
*.fbo.gov
also covers*.www.fbo.gov
*.filmthreat.com
also covers*.www.filmthreat.com
*.final-score.com
also covers*.e.final-score.com
,*.www.final-score.com
*.footaction.com
also covers*.www.footaction.com
*.freelancer.com
also covers*.www.freelancer.com
*.freelancer.co.uk
also covers*.www.freelancer.co.uk
*.fwmrm.net
also covers2912a.v.fwmrm.net
*.futurequest.net
also coverswww.service.futurequest.net
*.gaiaonline.com
also covers*.cdn.gaiaonline.com
*.gsn.com
also coverswww.tv.gsn.com
*.garmin.com
also coverswww.garmin.com
*.gigaserver.cz
also covers*.www.gigaserver.cz
*.gizmodo.com
also covers20khz.gizmodo.com
,es.gizmodo.com
,factually.gizmodo.com
,fieldguide.gizmodo.com
,homeofthefuture.gizmodo.com
,indefinitelywild.gizmodo.com
,io9.gizmodo.com
,lego.gizmodo.com
,offworld.gizmodo.com
,paleofuture.gizmodo.com
,reframe.gizmodo.com
,space.gizmodo.com
,sploid.gizmodo.com
,throb.gizmodo.com
,toyland.gizmodo.com
,us.gizmodo.com
,www.gizmodo.com
,cache.gizmodo.com
google.*
also coversgoogle.com
images.google.*
also coversimages.google.com
*.googleusercontent.com
also covers*.corp.googleusercontent.com
google.*
also coversgoogle.com
*.gostats.com
also coverswww.ssl.gostats.com
*.griffith.edu.au
also covers*.secure.griffith.edu.au
*.hitbtc.com
also coversaffiliate.hitbtc.com
,auth.hitbtc.com
,blog.hitbtc.com
,demo.hitbtc.com
,forum.hitbtc.com
,www.hitbtc.com
*.home.pl
also covers*.akcje.home.pl
,*.panel.home.pl
,*.poczta.home.pl
,*.m.poczta.home.pl
*.poczta.home.pl
also covers*.m.poczta.home.pl
*.honest.com
also coversblog.honest.com
,img.honest.com
,www.honest.com
*.hotpics-amateur.com
also coverswww.collection.hotpics-amateur.com
*.imageshack.us
also coverswww.imageshack.us
,imagizer.imageshack.us
,post.imageshack.us
,a.imageshack.us
*.bkb.ch
also coversintegration.quotes.bkb.ch
*.kidsfootlocker.com
also covers*.www.kidsfootlocker.com
*.kixeye.com
also covers*.cdn.kixeye.com
*.ladyfootlocker.com
also covers*.www.ladyfootlocker.com
*.lastpass.com
also coverswww.lastpass.com
,0.lastpass.com
,account.lastpass.com
,accounts.lastpass.com
,blog.lastpass.com
,download.lastpass.com
,enterprise.lastpass.com
,forums.lastpass.com
,helpdesk.lastpass.com
,localvault.lastpass.com
,m.lastpass.com
,manda.lastpass.com
,pollserver.lastpass.com
,portable.lastpass.com
,rodan.lastpass.com
,service.lastpass.com
,teams.lastpass.com
,uber.lastpass.com
,vaul.lastpass.com
*.linuxnewmedia.com
also covers*.shop.linuxnewmedia.com
*.linuxnewmedia.de
also covers*.shop.linuxnewmedia.de
*.liquidweb.com
also coversmedia.cdn.liquidweb.com
*.london2012.com
also coverswww.festival.london2012.com
,tickets.london2012.com
,www.tickets.london2012.com
*.loopia.se
also covers*.www.loopia.se
*.lumosity.com
also coversstatic.sl.lumosity.com
*.foto.mail.ru
also coversavt.foto.mail.ru
,content.foto.mail.ru
*.my.mail.ru
also coverscontent.foto.my.mail.ru
,stat.my.mail.ru
,videoapi.my.mail.ru
*.maricopa.edu
also covers*.sis.maricopa.edu
*.maxymiser.com
also covers*.www.maxymiser.com
*.mediafire.com
also coversstaticcdn.mediafire.com
,www.mediafire.com
,www1.mediafire.com
,www2.mediafire.com
,cdn.mediafire.com
,cdnssl.mediafire.com
,m.mediafire.com
*.meituan.com
also coverswww.meituan.com
,analytics.meituan.com
,b.meituan.com
,daili.meituan.com
,hotel.meituan.com
,mos.meituan.com
,p0.meituan.com
,p1.meituan.com
,passport.meituan.com
,report.meituan.com
,s0.meituan.com
,s1.meituan.com
,waimaie.meituan.com
,i.meituan.com
,waimai.meituan.com
,kaidian.waimai.meituan.com
*.mentor.com
also covers*.store1.mentor.com
*.micgadget.com
also covers*.store.micgadget.com
*.msnbc.msn.com
also covers*.today.msnbc.msn.com
*.mytalkdesk.com
also coverswww.mytalkdesk.com
*.nature.nps.gov
also coversscience.nature.nps.gov
*.navteq.com
also coverscss.mapreporter.navteq.com
*.itespresso.fr
also coversquiz.itespresso.fr
*.oberlin.edu
also covers*.cs.oberlin.edu
,oncampus.csr.oberlin.edu
*.onesite.com
also covers*.admin.onesite.com
*.onsugar.com
also coverssecure.*.onsugar.com
,www.*.onsugar.com
*.pair.com
also covers*.webmail.pair.com
*.upc.edu
also covers*.blog.upc.edu
*.poppysports.com
also covers*.www.poppysports.com
*.prxy.com
also coverswww.prxy.com
*.ru.nl
also covers*.cmbi.ru.nl
,*.hosting.ru.nl
,*.portalhelp.hosting.ru.nl
*.hosting.ru.nl
also covers*.portalhelp.hosting.ru.nl
*.royalmail.com
also covers*.shop.royalmail.com
*.rtems.org
also coversdevel.rtems.org
,docs.rtems.org
,git.rtems.org
,lists.rtems.org
,wiki.rtems.org
,www.rtems.org
*.secunet.com
also coverswww.secunet.com
*.superforum.org
also covers*.dating.superforum.org
*.sxnarod.com
also coverswap.dating.sxnarod.com
*.sheetmusicplus.com
also coversssl.assets.sheetmusicplus.com
*.skrill.com
also coversaccount.skrill.com
,help.skrill.com
,sso.skrill.com
,www.skrill.com
*.snagajob.com
also coverswww.snagajob.com
*.spoki.lv
also covers*.www.spoki.lv
*.stumbleupon.com
also covers*.b9.stumbleupon.com
*.symanteccloud.com
also coversbuy.symanteccloud.com
,static.symanteccloud.com
,static1.symanteccloud.com
,static2.symanteccloud.com
,static3.symanteccloud.com
*.ad-srv.net
also covers*.ad.ad-srv.net
*.o2.cz
also covers*.www.o2.cz
*.textbooks.com
also covers*.www.textbooks.com
*.escapistexpo.com
also coverswww.sec.escapistexpo.com
*.ubs.com
also covers*.ibb.ubs.com
*.ucsd.edu
also coversa4.ucsd.edu
,acs-webmail.ucsd.edu
,altng.ucsd.edu
,aventeur.ucsd.edu
,cinfo.ucsd.edu
,facilities.ucsd.edu
,gradapply.ucsd.edu
,graduateapp.ucsd.edu
,jacobsstudent.ucsd.edu
,myucsdchart.ucsd.edu
,sdacs.ucsd.edu
,shs.ucsd.edu
,ted.ucsd.edu
,ucsdbkst.ucsd.edu
,a.ucsd.edu
,acms.ucsd.edu
,bookstore.ucsd.edu
,www.bookstore.ucsd.edu
,cs.ucsd.edu
,www.cs.ucsd.edu
,cse.ucsd.edu
,www.cse.ucsd.edu
,ece.ucsd.edu
,www.ece.ucsd.edu
,hdh.ucsd.edu
,www.hdh.ucsd.edu
,hds.ucsd.edu
,www.hds.ucsd.edu
,maeweb.ucsd.edu
,nanoengineering.ucsd.edu
,www.nanoengineering.ucsd.edu
,ne-web.ucsd.edu
,ne.ucsd.edu
,neweb.ucsd.edu
,roger.ucsd.edu
,se.ucsd.edu
,structures.ucsd.edu
,www.structures.ucsd.edu
,uxt.ucsd.edu
,www-cs.ucsd.edu
,www-cse.ucsd.edu
,www-ne.ucsd.edu
,www-structures.ucsd.edu
,act.ucsd.edu
,health.ucsd.edu
,libraries.ucsd.edu
,studenthealth.ucsd.edu
,www-act.ucsd.edu
,accesslink.ucsd.edu
,acs.ucsd.edu
,cri.ucsd.edu
,desktop.ucsd.edu
,financiallink.ucsd.edu
,iwdc.ucsd.edu
,marketplace.ucsd.edu
,mytritonlink.ucsd.edu
,www.mytritonlink.ucsd.edu
,resnet.ucsd.edu
,software.ucsd.edu
,sysstaff.ucsd.edu
,tritonlink.ucsd.edu
,www.tritonlink.ucsd.edu
,uclearning.ucsd.edu
,webmail.ucsd.edu
,www-acs.ucsd.edu
*.doe.gov
also coverswww.*.doe.gov
*.universalsubtitles.org
also coverss3.www.universalsubtitles.org
*.alaska.edu
also coverswww.*.alaska.edu
,biotech.inbre.alaska.edu
,lib.uaa.alaska.edu
,*.vpn.alaska.edu
*.unibe.ch
also coverswww.*.unibe.ch
*.udel.edu
also covers*.facilities.udel.edu
,*.nss.udel.edu
*.rug.nl
also coverswww.astro.rug.nl
*.uidaho.edu
also coverswww.*.uidaho.edu
,www2.sites.uidaho.edu
*.umass.edu
also covers*.oit.umass.edu
,*.spire.umass.edu
,*.umii.umass.edu
*.usf.edu
also covers*.stpete.usf.edu
*.soton.ac.uk
also coverswww.*.soton.ac.uk
*.uh.edu
also coverswww.*.uh.edu
,fp.my.uh.edu
,*.nsm.uh.edu
*.umaine.edu
also coverswww.*.umaine.edu
*.salford.ac.uk
also covers*.www.salford.ac.uk
*.waikato.ac.nz
also coverstools.its.waikato.ac.nz
,www.mngt.waikato.ac.nz
*.wisc.edu
also covers*.library.wisc.edu
*.uptodown.com
also coverswww.uptodown.com
,api.uptodown.com
,blog.uptodown.com
,dw.uptodown.com
,feeds.uptodown.com
,gstatic.uptodown.com
,img.uptodown.com
,stat.uptodown.com
,stc.uptodown.com
*.vaforvets.va.gov
also coverswww.*.vaforvets.va.gov
*.vba.va.gov
also coverswww.*.vba.va.gov
*.history.state.gov
also coverswww.history.state.gov
*.userecho.com
also coverswww.userecho.com
,blog.userecho.com
,feedback.userecho.com
*.vdopia.com
also coversmobile.sb.vdopia.com
*.wdfiles.com
also covers1.*.wdfiles.com
,2.*.wdfiles.com
,3.*.wdfiles.com
,4.*.wdfiles.com
,5.*.wdfiles.com
,6.*.wdfiles.com
,7.*.wdfiles.com
,8.*.wdfiles.com
,9.*.wdfiles.com
*.wikinvest.com
also covers*.www.wikinvest.com
*.wiley.com
also coversonlinelibrarystatic.wiley.com
,sp.onlinelibrary.wiley.com
*.wolframalpha.com
also coversapi.wolframalpha.com
,api-cn.wolframalpha.com
,api-maps.wolframalpha.com
,api-tw.wolframalpha.com
,developer.wolframalpha.com
,m.wolframalpha.com
,preview.wolframalpha.com
,products.wolframalpha.com
,volunteer.wolframalpha.com
,wc.wolframalpha.com
,www1.wolframalpha.com
,www3.wolframalpha.com
,www4b.wolframalpha.com
,www4c.wolframalpha.com
,www4d.wolframalpha.com
,www4f.wolframalpha.com
,www5a.wolframalpha.com
,www5b.wolframalpha.com
*.woot.com
also coversimages.deals.woot.com
,gzip.static.woot.com
cc @Hainish @cschanaj @koops76 @Bisaloo