Ycirn
commented
2 years ago Ill fix
Closed naslundx closed 2 years ago
Ycirn
commented
2 years ago Ill fix
ghost
commented
2 years ago My automation program worked pretty well. Turns out most of the duplicate sites were www-prefixed sites. I have a list of all the non-duplicate websites now.
`
'https://absolutbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://avangard.ru/rus/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://avtogradbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://autokreditbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://agroros.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ai-bank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://bank.yandex.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ak-bars.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://aksonbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://aktivbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://akcia-bank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://alexbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://alefbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://albank.ru/ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://altbb.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://capitalbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://alfabank.com/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://atb.su/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://andjibank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mfk-bank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://aresbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://bank-arzamas.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://aspectbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://assotsiatsiyabank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://baikalrosbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://byrich.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://vtb.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://norvikbank.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://zemsky.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://unicreditbank.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://open.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://raiffeisen.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://abr.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://sberbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ria.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://gazeta.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://kp.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://riafan.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://pikabu.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://kommersant.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mk.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://yaplakal.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rbc.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://bezformata.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://gazprom.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://magnit.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://nornickel.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://surgutneftegas.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://tatneft.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://evraz.com/ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://nlmk.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://sibur.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://severstal.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://metalloinvest.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://nangs.org': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rmk-group.ru/ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://tmk-group.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ya.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://polymetalinternational.com/ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://uralkali.com/ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://eurosib.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://omk.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://sberbank.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://vtb.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://gazprombank.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://gosuslugi.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mos.ru/uslugi': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://kremlin.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://government.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mil.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://nalog.gov.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://customs.gov.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://pfr.gov.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rkn.gov.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://lenta.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ria.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ria.ru/lenta/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rbc.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rt.com/': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://en.kremlin.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://smotrim.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://tass.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://tvzvezda.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://vsoloviev.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://1tv.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://vesti.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://online.sberbank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://iz.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ukraina.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://lukoil.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mos.ru/uslugi/': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://kremlin.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://government.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://newletters.kremlin.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://20.kremlin.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://constitution.kremlin.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://belres.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://belta.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://ctv.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://mil.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belarus24.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://alfaradio.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://kgb.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://sb.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://sputnik.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://radiomir.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://radiostalica.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://radiobrestfm.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://tvrmogilev.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://minsknews.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://zarya.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://grodnonews.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://tvr.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://yandex.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belarusbank.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://nbrb.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://brrb.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belapb.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://bankdabrabyt.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mail.rkn.gov.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://cloud.rkn.gov.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mvd.gov.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://pwd.wto.economy.gov.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://stroi.gov.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://proverki.gov.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ont.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://bgb.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belres.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://energo.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'http://bellegprom.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belneftekhim.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://shop-rt.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belarusbank.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belarus.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belmarket.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rec.gov.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mil.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://gazprombank.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://president.gov.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://cleanbtc.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://bonkypay.com/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://changer.club/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://superchange.net': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mine.exchange/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://platov.co': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ww-pay.net/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://delets.cash/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://betatransfer.org': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ramon.money/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://coinpaymaster.com/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://bitokk.biz/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://netex24.net': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://cashbank.pro/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://flashobmen.com/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://abcobmen.com/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ychanger.net/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://multichange.net/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://24paybank.ne': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://royal.cash/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://prostocash.com/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://baksman.org/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://kupibit.me/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://abcobmen.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://fsb.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mcs.mail.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://roe.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://1tv.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://tass.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://relcom.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://gosuslugi41.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://uslugi27.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://gosuslugi29.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://gosuslugi.astrobl.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://prokuratura.gov.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rusvesna.su/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://oac.gov.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://bisr.gov.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mpt.gov.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mfa.gov.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mvd.gov.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mininform.gov.by': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mintrans.gov.by/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://iecp.ru/ep/ep-verification': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://iecp.ru/ep/uc-list': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://uc-osnovanie.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://nucrf.ru/': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://belinfonalog.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://roseltorg.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://astralnalog.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://nwudc.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://center-inform.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://kk.bank/UdTs': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://structure.mil.ru/structure/uc/info.htm': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ucpir.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://dreamkas.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://e-portal.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://izhtender.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://imctax.parus-s.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://icentr.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://kartoteka.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rsbis.ru/elektronnaya-podpis': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://stv-it.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://crypset.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://kt-69.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://24ecp.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://kraskript.com': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ca.ntssoft.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://y-center.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rcarus.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://rk72.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://squaretrade.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ca.gisca.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://otchet-online.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://udcs.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://cit-ufa.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://elkursk.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://icvibor.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ucestp.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mcspro.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://infotrust.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://epnow.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://ca.kamgov.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://mascom-it.ru': { number_of_requests: 0, number_of_errored_responses: 0 },
'https://cfmc.ru': { number_of_requests: 0, number_of_errored_responses: 0 }`Just tested these and they work fine.
Ycirn
commented
2 years ago Merge #51 instead, it removes 40+- duplicates/useless links.
naslundx
commented
2 years ago Yes, we can close this
I ran jshint through the JS part and found doubles. Someone please verify.