Closed dubstard closed 1 year ago
Apologies for the cold mentions, just an idea i had. I know how to do it on my machine, but not via js and i think it would be neat to have this baked right in.
cat phishfort_whiteliost.txt | sort -u
cat raw_URL_list_for_blocking.txt
cat raw_URL_list_for_blocking.txt| grep -vwE '^.*\b(1in.ch|1inch.exchange|1inch.io|aave.com|acala.network|airswap.io|algoexplorer.io|ampleforth.org|aurora.tari.com|axieinfinity.com|badger.finance|balancer.exchange|bancor.network|bflgroup.ae|binance-cn.com|binance.charity|binance.co|binance.co.ug|binance.com|binance.je|binance.org|binance.us|binance.vision|binance.zendesk.com|binancecn.net|binancelite.com|binancezh.co|binancezh.com|binancezh.net|bisq.network|bit-z.com|bitcoin.co.th|bitfinex.com|bitforex.com|bitkub.com|bitkubnext.com|bitkubnext.exchange|bitmex.com|bitso.com|bittrex.com|blockchain.com|bolha.com|bx.in.th|cex.io|chain.link|chainlink.com|changelly.com|changenow.io|coinbase.com|coindash.io|coindesk.com|coingecko.com|coinmarketcap.com|compound.finance|consensys.net|cryptocompare.com|cryptokitties.co|cscs.ng|curve.fi|cyanre.co.za|decentral.ca|defipulse.com|degentrilogy.com|dether.io|dex.ag|dfinity.org|dharma.io|digifinex.com|district0x.io|duneanalytics.com|dydx.exchange|eharmony.com|eos.io|eth.link|etherdelta.com|etherealsummit.com|ethereum.org|etherscan.io|ethfinex.com|ethgasstation.info|ethglobal.co|ethpool.org|exchange.idex.io|exodus-dev.io|exodus-prod.io|exodus-services.io|exodus-stage.io|exodus.com|exodus.io|exodus.netlify.com|exodus.workers.dev|f2pool.com|facebook.com|fantom.network|fkc.bank|fortmatic.com|ftx.blue|ftx.com|ftx.cool|ftx.digital|ftx.io|ftx.page|ftx.soy|ftx.tech|ftx.us|gate.io|gemini.com|gilded.finance|gitcoin.co|github.com|github.io|gmail.com|gnosis.pm|golem.network|google.com|hblabs.com.ng|hbng.biz|hbng.com|hbng.org|hbnggroup.biz|hbnggroup.com|hbngonline.com|hbnng.com.ng|helpag.com|hitbtc.com|hk.ftx.tech|holaplex.com|idax.pro|idex.io|idex.market|jaxx.io|karura.info|katana.roninchain.com|kattana.io|kia.com|klar.mx|kraken.com|kusama.network|kyber.network|kyberswap.com|lbank.info|ledger.com|liquid.com|localbitcoins.com|localbitcoins.net|localcryptos.com|localethereum.com|looksrare.org|luno.com|magicalcryptoconference.com|makerdao.com|matcha.xyz|matic.network|medium.com|metamask.io|mooniswap.exchange|mstable.org|multis.co|myalgo.com|mycrypto.com|myetherwallet.com|mymonero.com|netlifyglobalcdn.com|nett7.com|nexo.io|nexusmutual.io|numer.ai|nuo.network|oasis.app|octopus.ng|oex.com|okex.com|opensea.io|oppo.ae|oppo.bg|oppo.bh|oppo.co.ke|oppo.com.bd|oppo.com.eg|oppo.com.hk|oppo.com.kz|oppo.com.lk|oppo.com.mm|oppo.com.ng|oppo.com.ph|oppo.com.ro|oppo.com.uz|oppo.hk|oppo.hk.com|oppo.hu|oppo.id|oppo.lk|oppo.lol|oppo.ma|oppo.ph|oppo.pk|oppo.qa|oppo.sa.com|oppo.sg|oppo.tm|oppo.za.com|oppopad.com|originprotocol.com|pancakeswap.ai|pancakeswap.blog|pancakeswap.com|pancakeswap.cz|pancakeswap.finance|pancakeswap.info|pancakeswap.love|paradex.io|parity.io|paxful.com|paxful.com.cn|paypal.com|phantom.app|phishfort.com|polkadot-statue.io|polkadot.com|polkadot.io|polkadot.network|poloniex.com|post.at|radarrelay.com|raydium.io|recordedfuture.com|reddit.com|ripple.com|saga.co.uk|shapeshift.com|shapeshift.io|skymavis.com|solana.com|solana.foundation|solanahackerhouse.com|solanamonthly.com|solanaweekly.com|sphere.finance|spherefinance.store|spherefinance.xyz|stakedao.org|staratlas.com|substrate.dev|substrate.io|sushiswap.org|sushiswapclassic.org|switch.ag|switcheo.network|sybil.org|synthetix.exchange|synthetix.io|tari.com|techdata.com|tenx.tech|thehashmasks.com|thepanomirror.com|tokenlists.info|tokenlists.org|tornado.cash|totalcoin.io|trezor.io|trubi.io|trustwallet.com|tymebank.co.za|tymedigital.co.za|tymedigital.com|unipig.exchange|unisocks.exchange|uniswap.exchange|uniswap.finance|uniswap.info|uniswap.io|uniswap.org|uniswap.pink|upbit.com|uphold.com|usscyber.com|valr.com|wallet.mymonero.com|wallet.roninchain.com|wallet.trezor.io|walletconnect.com|walletconnect.org|walletsrecovery.org|web3.foundation|workers.dev|wsce.world|www.hbng.com|xliquidus.com|xmr.to|y.at|yam.finance|yearn.finance|yellowcard.io|ygov.finance|zapper.fi|zb.com|zerion.io)\b.*$' >Filtered_URL_list_for_blocking.txt
#Filtered_URL_list_for_blocking.txt should now contain only domains which are not listed above
I tested this "PoC" with two URLs
#!/usr/bin/env node
require('shelljs/global');
cat('phishfort_whiteliost.txt').sort('-u');
cat('raw_URL_list_for_blocking.txt');
cat('raw_URL_list_for_blocking.txt').grep('-vwE', '^.*\b(1in.ch|1inch.exchange|1inch.io|aave.com|acala.network|airswap.io|algoexplorer.io|ampleforth.org|aurora.tari.com|axieinfinity.com|badger.finance|balancer.exchange|bancor.network|bflgroup.ae|binance-cn.com|binance.charity|binance.co|binance.co.ug|binance.com|binance.je|binance.org|binance.us|binance.vision|binance.zendesk.com|binancecn.net|binancelite.com|binancezh.co|binancezh.com|binancezh.net|bisq.network|bit-z.com|bitcoin.co.th|bitfinex.com|bitforex.com|bitkub.com|bitkubnext.com|bitkubnext.exchange|bitmex.com|bitso.com|bittrex.com|blockchain.com|bolha.com|bx.in.th|cex.io|chain.link|chainlink.com|changelly.com|changenow.io|coinbase.com|coindash.io|coindesk.com|coingecko.com|coinmarketcap.com|compound.finance|consensys.net|cryptocompare.com|cryptokitties.co|cscs.ng|curve.fi|cyanre.co.za|decentral.ca|defipulse.com|degentrilogy.com|dether.io|dex.ag|dfinity.org|dharma.io|digifinex.com|district0x.io|duneanalytics.com|dydx.exchange|eharmony.com|eos.io|eth.link|etherdelta.com|etherealsummit.com|ethereum.org|etherscan.io|ethfinex.com|ethgasstation.info|ethglobal.co|ethpool.org|exchange.idex.io|exodus-dev.io|exodus-prod.io|exodus-services.io|exodus-stage.io|exodus.com|exodus.io|exodus.netlify.com|exodus.workers.dev|f2pool.com|facebook.com|fantom.network|fkc.bank|fortmatic.com|ftx.blue|ftx.com|ftx.cool|ftx.digital|ftx.io|ftx.page|ftx.soy|ftx.tech|ftx.us|gate.io|gemini.com|gilded.finance|gitcoin.co|github.com|github.io|gmail.com|gnosis.pm|golem.network|google.com|hblabs.com.ng|hbng.biz|hbng.com|hbng.org|hbnggroup.biz|hbnggroup.com|hbngonline.com|hbnng.com.ng|helpag.com|hitbtc.com|hk.ftx.tech|holaplex.com|idax.pro|idex.io|idex.market|jaxx.io|karura.info|katana.roninchain.com|kattana.io|kia.com|klar.mx|kraken.com|kusama.network|kyber.network|kyberswap.com|lbank.info|ledger.com|liquid.com|localbitcoins.com|localbitcoins.net|localcryptos.com|localethereum.com|looksrare.org|luno.com|magicalcryptoconference.com|makerdao.com|matcha.xyz|matic.network|medium.com|metamask.io|mooniswap.exchange|mstable.org|multis.co|myalgo.com|mycrypto.com|myetherwallet.com|mymonero.com|netlifyglobalcdn.com|nett7.com|nexo.io|nexusmutual.io|numer.ai|nuo.network|oasis.app|octopus.ng|oex.com|okex.com|opensea.io|oppo.ae|oppo.bg|oppo.bh|oppo.co.ke|oppo.com.bd|oppo.com.eg|oppo.com.hk|oppo.com.kz|oppo.com.lk|oppo.com.mm|oppo.com.ng|oppo.com.ph|oppo.com.ro|oppo.com.uz|oppo.hk|oppo.hk.com|oppo.hu|oppo.id|oppo.lk|oppo.lol|oppo.ma|oppo.ph|oppo.pk|oppo.qa|oppo.sa.com|oppo.sg|oppo.tm|oppo.za.com|oppopad.com|originprotocol.com|pancakeswap.ai|pancakeswap.blog|pancakeswap.com|pancakeswap.cz|pancakeswap.finance|pancakeswap.info|pancakeswap.love|paradex.io|parity.io|paxful.com|paxful.com.cn|paypal.com|phantom.app|phishfort.com|polkadot-statue.io|polkadot.com|polkadot.io|polkadot.network|poloniex.com|post.at|radarrelay.com|raydium.io|recordedfuture.com|reddit.com|ripple.com|saga.co.uk|shapeshift.com|shapeshift.io|skymavis.com|solana.com|solana.foundation|solanahackerhouse.com|solanamonthly.com|solanaweekly.com|sphere.finance|spherefinance.store|spherefinance.xyz|stakedao.org|staratlas.com|substrate.dev|substrate.io|sushiswap.org|sushiswapclassic.org|switch.ag|switcheo.network|sybil.org|synthetix.exchange|synthetix.io|tari.com|techdata.com|tenx.tech|thehashmasks.com|thepanomirror.com|tokenlists.info|tokenlists.org|tornado.cash|totalcoin.io|trezor.io|trubi.io|trustwallet.com|tymebank.co.za|tymedigital.co.za|tymedigital.com|unipig.exchange|unisocks.exchange|uniswap.exchange|uniswap.finance|uniswap.info|uniswap.io|uniswap.org|uniswap.pink|upbit.com|uphold.com|usscyber.com|valr.com|wallet.mymonero.com|wallet.roninchain.com|wallet.trezor.io|walletconnect.com|walletconnect.org|walletsrecovery.org|web3.foundation|workers.dev|wsce.world|www.hbng.com|xliquidus.com|xmr.to|y.at|yam.finance|yearn.finance|yellowcard.io|ygov.finance|zapper.fi|zb.com|zerion.io)\b.*$').to('Filtered_URL_list_for_blocking.txt');
//Filtered_URL_list_for_blocking.txt should now contain only domains which are not listed above
Hey @dubstard - thank you again for your work in helping maintaining this repo and all your contributions! 🔥
Within our test/
directory, we have an extra "whitelist" that will prevent domains from being added to the blacklist, even if they aren't in src/config.json
. It will fail on the CI - is this something that will help your end goal? Essentially we are treating it as an Alexa100 list, but crypto.
https://github.com/MetaMask/eth-phishing-detect/blob/main/test/dapps.json
have an extra "whitelist" that will prevent domains from being added to the blacklist, even if they aren't in src/config.json
This is exactly what i was hoping for! Ah so it has already been thought of, perfect, thanks! I didn't know that, neat!
Another related idea- Parse all merged PRs related to removing FPs, and build and additional array of "known good - blocked by accident" to compare against This would prevent blocking legit stuff twice by mistake or overlook.
I can prepare the array myself
@409H Could we please add all legit snap URLs to an extra Anti FP whitelist check similar to tranco and the other one
see https://github.com/MetaMask/eth-phishing-detect/issues/13554 https://github.com/MetaMask/eth-phishing-detect/pull/13545
Thanks!
cc @409H The idea above - extracting all legitimate URLs which were mistakenly blocked for whatever reason and then removed from the blocklist would effectively prevent blocking same legitimate URLs twice by mistake, which is sub optimal and embarrassing.
https://github.com/MetaMask/eth-phishing-detect/issues/13554
Additional anti FP WH checks added via https://github.com/MetaMask/eth-phishing-detect/pull/14535 thanks @409H
Hi,
When I accidentally block a good site I feel horrible! So an idea came to me:
Not sure if this is even possible. I would like to suggest the following extra check in the CI/CD automated pipeline, that parses and checks each pull request for known trusted sites in the newly proposed contents in the
blacklist
array and would subsequently prevent them from getting blocked, even if they are not whitelisted per se in MetaMask's own whitelist!As many organizations have a bunch of legitimate domains, apart from their main one, it is extremely time consuming to check each one individually, in order not to accidentally block a legitimate resource.
For example:
1inch
For 1inch the main URL is 1inch.io But those are also not harmful:
As they are registered by the 1inch team to prevent cyber squatters from getting them.
Pancake
For Pancake swap the main URL is
pancakeswap.finance
All those are legit! Same goes for
pancakeswap-lotterystats.info
- again not an official resource by the pancake team, but not harmful in any way, community developed and driven.We communicate with some of the organizations DEXs, DAOs and etc and I can check with them directly before blocking somehting: I also refer to the projects official page etc, but sometimes there are fan made pages which are also perfectly legitimate
Uniswap
Another example -
Uniswap.fish
is not official, but is not a scam!So I would like (if possible) to use this individually vetted whitelist maintained by PhishFort (where Metamask is a customer AFAIK).
cc @409H @legobeat @danfinlay