CI/CD DeDupe broke

[dubstard]

Previously adding a duplicate was not possible, due to CI CD redundancy checks. Lately this has changed and some dupes have snuck in, possibly in rare scenarios / edge cases, so not necessarily broken entirely. Likely when a bot does a merge if it the timing is close with another merge.

Each of those is present more than once:

    "airdrop.claimfuelet.net",
    "allocations.c-zksync.com",
    "bakeriyswap.web.app",
    "central-logs.pages.dev",
    "connectdappsolution.rf.gd",
    "dapddaris.vercel.app",
    "get-lnds.io",
    "lgime-tumskb.webflow.io",
    "mainnetbridges.pages.dev",
    "mealtimaaslogai.webflow.io",
    "meta-connect-restoration.webflow.io",
    "metaamaskulogin.webflow.io",
    "metaeskkextin.webflow.io",
    "metamakchmxtensio.webflow.io",
    "metamakloggin.webflow.io",
    "metamasekxtens.webflow.io",
    "metamaskdesk-supportteam-site.webflow.io",
    "metamasklo.cfd",
    "metamaskuixtension.webflow.io",
    "metamasmextension.webflow.io",
    "metamsesask.azurewebsites.net",
    "metamuastnsn.webflow.io",
    "metaschurexttnsion.webflow.io",
    "metmaskichrom.webflow.io",
    "metsmasaek.azurewebsites.net",
    "mettamask-wallet-web.webflow.io",
    "mettamskk-wallet.webflow.io",
    "mitamsk-extensin.webflow.io",
    "mitamskxtnsin.webflow.io",
    "mitasskelchmextin.webflow.io",
    "morphorg.xyz",
    "mute.pro-multicoin.net",
    "mwtwmaskextension.webflow.io",
    "mytamsk-logns.webflow.io",
    "secure-metskelgin.webflow.io",
    "sexy-bakery.org",
    "smart-bsc.cc",
    "solutiondappfix.rf.gd",
    "spprtmetmask.webflow.io",
    "swiftdapprotocol.pages.dev",
    "trust-connect-restoration.webflow.io",
    "westernautomobileassembly.com",
    "zksync-binance.com",
    "zksync-binance.net",

#show only blacklist, raw as-is
$ curl -k https://raw.githubusercontent.com/MetaMask/eth-phishing-detect/main/src/config.json | tail -169091 | wc -l
169091

#show only blacklist, but sort and remove duplicates
$ curl -k https://raw.githubusercontent.com/MetaMask/eth-phishing-detect/main/src/config.json | tail -169091 | sort -u | wc -l
169047

[AlexHerman1]

@samczsun

[NikitaVr]

Thanks for highlighting this, it's also possible on the ChainPatrol side we might not always be running the tests fully, going to debug on our end too.

[dubstard]

total 173260

$ curl -k https://raw.githubusercontent.com/MetaMask/eth-phishing-detect/main/src/config.json | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4480k  100 4480k    0     0  3879k      0  0:00:01  0:00:01 --:--:-- 3882k
173260

unique 173241

$ curl -k https://raw.githubusercontent.com/MetaMask/eth-phishing-detect/main/src/config.json | sort -u | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4480k  100 4480k    0     0  5902k      0 --:--:-- --:--:-- --:--:-- 5903k
173241

173210 lines in the blocklist array at the moment, from which 173199 are unique, so a few duplicates still. Maybe it has already gotten fixed, and those are "leftovers" from pre-fixing this.

Dupes at the moment

    "084106917.com",
    "aiiocation-zircuitl2.net",
    "iayerzerofoundatlon.com",
    "inceptionlrt.org",
    "io-wb.site",
    "layerzeros.xyz",
    "op-tokens.com",
    "stakeairdrop.pro",
    "stretto-cases.com",
    "zksyn.co",
    "zksynschain.com",

Each of has snuck in more than once

[samczsun]

No longer relevant