Label which links are duplicate / remove duplicate links

[9mido]

Some links belong in multiple categories which is how things are currently set up. If a link is repeated somewhere else, label it as a duplicate to avoid people from re-reading the same thing again. Or simply, just put next to each duplicate link something like "this link shows up in multiple categories", "duplicate", " * ", color code them, or whatever you think is easier.

Or instead of having category headers and the links below them, just have all the links shown without headers and next to each link say which categories they belong to and do not repeat any of the links. This will be better for SEO.

https://www.siteliner.com/endthefud.org?siteliner=site-dashboard&siteliner-sort=scan_time&siteliner-from=1&siteliner-message=

There are also quite a few links that bring you to the same URL but have different anchor tags referencing that particular category. I don't think this is necessary. https://safehodl.github.io/failure is an example of this.

Find all URLs and sort them:

wget -q https://endthefud.org/ -O - | tr "\t\r\n'" ' "' | grep -i -o '<a[^>]\+href[ ]*=[ \t]*"\(ht\|f\)tps\?:[^"]\+"' | sed -e 's/^.*"\([^"]\+\)".*$/\1/g' | sort

Output:

http://btcco2.com http://btcco2.com http://btcco2.org http://btcco2.org https://allenfarrington.medium.com/ https://arxiv.org/abs/1710.10377 https://bitcoinandenergy.medium.com/bitcoin-mining-and-the-case-for-more-energy-90094ce25fac https://bitcoinmagazine.com/business/bitcoin-energy-use-compare-industry https://bitcoinmagazine.com/business/op-ed-bitcoin-miners-consume-reasonable-amount-energy-and-its-all-worth-it https://bitcoinmagazine.com/culture/bitcoin-vs-world-military-emissions https://bitcoinmagazine.com/culture/the-hidden-costs-of-the-petrodollar https://bitcoin.stackexchange.com/questions/6062/what-effects-would-a-scalable-quantum-computer-have-on-bitcoin https://blog.coinbase.com/fact-check-is-bitcoin-mining-environmentally-unfriendly-3559823af6f1 https://blog.getadmiral.com/registration-wall https://blog.lopp.net/are-chinese-miners-threat-bitcoin/ https://blog.lopp.net/are-chinese-miners-threat-bitcoin/ https://blog.lopp.net/who-controls-bitcoin-core-/ https://blog.trezor.io/three-myths-about-bitcoins-energy-consumption-ef613a1f3d5 https://casebitcoin.com/critiques https://casebitcoin.com/critiques/bitcoin-can-be-cloned https://casebitcoin.com/critiques/bitcoin-gets-hacked https://casebitcoin.com/critiques/bitcoin-is-a-bubble https://casebitcoin.com/critiques/bitcoin-is-pumped-by-tether https://casebitcoin.com/critiques/bitcoin-is-used-by-criminals https://casebitcoin.com/critiques/bitcoin-wastes-energy https://casebitcoin.com/critiques/china-controls-bitcoin https://casebitcoin.com/critiques/gov-will-shut-it-down https://casebitcoin.com/critiques/no-intrinsic-value https://coinshares.com/research/bitcoin-mining-network-december-2019 https://danhedl.medium.com/bitcoins-distribution-was-fair-e2ef7bbbc892 https://danhedl.medium.com/pow-is-efficient-aa3d442754d3 https://danheld.substack.com/p/dont-fear-tether https://dergigi.com/2018/06/10/bitcoin-s-energy-consumption/ https://dergigi.com/2021/08/02/implications-of-outlawing-bitcoin/ https://dergigi.com/bitcoin/ https://docsend.com/view/adwmdeeyfvqwecj2 https://download.wpsoftware.net/bitcoin/pos.pdf https://endthefud.org/altcoins https://endthefud.org/ban https://endthefud.org/china https://endthefud.org/control https://endthefud.org/criminals https://endthefud.org/energy https://endthefud.org/governments https://endthefud.org/inflation https://endthefud.org/justice https://endthefud.org/miners https://endthefud.org/money https://endthefud.org/ponzi https://endthefud.org/PoS https://endthefud.org/PoW https://endthefud.org/tech https://en.m.wikipedia.org/wiki/Subjective_theory_of_value https://github.com/libbitcoin/libbitcoin-system/wiki/ASIC-Monopoly-Fallacy https://github.com/libbitcoin/libbitcoin-system/wiki/Cryptoeconomics https://github.com/libbitcoin/libbitcoin-system/wiki/Energy-Exhaustion-Fallacy https://github.com/libbitcoin/libbitcoin-system/wiki/Energy-Waste-Fallacy https://github.com/libbitcoin/libbitcoin-system/wiki/Proof-of-Stake-Fallacy https://github.com/libbitcoin/libbitcoin-system/wiki/Regression-Fallacy https://github.com/libbitcoin/libbitcoin-system/wiki/Shitcoin-Definition https://github.com/libbitcoin/libbitcoin-system/wiki/Value-Proposition https://github.com/pox/EndTheFUD https://github.com/pox/EndTheFUD https://hackernoon.com/the-bitcoin-vs-visa-electricity-consumption-fallacy-8cf194987a50 https://hassmccook.medium.com/comparing-bitcoins-environmental-impact-f56b18014f64 https://hbr.org/2021/05/how-much-energy-does-bitcoin-actually-consume http://shitcoin.review http://shitcoin.review https://hugonguyen.medium.com/ https://hugonguyen.medium.com/proof-of-stake-the-wrong-engineering-mindset-15e641ab65a2 https://hugonguyen.medium.com/work-is-timeless-stake-is-not-554c4450ce18 https://journal.bitcoinreserve.com/bitcoin-a-bold-american-future/ https://mdotbit.medium.com/addressing-concerns-about-bitcoins-electricity-use-378e0de4af42 https://medium.com/@BobMcElrath/whats-wrong-with-proof-of-stake-77d4f370be15 https://medium.com/coinshares/beware-of-lazy-research-c828c900b7d5 https://medium.com/hackernoon/asic-resistance-is-nothing-but-a-blockchain-buzzword-b91d3d770366 https://medium.com/hackernoon/betterhash-decentralizing-bitcoin-mining-with-new-hashing-protocols-291de178e3e0 https://medium.com/hackernoon/bitcoin-miners-beware-invalid-blocks-need-not-apply-51c293ee278b https://medium.com/hackernoon/green-energy-and-bitcoin-eb7961f38382 https://medium.com/john-pfeffer/an-institutional-investors-take-on-cryptoassets-690421158904 https://medium.com/@knut.svanholm https://medium.com/@mukeat/carbon-footprint-differences-btc-vsbanks-a5f8f0b844d0 https://medium.com/@nic__carter https://medium.com/@nic__carter/a-most-peaceful-revolution-8b63b64c203e https://medium.com/@nic__carter/in-support-of-the-proof-of-work-un-fair-launch-cd6e8f06358f https://medium.com/@nic__carter/its-the-settlement-assurances-stupid-5dcd1c3f4e41 https://medium.com/@nic__carter/noahbjectivity-on-bitcoin-mining-2052226310cb https://medium.com/@nic__carter/on-bitcoin-the-gray-lady-embraces-climate-lysenkoism-a2d31e465ec0 https://medium.com/stakin/centralization-of-stake-in-pos-f7ccb8f8254 https://mises.org/library/inflation-and-deflation https://nakamotoinstitute.org/literature/ https://nakamotoinstitute.org/mempool/bitcoin-cannot-be-banned/ https://nakamotoinstitute.org/mempool/bitcoin-cant-be-copied/ https://nakamotoinstitute.org/mempool/bitcoin-does-not-waste-energy/ https://nakamotoinstitute.org/mempool/bitcoin-is-not-a-pyramid-scheme/ https://nakamotoinstitute.org/mempool/bitcoin-is-not-backed-by-nothing/ https://nakamotoinstitute.org/mempool/bitcoin-is-not-for-criminals/ https://nakamotoinstitute.org/mempool/bitcoin-is-not-too-slow/ https://nakamotoinstitute.org/mempool/bitcoin-is-not-too-volatile/ https://nakamotoinstitute.org/mempool/bitcoin-is-the-great-definancialization/ https://nakamotoinstitute.org/mempool/bitcoin-not-blockchain/ https://nakamotoinstitute.org/mempool/bitcoin-obsoletes-all-other-money/ https://nakamotoinstitute.org/mempool/ethereum-is-doomed/ https://nakamotoinstitute.org/mempool/how-we-know-bitcoin-is-not-a-bubble/ https://nakamotoinstitute.org/mempool/speculative-attack/ https://nakamotoinstitute.org/mempool/the-bitcoin-central-banks-perfect-monetary-policy/ https://nakamotoinstitute.org/mempool/the-bullish-case-for-bitcoin/ https://nakamotoinstitute.org/shelling-out/ https://nymag.com/intelligencer/2021/05/jack-dorsey-says-bitcoin-is-climate-friendly-is-he-right.html https://quillette.com/2021/02/21/can-governments-stop-bitcoin/ https://safehodl.github.io/failure https://safehodl.github.io/failure/#block-rewards-will-stop https://safehodl.github.io/failure/#can-be-forked-infinitely https://safehodl.github.io/failure/#central-bank-digital-currencies https://safehodl.github.io/failure/#disinflationary-money-breaks-the https://safehodl.github.io/failure/#governments-will-stop https://safehodl.github.io/failure/#miners-are-centralized-and https://safehodl.github.io/failure/#quantum-computing-will-break https://safehodl.github.io/failure/#scarcity-can-be-changed https://saifedean.com/book/ https://saifedean.com/book/ https://timevalueofbtc.medium.com/ https://tomerstrolight.medium.com/ https://tomerstrolight.medium.com/bitcoins-fundamental-value-is-that-it-is-the-most-excellent-money-that-exists-24be5e1d6b98 https://tomerstrolight.medium.com/the-problem-with-ethereum-af9692f4af95 https://tomerstrolight.medium.com/why-bitcoins-imitators-are-scams-e38fab4c78ba https://tomerstrolight.medium.com/why-bitcoin-the-series-660fe20ec244 https://tonevays.com/ https://tonevays.com/blog/what-is-crypto-decentralization-and-why-bitcoin-stands-alone https://unchained-capital.com/blog/author/plewis/ https://unchained-capital.com/blog/bitcoin-is-antifragile/ https://voskuil.org/cryptoeconomics/ https://www.amazon.com/Thank-God-Bitcoin-Corruption-Redemption/dp/1641991216/ref=sr_1_1?dchild=1&keywords=thank+god+for+bitcoin&qid=1607152012&sr=8-1 https://www.citadel21.com/bitcoin-is-the-first-global-market-for-electricity-and-will-unleash-renewables https://www.coincenter.org/how-bitcoin-could-drive-the-clean-energy-revolution/ https://www.coindesk.com/bitcoin-mining-wastes-energy-thats-good-thing https://www.coindesk.com/frustrating-maddening-all-consuming-bitcoin-energy-debate https://www.coindesk.com/the-last-word-on-bitcoins-energy-consumption https://www.coindesk.com/what-bloomberg-gets-wrong-about-bitcoins-climate-footprint https://www.fidelitydigitalassets.com/articles/addressing-bitcoin-criticisms https://www.fidelitydigitalassets.com/articles/addressing-bitcoin-criticisms https://www.fidelitydigitalassets.com/articles/addressing-bitcoin-criticisms https://www.fidelitydigitalassets.com/articles/addressing-bitcoin-criticisms https://www.fidelitydigitalassets.com/articles/addressing-bitcoin-criticisms https://www.forbes.com/sites/rogerhuang/2020/12/21/heres-why-quantum-computing-will-not-break-cryptocurrencies/?sh=24a95b3c167b https://www.lopp.net/bitcoin-information/getting-started.html https://www.lopp.net/bitcoin-information/getting-started.html#misconceptions https://www.lynalden.com/author/penumbra/ https://www.lynalden.com/bitcoin-ponzi-scheme/ https://www.lynalden.com/bitcoin-security-modeling/ https://www.lynalden.com/bitcoins-network-effect/ https://www.lynalden.com/ethereum-analysis/ https://www.lynalden.com/fraying-petrodollar-system/ https://www.lynalden.com/misconceptions-about-bitcoin/ https://www.lynalden.com/misconceptions-about-bitcoin/ https://www.lynalden.com/misconceptions-about-bitcoin/ https://www.lynalden.com/misconceptions-about-bitcoin/ https://www.pairagraph.com/dialogue/9f0dfc47978740ebb0439efaadf23f41/2 https://www.swanbitcoin.com/bitcoins-energy-usage-is-not-a-problem-heres-why-by-lyn-alden/ https://www.theblockcrypto.com/post/11481/asic-chip-technology-is-renewable-energys-future https://www.truthcoin.info/blog/measuring-decentralization/ https://www.truthcoin.info/blog/pos-still-pointless/ https://www.truthcoin.info/blog/pow-and-mining https://www.truthcoin.info/blog/pow-cheapest/ https://www.upstreamdata.ca/post/natural-gas-venting-how-bitcoin-solved-a-160-year-old-problem https://youtu.be/ns_I2LpeAbQ?t=26084

Find only duplicates:

wget -q https://endthefud.org/ -O - | tr "\t\r\n'" ' "' | grep -i -o '<a[^>]\+href[ ]*=[ \t]*"\(ht\|f\)tps\?:[^"]\+"' | sed -e 's/^.*"\([^"]\+\)".*$/\1/g' | sort | uniq -d

Output:

http://btcco2.com http://btcco2.org https://blog.lopp.net/are-chinese-miners-threat-bitcoin/ https://github.com/pox/EndTheFUD http://shitcoin.review https://saifedean.com/book/ https://www.fidelitydigitalassets.com/articles/addressing-bitcoin-criticisms https://www.lynalden.com/misconceptions-about-bitcoin/

[pox]

@9mido looking at the articles in the output above, AFAICS they all appear with the same exactly link when they appear multiple times, which is fine (browser shows visited links in different color). Did I miss something?

[9mido]

Not all websites change the color of visited links though. Google search results seem to do this, but when I visit links on endthefud.org and a few other websites, the visited links do not always change colors when you return back.

Links appearing multiple times on the same page is not good for SEO. As a user who was going through links on endthefud.org, I found myself clicking on the same link again and got frustrated because I already read that article.

[pox]

If visited links don't change colors then it's definitely a CSS problem. I'll get on it.

[9mido]

Ok the CSS is one thing for previously visited links but another CSS thing for coloring duplicate links. If coloring links, have a color key somewhere on the page that tells everybody that this colored link is a duplicate and this colored link is previously visited.

If you want more traffic to endthefud.org from search engine results, it would be best to get rid of duplicate links so that endthefud.org can rank higher on search engines.

I think the whole page should be better organized. Get rid of duplicate links. Show the link only once. But show all categories under that one and only link. Don't show the same link with different anchor tags either.

Maybe try something like this:

Link Energy, Governments, Etc.

What you could add also is some sort of filtering mechanism for each category. For example, if I filter via a dropdown menu or checkbox by Energy then all links that have the Energy category will be returned. If you filter by Energy and Governments then only links with Energy and Governments combined will be returned.

[pox]

Ok the CSS is one thing for previously visited links but another CSS thing for coloring duplicate links. If coloring links, have a color key somewhere on the page that tells everybody that this colored link is a duplicate and this colored link is previously visited.

Once a link is visited, all its duplicates should appear as visited links in the page.

If you want more traffic to endthefud.org from search engine results, it would be best to get rid of duplicate links so that endthefud.org can rank higher on search engines.

Honestly I don't really care about SEO. The organization of the website is meant to help people easily find all the relevant articles by topic. I don't want people to need to scan a long list of articles of unrelated topics just to tease out articles for a specific one. That's why I think duplicate links are fine. If it hurts SEO so be it. I'm optimizing for this page to be used as a bookmark collection basically where the lookup is by topic, not by article, not by author, etc.

What you could add also is some sort of filtering mechanism for each category. For example, if I filter via a dropdown menu or checkbox by Energy then all links that have the Energy category will be returned. If you filter by Energy and Governments then only links with Energy and Governments combined will be returned.

There are some guys working on a twitter bot like this. Where you can mention the bot and one or more tags and you get relevant articles in a response. It's a nice concept. Doing it on a statically generated website like this isn't trivial, though. Probably requires some custom JS and I'm not sure if that jibes well with Github Pages to be honest. If someone can show me a proof-of-concept page that does this using Github Pages I'll be happy to consider adding tag filtering to endthefud.

[pox]

I've updated the CSS to show visited links in purple. This ameliorates the problem somewhat, although some links (especially those opening in Medium, I've noticed) still don't get colored properly on my mobile device. Maybe it's because they're opened by the Medium app instead of the browser itself... if someone knows how to fix this I'd be eager to learn.

[9mido]

What about putting something like an * or something like (duplicate) next to each duplicate link then to indicate that it is a duplicate? Or just color the duplicate links a different color than unique links?

The only thing I don't like about the CSS purple links is that once you visit endthefud.org again after you clicked a few of the links, those links are still purple and they do not reset themselves. So basically we are back to the original problem again of not knowing which links are duplicate. If you click all of the links on endthefud.org and return, there is no way of knowing which ones are duplicate. I think it would be better to reverse the purple CSS link change you made and label the duplicate links in a different way.