search

lindylearn / aboutideasnow

Find people to talk to or collaborate with by searching across the /about, /ideas and /now pages of 1000s of personal websites.
https://aboutideasnow.com
MIT License
192 stars 6 forks source link

Scraping Privacy Policy #13

Open mdrews93 opened 6 months ago

mdrews93 commented 6 months ago

Is the database strictly opt-in, where only sites submitted by their creators through the form are indexed in the database?

Are any sites scraped and indexed without asking the site creator for their consent to be included in this search tool? I ask because I have a web space with a /now on a domain that has many other personal sites with /now pages but I do not want my pages to be indexed.

Edit to add: came across this post on hacker news

we built a simple site that indexes 7k+ personal sites [0]

[0] gathered from: 1) https://nownownow.com/ and similar sites 2) checking all HN posts since 2020 with more than 100 upvotes

which similar sites were scraped? which HN posts survived the 100+ up votes since 2020 filter?