search

riderkick / FMD

Forked from https://sf.net/p/fmd/
GNU General Public License v2.0
716 stars 212 forks source link

How are the sites URLs scraped? #1555

Open pmwoodward3 opened 5 years ago

pmwoodward3 commented 5 years ago

I see that each site has a .db file (sqllite) that contains all the URLs and some basic metadata. My question is, how is that collected and put into these .db files. I would love to see the script that scrapes this data and organizes the metadata.

Anyone have any ideas?

Thanks

Tmp341 commented 5 years ago

Probably this.