search

repology / repology-updater

Repology backend service to update repository and package data
https://repology.org
GNU General Public License v3.0
501 stars 176 forks source link

Fetch and store project homepages #203

Open AMDmi3 opened 7 years ago

AMDmi3 commented 7 years ago

As long as we have enough diskspace, we could be an archive.org for F/OSS projects. It's also useful to monitor homepage changes and do some data mining (for @Donatallo project, for instance)

AMDmi3 commented 6 years ago

.oO( Should we honor robots.txt if we implement this? )

AMDmi3 commented 6 years ago

Or rather talk with archive.org people about submitting a list of project homepages to them to process