leopart-hq / leopart

This is a search engine for publicly available KiCad files on GitHub. The KiCad files are parsed and the contained components indexed to make the projects searchable by the components they contain. This project was started as a university lab course and is now an open source project.
https://leopart.org
Other
8 stars 1 forks source link

Improve GitHub crawling #3

Open LasseMoench opened 4 years ago

LasseMoench commented 4 years ago

The GitHub crawler can probably be improved. It currently only searches for the words 'pcb' or 'KiCad' in the readmes of all repositories, and then searches each repo that was found for kicad pcb files. It still doesn't seem to find all repositories that match these criteria, see Issue #1. It might be wise to look into the GitHub API documentation again and determine better ways to find pcb files. GitHub keeps an internal score on how computationally intensive API requests are, so attention has to be paid to the amount of abuse rate limits that are caused by this.