vipyrsec / dragonfly-mainframe

The mainframe for Dragonfly
https://docs.vipyrsec.com/dragonfly-mainframe/
MIT License
5 stars 1 forks source link

Optimize batch queue #151

Closed Robin5605 closed 1 year ago

Robin5605 commented 1 year ago

Optimize the batch queue endpoint so that we deduplicate before fetching package metadata from PyPI. This should result in a drastic performance increase from before, where we were sending 100 requests to PyPI every minute.