search

sethmlarson / pypi-data

Data about packages and maintainers on PyPI
Apache License 2.0
122 stars 8 forks source link

Batch insert would likely be many times faster #31

Closed jonathan-s closed 3 months ago

jonathan-s commented 4 months ago

~Right now you're writing each entry to the database one by one. Though if you were to write the data to disk into a csv first or similar. Then insert everything into the db it would go many times faster.~

Though I guess it could also be the requests that is the bottleneck. Seems like it is indeed the requests being the bottleneck.