Calls to collect/index_packages API endpoint takes too much time and often timeout

nexB / purldb

Tools to create and expose a database of purls (Package URLs). This project is sponsored by NLnet project https://nlnet.nl/project/vulnerabilitydatabase/ and nexB for https://www.aboutcode.org/ Chat is at https://gitter.im/aboutcode-org/discuss

https://purldb.readthedocs.io/

29 stars 20 forks source link

Calls to collect/index_packages API endpoint takes too much time and often timeout #493

Open pombredanne opened 2 weeks ago

pombredanne commented 2 weeks ago

Calls to collect/index_packages API endpoint takes too much time and often timeout. We should instead return immediately and do all the bidding in a background task including:

collect versions (one, all or vers)
collect source packages
then queue these for actual data collection

pombredanne commented 1 day ago

Same for the plain collect pkg:maven/com.amazonaws/aws-java-sdk-kms@1.11.277 for example. We should NOT do the purl2vcs call in the request as this is impractically slow sometimes but only in the background.