search

scanoss / minr

SCANOSS Mining tool
23 stars 11 forks source link

[Question] Why minr import so slow on if ldb on NFS? #35

Closed mengzhuo closed 1 year ago

mengzhuo commented 1 year ago

We are importing 1000+ random code base from Github and takes 2 days to complete. Both CPU and IO are less than 10%, we can comfirm that NFS io limit is 500MiBps, but minr import only runs about 2 MiBps.

BTW, I'm curious about how scanoss deployed and hardware requirement, since our mine takes too long.

mscasso-scanoss commented 1 year ago

Hello @mengzhuo,

I apologize for the delayed response. The current version of "minr" operates on a single thread, which accounts for the observed poor performance and limited resource utilization. A new version of "minr" is set to be released soon. Furthermore, the "import" functionality has been moved from "minr" to "ldb." You can already test this feature in the beta release: https://github.com/scanoss/ldb/releases/tag/v4.0.0_beta

Feel free to share any additional feedback or comments you may have.