scanoss / engine

SCANOSS Open Source Inventory Engine
GNU General Public License v2.0
34 stars 20 forks source link

query very slow #25

Closed tiandiyixian closed 2 years ago

tiandiyixian commented 2 years ago

Hello, there is a question. With the increase of ldb data, the query becomes slower . Now the database has about 600GB of data, and it takes a long time to query a snippet For example, this is the query result "id": "snippet", "status": "pending", "lines": "1-49", "oss_lines": "13-61", "matched": "97%", "purl": [ "pkg:github/v2v3v4/BigWorld-Engine-2.0.1" ], "quality": [], "cryptography": [], "server": { "hostname": "localhost.localdomain", "version": "4.3.8", "flags": "0", "elapsed": "57.364302s" }, When there are many query files, it will take a few days to get a result. My machine configuration is AMD Ryzen Threadripper 3970X 32-Core Processor, 128Gb memory, 30Tb mechanical hard disk, Centos OS 7.8, what is the best way to find it? Speeding up, because I noticed that the system resources were used very little during the actual query. I don't know what went wrong. I want to ask if there are any good suggestions. This is the result of executing scanoss -t: Analysis complete Test executed in 401967ms Average file scanning time is 4019ms Performance is 24 fingerprints per second

mscasso-scanoss commented 2 years ago

Dear @tiandiyixian, You are having a really bad performance in your hard disk. Take this scanoss -t values as reference:

So, it is difficult give you some extra advice. You should improve your hard disk performance, a lot.

tiandiyixian commented 2 years ago

Changed from hdd to ssd, the problem is solved, thank you!