search

apache / incubator-devlake

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
https://devlake.apache.org/
Apache License 2.0
2.55k stars 508 forks source link

[Bug][Go-Git] Libgit2 has a better performance then go-git #7202

Open Geni2 opened 5 months ago

Geni2 commented 5 months ago

Search before asking

What happened

I'm using the developer setup of DevLake and while collecting the data for the same project I noticed that by using libgit2 it takes 3 minutes and with go-git, it takes ~43 minutes this is an outrageous performance penalty.

What do you expect to happen

There shouldn't be this outrageous difference. Because it slowest the process by almost 20 times.slowed

How to reproduce

Go to the .env (env.example) file and change the value of this variable USE_GO_GIT_IN_GIT_EXTRACTOR=false Collect the data for the same project and see the time duration when you use go-git and libgit2

Anything else

No response

Version

dev

Are you willing to submit PR?

Code of Conduct

klesh commented 5 months ago

Yeah, it is too much... We were hoping the go-git could help with Incremental Data Collection and Canceling during Clone phase which doesn't seem to be possible with libgit2

d4x1 commented 5 months ago

You're right, and we have to delay switching to go-git.
Since DevLake reads the repo, if go-git can be thread safe when reading, we can test then and see if we can switch to go-git.

BTW, go-git is about 6 time slower than libgit2 in my test case.

Geni2 commented 5 months ago

Even though go-git may provide some features that libgit2 doesn't, I agree with @d4x1 that it's better to keep it on testing and see if the time difference can be improved in go-git.

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.