Closed danielhoherd closed 5 years ago
I think this is happening because of the gitpython
module. Do you have any thoughts why it could be happening?
I am not sure why yet, but I am looking into it. I have a fork/branch here with some extra logging where I am investigating: https://github.com/danielhoherd/GithubCloner/tree/logging
Run it with export LOGLEVEL=DEBUG ; githubcloner.py
It looks like the problems are happening in git. I don't know if the root cause is in git or in the threading. For some unknown reason I cannot get GIT_PYTHON_TRACE to work, but I suspect it's because of threading.
I believe the same, maybe let's try to check it via a separate process on each thread?
I am testing the issue again, I will let you know the results.
Hi @danielhoherd
As discussed on https://github.com/mazen160/GithubCloner/issues/9, the issue is occuring because of git. It's not githubcloner related, probably the best fix it to slow down cloning to 1 thread at github cloner + to report the issue to git devs.
Closing issue because the above answer is sufficient.
I have seen several instances of inconsistent behavior when cloning the same user or org. I observed this while iterating on the API rate limit exceeded and the incomplete repository issues.
Here is the cycle I am using to test iterations:
I chose mazen160 for this example because it is a public data set within the author's control, but I have observed it with both organizations and other users.
When iterating with the above 7 times, I observed these results: