Closed 8465231 closed 3 years ago
Justintime50
commented
3 years ago I'm not aware of GitHub pull limits, can you link so some documentation about this?
I'm able to clone/pull upwards of 100 repos with this script; however, it should work for much more than that. If a repo has already been cloned, it will skip cloning it again. For pulls, if there have been changes since, it will pull those new changes, otherwise it will state there are no new changes.
8465231
commented
3 years ago Ok, first I think I remembered wrong, the new rate limits were for docker hub it seems (just remember seeing something about it late last year). Sorry about that.
Secondly the github limits are here: https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting
I am thinking the issue could be how the old script I was using interacted, I get the feeling it was making unnecessary requests (checking the repos, listing out what was going to be downloaded etc) combined with no using an account and using up the 5k request/hour much faster then it should.
I only ran into this issue on very large projects.
Good to know it will only download changes, exactly what I want! I will hopefully have time to try it next week.
Justintime50
commented
3 years ago Sounds great! Yeah for an authenticated user you can make 5000 API requests an hour. This script only makes 1-3 API calls, the rest are just normal git clone/pull operations and you shouldn't run into troubles with large or many projects.
I'll close this issue for now. Best of luck!
8465231
commented
3 years ago Ok, sorry again. This is purely user error I know. But I don't have a clue how to use this script. I only have basic linux/bash experience,, I am talking been using linux for ~6 months level of experience.
Never messed with python and "pip3" is not an installable package nor can I find any details on how to use it to install something.
What are the complete steps to install / set this up on preferably alpine or ubuntu from a blank slate fresh install (aka, docker)?
It would be really cool if this was available in docker form for noobs like me ;-)
If you don't have time to help me, that is ok, I am sure I can find something else that will work. Thanks for your work on this.
Justintime50
commented
3 years ago All the usage instructions should be in the README. pip3 may not be available depending on your platform, try simply pip. If you don't have Python or Pip installed, that falls outside the scope of the help I'd provide on this project since that's a prerequisite to using GitHub Archive.
There are a plethora of guides out there on the subject, here is just one that could help get you started: https://docs.python-guide.org/starting/install3/linux/.
Docker would be a nice feature. Feel free to open a new issue and/or sponsor the project and I may look into adding that feature. If you'd like to do it yourself, simply build a Python Docker image, copy the project files to it, and add the main cli.py file as the entrypoint.
8465231
commented
3 years ago Ok, thanks for your time. I have python installed but any of those commands just gives an error. I obviously don't have a clue what I am doing.
Oh well, if you ever make a docker I would be all over this.
Forgive me if this is the wrong way of asking a question, brand new to Github.
I have been using a script to download github projects but with the new pull limits that github implemented I have 2 issues.
1: It will fail to download all the repos for a project (lineageOS or signal for example)
2: It will not let me know that it failed or where it failed (aka, was rejected).
Does this script have a way of dealing with this issue? Can it pick up where it left off? Or set a limit of the number of pulls to do at a time and then wait 6 hours to do more so as to not get cut off?
I am not working with all the projects right away, I am archiving a lot of them as a backup so I might not know that something is missing for some time and it could be a real bummer if it was found missing later.
Planning on setting this script up in a docker when I have some time next week but figured I would ask before I put too much time into it.