Closed gridhead closed 8 months ago
Here's the first run-through with regular pauses when several requests have been made.
I plan on getting rid of these safety nets now to see if the Python bindings for GitLab take care of the HTTP exceptions by itself - especially the one that arise due to rate limits and transient errors.
So it seems that python-gitlab
does work to protect from any process abortion due to rate limiting and transient 5XX errors.
Time to take it up a notch and get rid of all requests
involvements that I have with GitLab.
EDIT - This import run took 28 minutes and 12 seconds to complete.
I moved the stat
-ing scripts from requests
to python-gitlab
and from what I see, it looks good. I would be interested to see if I can create one GitLab object during the initial phases of the program execution and then, reuse that same session later instead of creating one down the line. Here's the logs from the same and this time around, this run took 33 minutes and 34 seconds to complete.
The sharing of the same GitLab object worked and this run took 20 minutes and 21 seconds to complete.
This looks great! :tada: Nice work @gridhead. I will be able to test this out once it is merged and cut into a new release. The easiest way for me to run this would be with a new RPM package update sent to the Fedora repositories, but I could try something out with a new Pip package update too. I would prefer the RPM package update though.
The code has been restored to its former glory.
---------- coverage: platform linux, python 3.11.7-final-0 -----------
Name Stmts Miss Cover Missing
----------------------------------------------------------------
pagure_exporter/__init__.py 2 0 100%
pagure_exporter/conf/__init__.py 0 0 100%
pagure_exporter/conf/standard.py 77 0 100%
pagure_exporter/main.py 51 0 100%
pagure_exporter/view/__init__.py 0 0 100%
pagure_exporter/view/dcrt.py 19 0 100%
pagure_exporter/view/misc.py 9 0 100%
pagure_exporter/view/repo.py 48 0 100%
pagure_exporter/view/stat.py 40 0 100%
pagure_exporter/view/tkts.py 86 0 100%
pagure_exporter/work/__init__.py 0 0 100%
pagure_exporter/work/keep.py 21 0 100%
pagure_exporter/work/repo.py 74 0 100%
pagure_exporter/work/stat.py 38 0 100%
pagure_exporter/work/tkts.py 133 0 100%
----------------------------------------------------------------
TOTAL 598 0 100%
Coverage HTML written to dir htmlcov
Coverage XML written to file coverage.xml
Current things to do
requests
implementation of GitLab access
This should fix #60.