osc is slow

[bmwiedemann]

Describe the bug When working on my projects (Slowroll, reproducible builds), I often run many osc calls on thousands of packages. Since this runs in the PRG2 DC, latency is not limiting speed, but I found that osc takes around 6x longer than the equivalent curl call. The time output suggests, it takes approx 520ms of extra CPU time to process something.

Versions

• Operating system: (source /etc/os-release; echo $PRETTY_NAME) openSUSE Leap 15.5
• Packages: rpm -q osc osc-1.9.2-150500.423.1.noarch

To Reproduce Steps to reproduce the behavior:

1. setup ~/.netrc with machine api.opensuse.org login USER password PASS
2. run curl + osc to compare how long they take:

   time curl -s -n https://api.opensuse.org/source/openSUSE:Factory/zziplib >/dev/null
   time osc api /source/openSUSE:Factory/zziplib >/dev/null

Expected behavior osc should be closer to curl's speed

Screenshots, console outputs If applicable, add screenshots or console outputs to help explain your problem.

> time curl -s -n https://api.opensuse.org/source/openSUSE:Factory/zziplib >/dev/null

real    0m0.104s
user    0m0.024s
sys     0m0.000s

> time osc api /source/openSUSE:Factory/zziplib >/dev/null

real    0m0.628s
user    0m0.550s
sys     0m0.049s

Additional context For Slowroll, I check, delete and update approx 2000 packages each month, which might be 5k to 10k API calls. Would be good, if those would take only 1h instead of 3h or such.

[bmwiedemann]

Without authentication, the difference is even larger:

time curl -s https://api.opensuse.org/public/source/openSUSE:Factory/zziplib >/dev/null

real    0m0.042s
user    0m0.015s
sys     0m0.004s

[dmach]

The problem starts with loading the python modules:

time python3 -c "import osc.commandline"

real    0m0.245s
user    0m0.199s
sys     0m0.043s

time python3 -c "import osc.core"

real    0m0.239s
user    0m0.190s
sys     0m0.048s

I already made several improvements in the latest version (1.10.0), but they only scatch the surface :(

Also, running osc on python 3.6 seems to be way slower than using more recent python versions.

[dmach]

It turns out, that argparse spends some time on translations. Injecting the following code to osc reduces osc --help execution time from 560ms to 530ms on my machine.

import argparse

def _(msgid):
    return msgid

argparse._ = _
argparse.ngettext = _

Let me keep digging further...

[dmach]

Also caching formatter might help in some cases:

class OscArgumentParser(argparse.ArgumentParser):
    def _get_formatter(self):
        if not hasattr(self, "_formatter"):
            self._formatter = self.formatter_class(prog=self.prog)
        return self._formatter

[dmach]

FTR: the benchmark command I'm currently running:

hyperfine --ignore-failure --runs=20 'osc --help'