Slow import time

[LewisGaul]

The python_on_whales package is a bit slow to import (noticeable for scripts when you just want to show help output):

   > python -V
Python 3.12.1

   > echo 'import python_on_whales' > /tmp/script.py

   > time python3 /tmp/script.py

real    0m0.629s
user    0m0.569s
sys     0m0.057s

   > python -m cProfile /tmp/script.py | head
         1438040 function calls (1220502 primitive calls) in 1.141 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       20    0.000    0.000    1.881    0.094 cli_wrapper.py:1(<module>)
       45    0.001    0.000    1.531    0.034 __init__.py:1(<module>)
    468/1    0.017    0.000    1.141    1.141 {built-in method builtins.exec}
        1    0.000    0.000    1.141    1.141 script.py:1(<module>)
   3502/1    0.005    0.000    1.141    1.141 <frozen importlib._bootstrap>:1349(_find_and_load)

[LewisGaul]

Related to https://github.com/gabrieldemarmiesse/python-on-whales/issues/512 since importing these extra dependencies seems to be contributing around 25% of the time.

[gabrieldemarmiesse]

Thanks for the bug report, I reproduced it with tuna and -X importtime, and here are the results I found:

• Importing python-on-whales takes around 480ms
• Importing the download_cli module takes around 118ms, it will be fixed when #512 is fixed.
• The majority of the time left (80%) is used to create the pydantic models, see https://github.com/pydantic/pydantic/issues/6768

There is not much we can do about the pydantic models. We can provide solutions to users:

1) Users who are making CLI tools or other tools that are very sensitive to import times should use lazy import tools such as https://pypi.org/project/lazy-imports/ . I have used such tools in the past and found them very performant and easy to use. 2) Mojo is turning out to solve a bunch of issue that are present in python. For example, since it's a compiled language, the import times only impact the compilation speed and not the runtime speed. Once Mojo is mature enough, I would be very happy to make python-on-whales compatible. Users should be able to then make CLI tools that start very fast and are not impacted by import times by using the Mojo compiler.

Since there is not much we can do except fixing #512 and then much later (1+ year) provide a Mojo package, I would advise to close this issue.

[LewisGaul]

I think we should keep this open as import time can be important and I'd say it's currently too high, this is still currently a major pain point for me.

Following https://github.com/gabrieldemarmiesse/python-on-whales/issues/512#issuecomment-2137956659 would you be happy with removing the dependencies that account for ~20% of the time or prefer to wait a bit longer?

It might also be worth seeing if we can speed up the pydantic stuff, e.g. see https://docs.pydantic.dev/latest/concepts/performance/.

[gabrieldemarmiesse]

I think we can start removing the typer/tqdm dependency. Installing the docker client is documented well enough on the internet that we don't have to do it for our users.