search

Azure / azure-cli

Azure Command-Line Interface
MIT License
4.01k stars 2.98k forks source link

Improve embedded python speed #25106

Open bebound opened 1 year ago

bebound commented 1 year ago

Is your feature request related to a problem? Please describe.

CLI executes with embedded Python interpreter on Windows.

Compare with my dev environment, each command is noticeably slower. For example: Measure-Command {az account -h} takes 763ms and 342ms in MSI and dev env respectively.

Uncompress python3.10.zip?

I think uncompressing zip makes python run faster, but I can't get a stable result to compare the difference. PEP273 provide a 20-year-old benchmark, does the modern hardware widen the gap?

After some research, I find a difference between standard Python and embedded Python:

The standard library is included as pre-compiled and optimized .pyc files in a ZIP -- The embeddable package

If build-in packages is a zip file, Python need to uncompress it before import statement. This makes sense. To verify this, if C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python310.zip is remove, run python.exe raises this:

╰─ ./python.exe                                                                                                      ─╯
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe'
  sys.base_prefix = ''
  sys.base_exec_prefix = ''
  sys.platlibdir = 'lib'
  sys.executable = 'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe'
  sys.prefix = ''
  sys.exec_prefix = ''
  sys.path = [
    'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python310.zip',
    '.\\DLLs',
    '.\\lib',
    'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

We can extract python3.10.zip to python3.10.zip/Lib folder to save the uncompression time. In standard Python, there files are in Lib folder, we can also uncompress it there. After this operation, Measure-Command {az account -h} takes 684ms, which is reduced by 10%. It is still slower than dev env. I'll keep working on it.

Use 64bit embedded python?

In my dev env, I use 64bit Python, so I also try install CLI on 64bit embedded version. It's about 100ms faster than 32bit. But migration is not easy and it's a breaking change. https://github.com/Azure/azure-cli/pull/24607

We've released the 64-bit package in 2.51.

Convert packages to zip package?

Some packages consist of many files. Compression it to a zip file may speed up the load time.

There is another way to uncompress python3.10.zip, but it's too complex. I'd record it here. Create a python310._pth in CLI folder with this content and put its content into python310 folder. (ref:_pth files - Python). Please mind that after add python310._pth, the current folder is not in sys.path in embedded Python. (https://github.com/python/cpython/issues/77192)

C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python310
import site

PS: Add name.pth file in sys.prefix does not work because site package is not found when python310.zip is missing (site — Site-specific configuration hook).

yonzhan commented 1 year ago

Improve embedded python speed