Windows pyd slower when not loaded via load_dynamic

[fdc08b81-f10f-4d56-a4c0-de5650414d13]

BPO	31555
Nosy	@terryjreedy, @pfmoore, @tjguk, @skrah, @zware, @zooba, @Safihre

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['extension-modules', 'OS-windows', 'performance'] title = 'Windows pyd slower when not loaded via load_dynamic' updated_at = user = 'https://github.com/Safihre' ``` bugs.python.org fields: ```python activity = actor = 'steve.dower' assignee = 'none' closed = True closed_date = closer = 'steve.dower' components = ['Extension Modules', 'Windows'] creation = creator = 'Safihre' dependencies = [] files = [] hgrepos = [] issue_num = 31555 keywords = [] message_count = 9.0 messages = ['302768', '303363', '303370', '303394', '303412', '303414', '303435', '303459', '303462'] nosy_count = 7.0 nosy_names = ['terry.reedy', 'paul.moore', 'tim.golden', 'skrah', 'zach.ware', 'steve.dower', 'Safihre'] pr_nums = [] priority = 'normal' resolution = 'third party' stage = 'resolved' status = 'closed' superseder = None type = 'performance' url = 'https://bugs.python.org/issue31555' versions = ['Python 2.7'] ```

[fdc08b81-f10f-4d56-a4c0-de5650414d13]

I have a Python 2.7 C-extension that was made for performance, however, I noticed something strange on Windows: When I run locally python setup.py install and it installs it the "egg" way, the performance of calling the function in the module 500x in a loop is:

 yEnc C New  took   579 ms
 yEnc C New  took   580 ms
 yEnc C New  took   580 ms

But when I install it as a wheel locally or via pip without a single change to the C-code, this is the result:

 yEnc C New  took   702 ms
 yEnc C New  took   694 ms
 yEnc C New  took   691 ms

That's a 10-15% difference. I also checked macOS and Ubuntu, here it does not seem to be the case. By investigating (https://github.com/pypa/setuptools/issues/1154) I found that the difference is only that the "egg" way the module gets loaded via imp.load_dynamic

I cannot test under Python 3, since the module is (not yet) converted to Python 3-style.

To reproduce run and look at the yEnc C New score between the 2 ways (on Windows only of course):

git clone https://github.com/sabnzbd/sabyenc.git cd sabyenc python setup.py install python .\tests\speed_compare.py; python .\tests\speed_compare.py;python .\tests\speed_compare.py; pip uninstall sabyenc -y

pip install sabyenc # Or this: # python setup.py install bdist_wheel # pip install .\dist\sabyenc-3.3.1-cp27-cp27m-win_amd64.whl python .\tests\speed_compare.py; python .\tests\speed_compare.py;python .\tests\speed_compare.py;

[terryjreedy]

This issue is more likely to get attention if there is a difference in 3.6, or even better, 3.7.

[zooba]

The only difference between your two tests is the install tool, and neither setuptools nor wheel bugs are tracked here.

[fdc08b81-f10f-4d56-a4c0-de5650414d13]

No, the difference is not wheel vs setuptools. They both generate the identical pyd file. The difference is python: if the module is loaded by just being available on the path or if the module is loaded via imp.load_dynamic

I understand if it's closed because it's not python 3, but it's not external tool related.

[5531d0d8-2a9c-46ba-8b8b-ef76132a492c]

Does the difference stay at 10-15% if you run it 50000 times or is it a one time cost of around 100 ms?

[fdc08b81-f10f-4d56-a4c0-de5650414d13]

Very good question!

5000 times via imp.load_dynamic: yEnc C New took 5870 ms yEnc C New took 5878 ms yEnc C New took 5835 ms 5000 times via "pip: having the .pyd in site-packages" yEnc C New took 6489 ms yEnc C New took 6369 ms yEnc C New took 6390 ms

Difference \~450 ms

10000 times via imp.load_dynamic: yEnc C New took 11775 ms yEnc C New took 11720 ms yEnc C New took 11695 ms

10000 times via "pip: having the .pyd in site-packages" yEnc C New took 12338 ms yEnc C New took 12281 ms yEnc C New took 12345 ms

Difference \~500 ms

10000 times via imp.load_dynamic: yEnc C New took 23431 ms yEnc C New took 23406 ms yEnc C New took 23283 ms

10000 times via "pip: having the .pyd in site-packages" yEnc C New took 24401 ms yEnc C New took 24177 ms yEnc C New took 24482 ms

Difference \~1100 ms

So not very linearly scaling but still increasing. Odd.

[zooba]

The difference is on startup in how they generate sys.path entries.

Skip the install step and directly run the library in a clean virtual environment, so that no .pth or .egg files are in use. If the difference still occurs, show that it also occurs on a recent version of Python as well and we can look at it

I promise you, there is only one way that extension modules get loaded. The performance difference is not because of anything in core.

[fdc08b81-f10f-4d56-a4c0-de5650414d13]

If you know the problem, would you also know the solution? What is the difference in how they generate sys.path entries and how does this affect performance during execution of the module function? Just want to understand what is going on :)

[zooba]

I don't really care, since you're comparing multiple unsupported technologies, but my guess is that the egg install injects the library much earlier on sys.path, which avoids most of the relatively expensive file system scan.