python / cpython

The Python programming language
https://www.python.org
Other
63.11k stars 30.22k forks source link

during 'make install', pre-existing site-packages residents are recompiled. Twice. #72071

Open 1812807c-1832-40a0-91fc-6c391d0d1d1e opened 8 years ago

1812807c-1832-40a0-91fc-6c391d0d1d1e commented 8 years ago
BPO 27884
Nosy @bitdancer, @MattDMo

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['extension-modules', 'build', 'expert-installation'] title = "during 'make install', pre-existing site-packages residents are recompiled. Twice." updated_at = user = 'https://github.com/MattDMo' ``` bugs.python.org fields: ```python activity = actor = 'r.david.murray' assignee = 'none' closed = False closed_date = None closer = None components = ['Build', 'Extension Modules', 'Installation'] creation = creator = 'MattDMo' dependencies = [] files = [] hgrepos = [] issue_num = 27884 keywords = [] message_count = 2.0 messages = ['273824', '273825'] nosy_count = 2.0 nosy_names = ['r.david.murray', 'MattDMo'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = None url = 'https://bugs.python.org/issue27884' versions = ['Python 3.6'] ```

1812807c-1832-40a0-91fc-6c391d0d1d1e commented 8 years ago

I've been building Python 3 from source on Linux for a long time (at least since 3.2 days), and I honestly can't remember if this has happened before 3.6 or not, which is why I'm only tagging this 3.6.

Basically, when running 'make install' (after building either static or shared, I haven't been messing around with any other options except install prefix - $HOME) you'll start seeing messages like:

\===== Listing '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/init.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/anova_nistcertified.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/ar_panel.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/example_kernridge.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/gmm.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/kernridgeregress_class.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/ols_anova_original.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/onewaygls.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/penalized.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/predstd.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/runmnl.py'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/sympy_diff.py'... Listing '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/tests'... Compiling '/home/mattdmo/lib/python3.6/site-packages/statsmodels/sandbox/regression/tests/init.py'... \=====

and on and on and on. I usually don't pay much attention to them, but since I have nothing better to do I just stare idly at them and watch as all my third-party modules in 'site-packages' scroll by in alphabetical order very quickly. For the last few builds of 3.6, though, I've noticed that the modules get gone through twice. I have a pretty large number of modules (341, according to ls | wc -l), so it takes a while to go through them, and they're definitely being repeated.

Unfortunately, I don't know a whole lot about Makefiles or the internals of the installation process, so I don't have the slightest idea of where to begin even looking for the bug. But, I bet you ladies and gentlemen do :)

If it's necessary, I can rebuild and pipe all the output into a file for you to examine. Just let me know.

bitdancer commented 8 years ago

As far as I can tell, this is working as designed, and in fact you ought to see site-packages getting compiled *three* times (normal, -O, and -OO).

I wonder if this is a holdover from when we didn't have per-interpreter pyc files, and its just that no one noticed the need for recompile of site-packages didn't exist any more. On the other hand, I may be missing some piece of logic that means that it is still needed.