IDLE: Remove '', user_dir, and idlelib from sys.path when added

[terryjreedy]

BPO	25488
Nosy	@terryjreedy, @roseman
Files	idle-start.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/terryjreedy' closed_at = None created_at = labels = ['expert-IDLE', 'type-bug', '3.9'] title = "IDLE: Remove '', user_dir, and idlelib from sys.path when added" updated_at = user = 'https://github.com/terryjreedy' ``` bugs.python.org fields: ```python activity = actor = 'terry.reedy' assignee = 'terry.reedy' closed = False closed_date = None closer = None components = ['IDLE'] creation = creator = 'terry.reedy' dependencies = [] files = ['42041'] hgrepos = [] issue_num = 25488 keywords = [] message_count = 11.0 messages = ['253525', '253527', '260952', '260961', '260962', '277743', '277744', '277746', '286891', '286892', '296366'] nosy_count = 3.0 nosy_names = ['terry.reedy', 'markroseman', 'python-dev'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue25488' versions = ['Python 3.9'] ```

[terryjreedy]

Problem: Consider module idlelib.run. The following is correct.

C:\Users\Terry>python -c "import run"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'run'

But in IDLE started from icon or command line
>>> import run
>>> 

With respect to the goal of developing code that can run outside of IDLE, not raising ImportError is a bug. Suppose a user erroneously writes 'import run' or 'import help' (in repository IDLE). The error could be that the file is 'runner' or 'helper' or that 'run' or 'help' is not in the same directory. The import should fail rather than the user getting a error later because the wrong file was imported.

Reason: When python runs 'somepath/file.py', it prepends 'somepath' to sys.path. So when python runs '.../idlelib/whatever.py', it prepends '.../idlelib' to sys.path. This occurs in both IDLE and user processes.

When 'somepath/file.py' is either on startup or from the editor, IDLE first prepends 'somepath' to sys.path just as python does.

We are planning to add perhaps 50 new short lowercase (PEP-8) names to idlelib. This will greatly increase the possibility of accidentally matching something users write, if only in error, or even of masking a stdlib module.

Solution: In PyShell.main and run.main, remove the prepended idlelib entry *if it is there* (see below). On Windows, at least, for installed python, sys.path would then be identical on startup whether a file is run with python or IDLE.

Would this affect IDLE itself? I am sure not. AFAIK, IDLE does not exploit the presence of idlelib on sys.path, and always imports idlelib files via idlelib.

Further more, when IDLE is started from console Python with 'import idlelib.idle', which I regularly do for the repository version, '.../idlelib' is *not* prepended.  This is why IDLE should not depend on its presence and why removal should check first.  This also means that whether a file runs in IDLE depends on how IDLE is started, whether directly or via '>>> import idlelib.idle' within console python.

Note 1. idlelib.idle prepends repository 'lib' even when already present; this could be fixed also. Another issue to track down, probably not due to IDLE, is that at least on Windows the binary directory is present twice.)

Note 2: The contents of sys.modules would still be different, along with the problem that 'import tkinter; tkinter.font' runs in IDLE. (This example comes from a Stackoverflow question.) But this is a different issue and I cannot see a fix for it.

[terryjreedy]

When python is started without running a file (Windows icon or command line), '' is prepended to sys.path. When IDLE is started from 3.5 icon, '' is prepended temporally after and spatially before '.../idlelib', so the latter, to be removed, is sys.path[1] and not sys.path[0].

'' in sys.path interpreted as the current working directory.

When IDLE is started from the command line with '-m idlelib' instead of '.../idlelib/idle.py' (for instance), the current working directory is added instead of .../idlelib (in second position).

Adding the startup current directory is redundant with '' until the user imports os and changes the cwd. Then I expect the behavior of Shell to be different from python interactive mode, which would be a bug. I will try to find a test case later.

PyShell.main manipulates 'sys.path' in lines 1521 and 1525, but these are sys.argv of the idle process, not the user process, each of which have their own sys.argv. If filenames are given on the command line to be edited, the directory of each is added to sys.path *for the idle process* (line 1521). I suspect this is for completions. Line 1525 add the current working directory. I don't know why. Perhaps it is a holdover for -n single process mode.

[terryjreedy]

bpo-25507 addresses the tkinter.font problem of Note 2 in the first message.

[terryjreedy]

In private email, eryk sun reports "/usr/bin/idle3 instead prepends 3 directories: ['', current_directory_absolute, '/usr/bin']. If I use the -r option to run a script, it instead prepends script_directory_relative, '', '/usr/bin']. I think these should be [''] and [script_directory_absolute], respectively."

Windows does not have idle3. None of the directories above are needed to run IDLE and I believe any could contain masking files.

[terryjreedy]

Upload: A listing of console (cross-platform) and icon (Windows-specific) IDLE starting methods and the effect on sys.path and current directory. I believe that some of the details of the 'constant' part of sys.path are system dependent, but stable across methods for a particular python version.

[terryjreedy]

sys.path for 3.6.0b1 on Win 10. Start python from icon (start menu), console (py -3.6) or Explorer (click on python.exe) '', 'C:\\Programs\\Python36\\python36.zip', 'C:\\Programs\\Python36\\DLLs', 'C:\\Programs\\Python36\\lib', 'C:\\Programs\\Python36', 'C:\\Programs\\Python36\\lib\\site-packages',

Start Python from Explorer by double-clicking *.py file or context menu 'Open': replace '' with 'C:/user/Terry/lib' # directory containing file.

Running 'import sys; sys.path' (in user process) from IDLE Shell adds the following after ''. Start IDLE from icon: 'C:\\Programs\\Python36\\Lib\\idlelib', Start IDLE in console with py -3.6 -m idlelib: 'F:\\Python\\dev' # the CWD of the console Start IDLE in console with py -3.6 -m idlelib.idle: 'C:\\Programs\\Python36\\lib', # duplicate added by idle.py 'F:\\Python\\dev' # the CWD of the console Start IDLE from Python with 'import idlelib.idle' 'C:\\Programs\\Python36\\lib', # duplicate added by idle.py 'C:\\Programs\\Python36' # the CWD of the running Python (dup)

Currently, idle.py inserts the directory containing idlelib into sys.path. This is a duplicate unless running idlelib in a non-stanard location. Check first.

[1762cc99-3127-4a62-9baf-30c3d0f51ef7]

New changeset 5f788ad057ca by Terry Jan Reedy in branch '2.7': Issue bpo-25488: Stpp idle.py from adding a entry when it is a duplicate. https://hg.python.org/cpython/rev/5f788ad057ca

New changeset 013956a801e4 by Terry Jan Reedy in branch '3.5': Issue bpo-25488: Stpp idle.py from adding a entry when it is a duplicate. https://hg.python.org/cpython/rev/013956a801e4

New changeset 25ae49903496 by Terry Jan Reedy in branch '3.6': Issue bpo-25488: merge idle.py from 3.5. https://hg.python.org/cpython/rev/25ae49903496

[terryjreedy]

I usually start repository IDLE with 'import idlelib.idle' in repository console python, which I have pinned to my taskbar. This is at least as faster as entering a command in the console (which would block the console), keeps error messages for one version together in one place and separate from anything else, and leaves my console free to enter other commands.

The import change will make idlelib.idle.main() work to restart IDLE after closing it. Previously, idlelib.idle.idlelib.pyshell.main() was required, which I never did. I marked the import as subject to change, as I intend to move (and edit) much of pyshell.main to idle.py.

[terryjreedy]

I should have said that the problem with having '' at the beginning of the path is that when IDLE tries to import an stdlib module and '' (usually a user directory) contain a file with the same name, IDLE will import the user file instead and at some point likely exit. For example, a user file named code.py masks the stdlib code.py, which is essential to Shell operation (bpo-29395).

[terryjreedy]

The sys.pyth manipulation will be different in the idle and user processes. If IDLE is started with -n, so both processes are the same, even more care is needed.

[terryjreedy]

bpo-25488 and bpo-26143 address related issues of sys.path as seen by IDLE and user code. bpo-26143 focuses on what IDLE sees, and user modules shadowing IDLE modules, bpo-25488 focuses on what user code sees, and the presence of idlelib in sys.path making buggy code work. It includes startup information relevant to bpo-26143 also. Both should be solved by patching IDLE in same place.