do not add directory of sys.argv[0] into sys.path

[1b5f39fd-febf-4146-81ba-6d85f5f45122]

BPO	946373
Nosy	@mwhudson, @birkenfeld, @pfmoore, @josiahcarlson
Files	python-noautosys.patch: do not add magically directory of sys.argv[0]

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core'] title = 'do not add directory of sys.argv[0] into sys.path' updated_at = user = 'https://bugs.python.org/wrobell' ``` bugs.python.org fields: ```python activity = actor = 'georg.brandl' assignee = 'none' closed = True closed_date = None closer = None components = ['Interpreter Core'] creation = creator = 'wrobell' dependencies = [] files = ['5959'] hgrepos = [] issue_num = 946373 keywords = ['patch'] message_count = 22.0 messages = ['45910', '45911', '45912', '45913', '45914', '45915', '45916', '45917', '45918', '45919', '45920', '45921', '45922', '45923', '45924', '45925', '45926', '45927', '45928', '45929', '45930', '45931'] nosy_count = 9.0 nosy_names = ['mwh', 'anthonybaxter', 'georg.brandl', 'jvr', 'paul.moore', 'isandler', 'jlgijsbers', 'josiahcarlson', 'wrobell'] pr_nums = [] priority = 'normal' resolution = 'wont fix' stage = None status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue946373' versions = [] ```

[1b5f39fd-febf-4146-81ba-6d85f5f45122]

Python adds magically directory of sys.argv[0] into sys.path, i.e.

>>> import sys
>>> sys.path
['', '/usr/lib/python23.zip', '/usr/share/python2.3',
'/usr/share/python2.3/plat-linux2',
'/usr/share/python2.3/lib-tk',
'/usr/lib/python2.3/lib-dynload',
'/usr/lib/python2.3/site-packages',
'/usr/lib/python2.3/site-packages/gtk-2.0',
'/usr/share/python2.3/site-packages']

where '' (or /usr/bin when executed script is in /usr/bin directory, etc.) is added automatically.

It is useful in many circumstances but fails when name conflict occurs. For example, create getpass.py or pdb.py scripts which import getpass and pdb modules. Script names conflict with modules names and modules are not going to be imported because path to the scripts is appended into sys.path, so a script is imported instead of a module.

The solutions:

1. User of script with conflicting name (i.e. getpass.py or timeit.py) can set PYTHONPATH to system library path, i.e. /usr/lib/python2.3.
2. User can modify the script to delete site.path[0].
3. User can rename the script.
4. Python can be modified to not add magically directory of sys.argv[0].

The 1. is a tricky and not intuitive and quite funny: set PYTHONPATH to system library path to import system module (and only in specific circumstances). ;-P

The 2. is a dirty hack: hey, we gonna import system module, ain't it?

The 3. is, IMHO, not acceptable because there is more than 200 python system modules, more in the future and user cannot be forced to maintain script names blacklist.

The 4. is only acceptable, IMHO. It makes python more inconvenient but gives no trouble when names conflict occurs. Moreover, fourth solution makes python more standard with other languages behaviour, i.e. one has to set CLASSPATH to load Java classes.

Maybe there is another solution, but...

Patch attached.

[fd930960-0aae-426c-bbb8-8549e6684f33]

Logged In: YES user_id=971153

Would not this cause serious backward compatibility problems??

[1a90d5ba-e091-4ed6-a644-41c5568983da]

Logged In: YES user_id=29957

I've been bitten by this before. See e.g. the shtoom.py script clashing with the shtoom package. I used the hacky approach of moving '' to the end of sys.path. While it would be nice if this wasn't needed, I can't see this being anything other than a backwards compatibility nightmare. It will absolutely break a lot of things to change it.

[ef6b2a61-f027-4805-a66a-cde4eee277c3]

Logged In: YES user_id=341410

This "problem" will be fixed in Python 2.4 with the introduction of absolute and relative import semantics as given in PEP-328:

http://www.python.org/peps/pep-0328.html

As stated in the PEP, to use the obviously backwards incompatible semantics, the future import will be used for 2.4 and 2.5, where in 2.6 it will become the default.

from __future__ import absolute_import

[6177d10d-caa6-4863-994f-777a4a7de0eb]

Logged In: YES user_id=469548

wrobell, would you be willing to produce a version of the patch which implements PEP-328? I'll close this patch if not.

[1b5f39fd-febf-4146-81ba-6d85f5f45122]

Logged In: YES user_id=387193

i will not provide the patch for 328, so closing this issue

[1b5f39fd-febf-4146-81ba-6d85f5f45122]

Logged In: YES user_id=387193

I am opening it again to discuss it a little more... Question to Josiah Carlson or anybody who can answer:

How PEP-328 is going to solve problem I have described?

If I name my script email.py, which will try to import email standard Python package. Then run the script, it will import itself instead of Python package, because directory where email.py is installed is added to sys.path.

[ef6b2a61-f027-4805-a66a-cde4eee277c3]

Logged In: YES user_id=341410

If the entirety of PEP-328 made it into Python 2.4 (I don't have an installation of 2.4, so don't know), to import your 'email.py' module, you would use 'from . import email' after enabling the absolute import semantics with 'from __future__ import absolute_import'. You would then import the standard email package with 'import email'.

Is this not clear by reading PEP-328?

[1b5f39fd-febf-4146-81ba-6d85f5f45122]

Logged In: YES user_id=387193

But the problem is not with naming of my modules/packages (part about relative import of modules I do understand, of course), but with script naming.

For example consider script:

  #!/usr/bin/python
   import email
   print 1

And name the script email.py, then run it, please. Python tries to be too smart (IMHO) and variable sys.path is polluted with directory of email.py script, therefore standard email Python package will not be imported.

[ef6b2a61-f027-4805-a66a-cde4eee277c3]

Logged In: YES user_id=341410

If you were to make your 'email.py' file contain the following...

#!/user/bin/python
from __future__ import absolute_import
import email
print 1

It should import the email package.

[0aa1c2b5-8e64-4988-b25b-7db15cdc7c29]

Logged In: YES user_id=92689

That doesn't follow at all. The script email.py will _still be found first instead of the email module. Like wrobell, I don't see what this has _anything to do with relative vs. absolute imports.

While a common newbie gotcha, I don't think it's worth the trouble to try to "fix" this. Recommending "won't fix".

[pfmoore]

Logged In: YES user_id=113328

Another point - given a program which comprises a couple of .py files in the same directory (say main.py and support.py) it would be quite normal (at least for me!) to do "import support" from main.py. This patch would break this - and I'd find it difficult to accept what I'm doing as "a mistake".

Fixing this would involve adding something like

    sys.path.insert(0,
os.path.dirname(os.path.abspath(sys.argv[0]))

at the top of my main script. I'd hate to try to explain that to a beginner...

The use case here seems to be when a script itself has the same name as a standard library module. I'd have to say that this seems a fairly unlikely case - and easy to fix when it happens.

[1b5f39fd-febf-4146-81ba-6d85f5f45122]

Logged In: YES user_id=387193

I do understand that the patch will not be accepted. :-) That's ok. Too much fight with people's habits for me. :]

But, let's see what common script names are forbidden now (among others): array.py, binascii.py, bz2.py, collections.py, crypt.py, datetime.py, math.py, md5.py, mmap.py, parser.py, pwd.py, regex.py, resource.py, select.py, sha.py, syslog.py, time.py, timing.py, timeit.py, binhex.py, calendar.py, cgi.py, chunk.py, cmd.py, code.py, commands.py, compileall.py, compiler.py, copy.py, csv.py, decimal.py...

And in the future there can be even more depending on the new modules in Python itself and third party modules (i.e. spread.py, dbus.py, eca.py, etc.). If new module or package appears, then you will have to change your name of the script. I do understand that it is not frequent situation, but we should not to be forced to avoid certain _common_ words for script naming.

IMHO, it is a problem and should be fixed. The question is "How?". Maybe page listed below should be discussed (again I think):

http://hkn.eecs.berkeley.edu/~dyoo/python/__std__/

I would set Resolution to "Remind" if you agree to discuss it later and fix the problem somehow in the future. If not, then "Won't fix".

[ef6b2a61-f027-4805-a66a-cde4eee277c3]

Logged In: YES user_id=341410

I'm sorry, the bit I quoted shouldn't go into your email.py file, it should go into the module that wants to import Python's email package, and not your email.py module (my brain was fuzzy on the 16th).

Standard library name masking is exactly what the absolute imports PEP was seeking to fix. You use "from __future__ import absolute_imports", and from then on, you can do relative imports via "import .modulename" (note the leading period), and stdlib imports via "import modulename" (note the lack of a leading period). It also allows you to go higher up in paths via additional leading periods.

This /does/ in fact fix the problem mentioned, at the cost of having to change the import lines because of the changed import semantic. This allows users to choose names that they desire, even if it mirrors a standard library module name.

It also doesn't require any patches.

[0aa1c2b5-8e64-4988-b25b-7db15cdc7c29]

Logged In: YES user_id=92689

Standard library name masking is exactly what the absolute imports PEP was seeking to fix

Only in the context of submodule imports within packages. Which is _not at _all what is being described here. There is a main _script called email.py which wants to import the email module (that email happens to be a package is not relevant). There is _no relative import going on here, it just so happens that the script's parent dir is in sys.path before the std lib.

[ef6b2a61-f027-4805-a66a-cde4eee277c3]

Logged In: YES user_id=341410

Absolute imports will also fix that. A bare "from __future__ import absolute_imports;import email" will import the email package, at the cost of changing the semantics of relative imports. What is the problem? Why cannot it be used? What in this entire problem is not solved by absolute imports with its changed import semantic that already exists in Python 2.4?

[mwhudson]

Logged In: YES user_id=6656

Absolute imports will also fix that.

No it won't! The directory containing email.py is on sys.path, at the front.
So "import email" will find it.

What in this entire problem is not solved by absolute imports with its changed import semantic that already exists in Python 2.4?

Nothing at all is solved by a change that isn't in Python 2.4!

I still think this bug should be closed won't fix.

[ef6b2a61-f027-4805-a66a-cde4eee277c3]

Logged In: YES user_id=341410

A literal reading of "Guido's Decision" in PEP-328 says that if absolute imports were implemented, then the only thing missing is a future import, and an __init__.py file in the same path as email.py.

I finally got around to installing 2.4, and (unfortunately) it seems as though absoluteimport is not offered in the \_future__ module. What happened? I thought PEP-328 was accepted for inclusion in 2.4. Did someone not have the time to write the import hook?

[mwhudson]

Logged In: YES user_id=6656

A literal reading of "Guido's Decision" in PEP-328 says that if absolute imports were implemented, then the only thing missing is a future import, and an __init__.py file in the same path as email.py.

I don't think this __init__.py file had been mentioned before. However, even if it is there, THE DIRECTORY CONTAINING email.py IS ON sys.path! What's hard to understand about this?

I finally got around to installing 2.4, and (unfortunately) it seems as though absoluteimport is not offered in the \_future__ module. What happened?

It's awaiting an implementation, AFAIK.

[ef6b2a61-f027-4805-a66a-cde4eee277c3]

Logged In: YES user_id=341410

It's not hard to understand, but I could have sworn that in the discussion about absolute imports from the spring of last year that it wasn't just a package thing, it was supposed to functionally do-away with "" being in sys.path for all modules in which the future import had been performed.

It seems as though I was mistaken as to the reasons behind the PEP, but can you blame me? A single mechanism for handling stdlib vs. non-stdlib imports would be great (I would say should be the one true solution), and would solve the 10/week questions about imports in comp.lang.python.

[fd930960-0aae-426c-bbb8-8549e6684f33]

Logged In: YES user_id=971153

Seems like the main motivation for the original patch was to prevent accidental conflicts between user's and standard module names

Would emitting a warning (with an easy way to explicitly suppress it) in such a case solve the original problem?

I am not sure whether such a warning should be limited to user/system name conflicts or to any module name conflicts?

[birkenfeld]

Logged In: YES user_id=849994

This is too problematic to change the behavior. Closing as "Won't fix", the consensus of the comments.