Windows launcher docs don't fully explain shebang semantics

[ff73569b-4c81-47ac-bf49-9d0a6f79d01e]

BPO	24890
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['3.8', '3.9', '3.10', 'type-feature', 'OS-windows', 'docs'] title = "Windows launcher docs don't fully explain shebang semantics" updated_at = user = 'https://bugs.python.org/BrenBarn' ``` bugs.python.org fields: ```python activity = actor = 'eryksun' assignee = 'docs@python' closed = False closed_date = None closer = None components = ['Documentation', 'Windows'] creation = creator = 'BrenBarn' dependencies = [] files = [] hgrepos = [] issue_num = 24890 keywords = [] message_count = 3.0 messages = ['248796', '248813', '388173'] nosy_count = 7.0 nosy_names = ['paul.moore', 'tim.golden', 'docs@python', 'BrenBarn', 'zach.ware', 'eryksun', 'steve.dower'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue24890' versions = ['Python 3.8', 'Python 3.9', 'Python 3.10'] ```

[ff73569b-4c81-47ac-bf49-9d0a6f79d01e]

With the fix of issue bpo-17903 in place, the behavior of the Windows launcher for a shebang of "#!/usr/bin/env python" is now different from the behavior for the other "virtual commands" in that the PATH is searched for this particular virtual command but not the others. But this difference is not explained in the docs (see https://docs.python.org/3/using/windows.html#shebang-lines). The docs should be updated to make the difference clear.

In addition, as noted in bug bpo-24625, the launcher parses and executes shebang lines "normally" if the shebang is not one of the virtual commands. However, this is not actually stated on the documentation page; the only examples on the page either use the virtual commands or use a bare command like "python". Given that shebang lines don't normally work at all on Windows, and given that the shebang handling is already different on Windows because of the virtual commands, the docs shouldn't assume that it's clear what "works like a normal shebang" means. A note should be added to the docs to make this explicit. Something like "If the shebang line is not one of these virtual commands, it is executed as a normal program and the script filename is passed to it as a command-line argument."

[eryksun]

The patch for bpo-23465 (PEP-486) updated the docs to explain the behavior of #!/usr/bin/env as follows:

The /usr/bin/env form of shebang line has one further special
property. Before looking for installed Python interpreters, 
this form will search the executable PATH for a Python 
executable. This corresponds to the behaviour of the Unix 
env program, which performs a PATH search. 

This paragraph needs to be added to the 3.4 docs.

Another undocumented feature is adding custom commands in the [commands] section of py.ini. The read_commands function handles this.

Speaking of which, a side effect of defining SEARCH_PATH in PC/launcher.c is that PATH is searched in find_command if the command isn't found in the global "commands" array. This can lead to unexpected behavior when combined with SKIP_PREFIX (i.e. the skip_prefix function skips past any of the builtin_prefixes in the shebang). For example, #!/usr/bin/perl will search the current directory and PATH for "perl.COM" "perl.EXE", "perl.BAT", etc (i.e. trying each extension listed in PATHEXT). This behavior should be documented. Preferably I think it should also be constrained to only search when the prefix is /usr/bin/env, such as by adding a "search" boolean parameter to the find_command function.

[eryksun]

In summary, the launcher documentation needs to be extended as follows:

* Include an example of a shebang with a normal Windows path:

  #!"C:\Program Files\Python310\python.exe"

  Highlight the importance of quoting paths that contain spaces.
  Also, quoting ensures a real filesystem path will never
  mistakenly match a virtual command. 

  Due to questionable design in the launcher, it will still try 
  searching [1] for part of the name up to the first space, such
  as "C:\Program", but, presuming the path is quoted, the search
  will at least never succeed since double quotes are not
  allowed in filenames.

* Show how to define custom commands in py.ini:

  [commands]
  pypy3=C:\pypy3.7\pypy3.exe

  The launcher is limited to 100 commands since it uses a hard-
  coded array size.

  Note that regular CLI commands take precedence. In other words,
  if a command name is found in the standard search path [1], then
  the search result is used instead of the value defined in py.ini.

  This is buggy behavior that should be fixed. When reading py.ini, 
  it should not search for the command. The find_command() function
  needs a `search` parameter to support this. If it's false, 
  find_command() should only check its `commands` array.

* Explain that the Unix path prefixes "/usr/bin/env ", 
  "/usr/bin/", and "/usr/local/bin/" are ignored when matching
  a command name -- except for the already documented case of
  "/usr/bin/env python" (without a version spec).

  Except for reserved "python" commands, currently any command
  name will be searched for [1], with or without the
  "/usr/bin/env " prefix. For example:

  #!name

  The launcher will search for "name.COM", "name.EXE",
  "name.BAT", and so on.

  This is buggy behavior that should be fixed. The skip-prefix 
  feature needs to be smarter. The launcher should only search
  the filesystem for commands that use the "/usr/bin/env "
  prefix. The find_command() function needs a `search`
  parameter to support this, as mentioned above.

Bugs mentioned in this message need a separate issue. I don't know to what extent questionable behavior should be documented, with warnings about security and misbehavior. In practice it seems that most users limit themselves to a restricted subset of the launcher's capabilities, and never stray from the well-worn path. So I don't want to needlessly scare or worry people. But the buggy behavior SHOULD be taken seriously and fixed.

---

[1] The launcher uses the system's default non-safe search path for data files. It should enable the safe search via

    SetSearchPathMode(BASE_SEARCH_PATH_ENABLE_SAFE_SEARCHMODE)

The non-safe search path includes the application directory
(of the launcher); the current directory; the System32,
System and Windows directories (unsafe because they're after
the current directory); and the directories in the PATH
environment variable.

If there's no "." in the name, the launcher tries searching 
for the name plus each extension that's listed in the PATHEXT 
environment variable. To search for a filename that has no
extension, add a trailing dot to the command name, e.g.
"spam.".