PowerShell / PowerShell

PowerShell for every system!
https://microsoft.com/PowerShell
MIT License
43.55k stars 7.06k forks source link

7.4.0 Breaking Change: Tab expansion no longer converts tildes to full paths, breaking path passing to executables on Windows #20750

Closed bradwilson closed 3 weeks ago

bradwilson commented 5 months ago

Prerequisites

Steps to reproduce

With versions prior to 7.4.0, pressing Tab to get completion on a path with ~ converted that path into the full path for my home folder.

Type dir ~\down and press Tab:

For built-in commands (like dir), this is fine. For executables, this passes a path with ~ in it, which makes using such paths impossible unless the executable has its own support for mapping ~ to the home folder. For example, if I try to run notepad ~\.config\git\config, it will tell me the path isn't found; if I run notepad C:\Users\bradwilson\.config\git\config, it will open the file appropriately.

This breaks more than a decade of muscle memory expecting ~ to be translated into the correct path, as I've been using PowerShell as my shell since it was called Monad.

This appears to have been purposefully introduced in #19489. I cannot find any way to restore the old behavior short of sticking w/ version 7.3.9.

Expected behavior

PS> TabExpansion2 '~\Downloads' | select -ExpandProperty CompletionMatches

CompletionText                ListItemText        ResultType ToolTip
--------------                ------------        ---------- -------
C:\Users\bradwilson\Downloads Downloads    ProviderContainer C:\Users\bradwilson\Downloads

Actual behavior

PS> TabExpansion2 '~\Downloads' | select -ExpandProperty CompletionMatches

CompletionText ListItemText        ResultType ToolTip
-------------- ------------        ---------- -------
~\Downloads    Downloads    ProviderContainer C:\Users\bradwilson\Downloads

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.9
PSEdition                      Core
GitCommitId                    7.3.9
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Name                           Value
----                           -----
PSVersion                      7.4.0
PSEdition                      Core
GitCommitId                    7.4.0
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

rhubarb-geek-nz commented 5 months ago

Question - Would the enter autocorrect be useful if #20402 was implemented?

I don't want autocorrect ever on a command prompt that changes the command from what I think I am executing to something different without me confirming that is what I want. I don't care if it is Clippy, AI, ChatGPT, Code Pilot or Mechanical Turks. I want to execute the commands that I type and get the appropriate error if its wrong. If I want to cycle through the options then I can use the tab-expansion.

.. and I want the interpretation of what I type to be exactly the same as when I have the same text in a powershell script.

rhubarb-geek-nz commented 5 months ago
clippy-home
tats-u commented 5 months ago

$env:USERPROFILE is too long to typed every time. It is really pain of ass to type it frequently. Unquoted leading ~ must be converted to $env:USERPROFILE. If you want the verbatim result, you have just to quote it.

jhoneill commented 5 months ago

The problem with ~ being invisibly interpreted as $HOME in PSReadLine

And if it were invisible this would be a problem, but what happens is the text between the prompt and the cursor changes

But it is too late. You type X and press enter, it changes it to Y, prints and executes Y even if that is not what you wanted.

And that is why this should NOT be done out of the box and should only be done by people who are confident that the change is exactly what they want.... If someone wants to implement any kind of autocorrect you can tell them it's unwise, but it is their choice.

rhubarb-geek-nz commented 5 months ago

$env:USERPROFILE is too long to typed every time. It is really pain of ass to type it frequently. Unquoted leading ~ must be converted to $env:USERPROFILE. If you want the verbatim result, you have just to quote it.

$HOME is interpreted as home directory on all PowerShell platforms. As $Env:USERPROFILE on Windows and $Env:HOME on UNIX. So if you want to write portable scripts, use $HOME.

rhubarb-geek-nz commented 5 months ago

And that is why this should NOT be done out of the box and should only be done by people who are confident that the change is exactly what they want.... If someone wants to implement any kind of autocorrect you can tell them it's unwise, but it is their choice.

And you are welcome to put this PSReadLineWithAutoCorrection as a module in the PSGallery, but it does not need to be in the core project.

jhoneill commented 5 months ago

And that is why this should NOT be done out of the box and should only be done by people who are confident that the change is exactly what they want.... If someone wants to implement any kind of autocorrect you can tell them it's unwise, but it is their choice.

And you are welcome to put this PSReadLineWithAutoCorrection as a module in the PSGallery, but it does not need to be in the core project.

I wouldn't even publish it there. It's an answer to one question, and possibly a blog post on customizations which trap enter in PSR, but something for the unwary to download ... shudder We're at risk of getting into "I agree with you" , "NO!! I agree with you" land.

jhoneill commented 5 months ago

For people who want to use ~ as alias on the command line (without the tab 'fix' and hitting tab) there is the option to have psreadline switch it for them

@jhoneill, is your proposal to make the "enter autocorrect" a built-in option is PSReadLine?

NO! It's a specific hack - a workaround that people can use with bunch of caveats; possibly PSReadline could have a "swap tilde on enter" option but there are 101 other autocorrects people might want, and it shouldn't try to do them. And I don't mind telling someone on a thread like this that hack is possible, but I wouldn't push it out as widely as something to customize tab completion.

I would much prefer tilde expansion described in #20402, since it would allow tilde expansion in scripts. A lot of bash scripts use it, so it would be nice for linux users coming from bash as a default in a future release of powershell (assuming all goes well and there isn't some legacy windows code that relies on the tilde not expanding).

If one expands ~/ at the start of an unquoted string literal (and possibly doubly quoted, but probably not single quoted) it will avoid problems of ~ in file names. This is distinct from what happens with tab expansion (do we expect home to be expanded or not). It will lead to scripts that work in PowerShell 7 and not in Windows PowerShell, but that isn't a reason not to do it. Doing it will fix something that has been broken for a long time, and I can't think of a scenario where it not being broken anymore creates a risk.

If "enter autocomplete" was implemented as an option in PSReadLine, I wouldn't expect it to be enabled by default, since it would surprise users when their prompt changed after pressing enter (I believe @rhubarb-geek-nz was saying this).

Yes That's why it's fine (IMHO) to tell people how they can implement an "auto correct", but people would go mad if psr started "fixing" what they typed willy-nilly.

Question - Would the enter autocorrect be useful if #20402 was implemented? Any advantages compared to #20402?

It wouldn't. Like I say it's a hack, a short term workaround, a useful technique to have in your back pocket. 20402 is a fix.

What tab expansion does is then a matter of personal taste and can be addressed by changing tabexpansion2

mklement0 commented 5 months ago

Let me try another, hopefully comprehensive summary that may serve as the basis for deciding how to resolve the problem at hand:

Context:

The problem at hand arose from an intentional change with unintentional consequences:

Solution options:

(Potentially) separate considerations and workarounds:

bradwilson commented 5 months ago

$HOME is interpreted as home directory on all PowerShell platforms. As $Env:USERPROFILE on Windows and $Env:HOME on UNIX. So if you want to write portable scripts, use $HOME.

A maybe not insignificant nitpick here.

The meaning of ~ is context-dependent. What you're describing, in a script, is not actually related to my original issue, and script-mapping ~ on Windows is outside the scope of this issue, IMO.

Tab-competing/interpreting ~ interactively is dependent on your current location. If my current location is the file system, it shows me my home folder:

image

It's actually using the current provider's notion of 'home'. Some locations have no concept of home, like the registry:

image

For this reason, I think that the behavior of tab-competing ~ into $HOME rather than the provider's current notion of 'home' is a degenerate behavior fix. It will work in the majority case, but is a change in behavior.

bradwilson commented 5 months ago

Solution options:

The only 100% compatible solution is (a), though I suspect most people will be happy with any of the solutions.

Calls to external programs on Windows with arguments that are either an unquoted ~ in isolation or start with unquoted ~\ or ~/ and that the target program did not mean to expand the ~ to the value of $HOME itself anyway.

This is only effective if PowerShell stops quoting paths with spaces in them during tab completion, and instead reverts to bash's behavior of escaping spaces instead.

FWIW, PowerShell's behavior with tab completion on Linux is identical to Windows (that is, it quotes them rather than escaping them) and that of course breaks its own path translation, as does hand crafting the command using PowerShell's quoting. There's actually no way that I can see of today to get PowerShell on Linux to do the right thing here when paths have spaces:

bash image

pwsh image

(This issue isn't about fixing PowerShell's POSIX compliance; I'm just pointing out that its own behavior is self-defeating because of quoting, which would also be self defeating with your proposal for Windows to only transform unquoted paths.)

mklement0 commented 5 months ago

Thanks, @bradwilson.

I think that the behavior of tab-competing ~ into $HOME rather than the provider's current notion of 'home' is a degenerate behavior fix

This is only effective if PowerShell stops quoting paths with spaces in them during tab completion,


Yes, (a) is the simplest (immediate) fix, but it also prevents the introduction of useful new behavior, namely the preservation of abstractions in the tab-completed paths.

mklement0 commented 5 months ago

I spoke too soon with respect to expanding ~ to $HOME:

While doing so only for native (external) programs would indeed be the right solution and would mimic what PowerShell's tilde expansion already does (currently on Unix only), the challenge is to robustly detect whether the target command is an external program or not at tab-completion time:

bradwilson commented 5 months ago

the challenge is to robustly detect whether the target command is an external program or not at tab-completion time

I get why it's been suggested (because it makes converting "interactive exploration" into a script simpler) but I don't particularly like it. That said, the current behavior on Unix-likes does this during execution, despite it being wrong when the current location isn't the file system, so it's no more egregious than that decision (which is of course mandated? strongly suggested? by POSIX).

mklement0 commented 5 months ago

despite it being wrong when the current location isn't the file system

I don't think it's wrong: it's the only sensible thing to do (and I suspect most users aren't even aware of a potential conceptual issue, given that it is rare for the current PowerShell location not to be a file-system location):

mandated? strongly suggested? by POSIX).

rhubarb-geek-nz commented 5 months ago

... and the PowerShell-native use of ~ in the most important provider (FileSystem) it made sense to implement in PowerShell too.

I suggest that was the original sin! In the POSIX world, the tilde is interpreted by shells, not by the standard runtime libraries, the kernel or file systems. Assuming the classic interpretation of FileSystem, that does not know about users ( yes it knows about rights ) but not about users and their home directories

bradwilson commented 5 months ago

I'm assuming at this point we shouldn't expect a response until January because basically all of Microsoft takes December off. 😂

bradwilson commented 4 months ago

Now that the holidays are past, can we get someone from the PowerShell team to acknowledge this issue and discuss what the plans are for resolving it?

Thanks!

bradwilson commented 3 months ago

Guess I'm staying on 7.3.x forever. 😞

rhubarb-geek-nz commented 3 months ago

Guess I'm staying on 7.3.9 forever. 😞

7.3.11?

bradwilson commented 3 months ago

The point that I'm going to be left behind because of a dumb feature I can't disable isn't changed by saying "7.3.11?", but thanks.

bradwilson commented 3 months ago

There, I've edited it for you.

rhubarb-geek-nz commented 3 months ago

The point that I'm going to be left behind because of a dumb feature I can't disable isn't changed by saying "7.3.11?", but thanks.

Likewise I am stuck with multi-platform native library loading in modules not working at startup on 7.4.x

masterflitzer commented 3 months ago

The point that I'm going to be left behind because of a dumb feature I can't disable isn't changed by saying "7.3.11?", but thanks.

tilde being converted to full paths is a dumb feature, bash and zsh don't do it either and i always wondered why pwsh does this, it's irritating and in 99% of cases not the behavior you want, just because most binaries don't understand tilde doesn't mean the shell should default to bad behavior

honestly if you want to interact with binaries why not just write out $home, in most cases in a interactive shell context you interact with shell stuff and there it's a huge productivity plus if you can just work with tilde without it messing up your command line by expanding it everytime you press tab, in the cases where you don't want that behavior it's trivial to just type $home (on windows i had to always use $home while on linux i could use tilde, now i can work on both in the most productive way and still i don't loose any functionality because i always have $home when i need it)

you absolutely do not need to stay on v7.3.x, you just need to adapt a little but it's really not a big deal and i'm sure you'll find that it offers a productivity increasement when working with shell stuff, i will say tho a configuration option would be cool, cause why not give more options, but not expanding tilde should definitely be the default setting!

bradwilson commented 3 months ago

tilde being converted to full paths is a dumb feature

A "dumb feature" which has existed for almost 20 years.

you absolutely do not need to stay on v7.3.x, you just need to adapt a little

I'm not undoing TWO DECADES of habit for no reason.

Thanks for your feedback.

masterflitzer commented 3 months ago

cmd exist for even longer i think and its still a terrible shell, your argument is useless

bradwilson commented 3 months ago

An even more valuable post, again I thank you.

rhubarb-geek-nz commented 3 months ago

I'm not undoing TWO DECADES of habit for no reason.

Apparently tilde expansion did work in PowerShell 5 but not in a way that made sense to me.

masterflitzer commented 3 months ago

I doubt if you have two decades of using a feature that did not even exist in PowerShell 5.

tbf tilde did expand to full path in powershell 5, it's one of the things that make v5 not enjoyable to use

pwsh needs to move forward and get better instead of retaining weird behavior just because it used to work like that

bradwilson commented 3 months ago

PowerShell 2.0 is still installed on PCs that also have PowerShell 5.1 (as long as you also have .NET 3.5.1 installed). If you launch with powershell -version 2 it'll run that 2009 build, and you can clearly see tab expansion of ~ working there.

Tab expansion has been in since the April 2006 pre-1.0 beta.

bradwilson commented 3 months ago

Also, as I wrote back in November of last year:

I'm fine with the new behavior being the default, as long as I have a trivial way to opt out.

masterflitzer commented 3 months ago

Also, as I wrote back in November of last year:

I'm fine with the new behavior being the default, as long as I have a trivial way to opt out.

that's a very sensible approach so 👍 from me, as i said i'm also for more options, i'm not saying how you want it is terrible, just that the new behavior should be the default

archer884 commented 3 months ago

The new behavior is idiotic. Pretending it’s the way other shells work is even more idiotic. I USE the other shells all day. They work fine. Powershell is now broken.Sent from my iPhoneOn Jan 22, 2024, at 5:38 PM, masterflitzer @.***> wrote:

Also, as I wrote back in November of last year:

I'm fine with the new behavior being the default, as long as I have a trivial way to opt out.

that's a very sensible approach so 👍 from me, as i said i'm also for more options, i'm not saying how you want it is terrible, just that the new behavior should be the default

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

masterflitzer commented 3 months ago

The new behavior is idiotic. Pretending it’s the way other shells work is even more idiotic. I USE the other shells all day. They work fine. Powershell is now broken

open bash/zsh and type ls ~/bin and press tab, then do the same in powershell 7.4 and 7.3, idk what you're talking about, in v7.4 it will behave like bash/zsh, in 7.3 it will expand to C:\Users\username\bin\

the latter is idiotic behavior that is not present in any other shell i ever used, did i understand the issue wrong or what is going on?

rhubarb-geek-nz commented 3 months ago

The original PowerShell implemented the bash behaviour by not implementing what bash did.

PowerShell used to expand ~ with tab-completion, bash did not

bash expanded ~ to value of $HOME when passed to external commands, PowerShell did not

So the so-called "bash tilde like functionality" had no relation to what bash actually did.

The old PowerShell team might have thought they were implementing bash drop in replacement but in reality nothing in PowerShell works like bash and that is ok, because bash is bash, and PowerShell is PowerShell.

masterflitzer commented 3 months ago

the only thing that's missing in v7.4 is to replace tilde with $home when interacting with binaries

the tab completion behavior matches bash now which is imo superior to just expanding to absolute path (see my previous comment), that's my point

it's not about making pwsh a bash clone, i didn't say this, it's about improving shell behavior and many things bash does are battle tested and very good for productivity, also mind that i specifically mentioned one behavior of bash not that pwsh should do everything bash does

bradwilson commented 3 months ago

the only thing that's missing in v7.4 is to replace tilde with $home when interacting with binaries

It does this on non-Windows machines, as is required for POSIX. It just doesn't do it on Windows.

rhubarb-geek-nz commented 3 months ago

It does this on non-Windows machines, as is required for POSIX.

PowerShell is not a POSIX shell, and has nothing to do with any POSIX compliance, in fact it is closer to Python or Perl than any UNIX shell.

masterflitzer commented 3 months ago

powershell doesn't need to be posix compliant on windows or anything like that, posix doesn't matter in this discussion as far as windows goes, what matters is making powershell on windows nicer to use and reverting the changes of v7.4 would accomplish the opposite

bradwilson commented 3 months ago

powershell doesn't need to be posix compliant

They already translated ~ to $HOME on non-Windows OSes even before the 7.4.x change.

image

masterflitzer commented 3 months ago

none of my comments were non-windows related, i (almost) exclusivly use pwsh on windows and this issue is also mostly windows related as far as i understood it, but for clarity i will change my statement from powershell doesn't need to be posix compliant to powershell doesn't need to be posix compliant on windows

Jaykul commented 3 months ago

Seriously you two. Stop it. This is not productive.

tats-u commented 3 months ago

Why does PowerShell deliberately neglect to interpret the tilde as the user profile only in native commands in Windows? It has no consistency at all. All other OSes for PC and phones are POSIX. Windows has been disliked by developers who prefer terminals. They have got used to the behavior of other shells run in POSIX. PowerShell's behavior must compromise itself to those of other shells to welcome such developers to the CLI environment in Windows. They are sometimes forced to use Windows for Microsoft Office and some other closed-source products in their companies.

rhubarb-geek-nz commented 3 months ago

All other OSes for PC and phones are POSIX.

For differing values of "POSIX". Windows has received POSIX certification in the past, and with WSL could claim so now. You could not really say the same thing about Android. Yes it has a Linux kernel and C libraries with the same functions but I don't think you could claim it has a POSIX command interpreter available for users, possibly via adb or Termux.

tats-u commented 3 months ago

I forgot to say POSIX in my previous message includes OSes that are nearly compatible with it (Linux) and ones that use other POSIX-like OSes as their kernels (Android). Browsers in Android contain the string "Linux" in their user agents.

rhubarb-geek-nz commented 3 months ago

I forgot to say POSIX in my previous message includes OSes that are nearly compatible with it (Linux) and ones that use other POSIX-like OSes as their kernels (Android).

Android is a Linux system because it uses a Linux kernel. When you write an Android application you don't write it using the POSIX API, you use a Java based Android SDK.

So called Linux systems are really GNU-Linux, it is the the kernel along with the GNU libraries and programs that make Linux distributions POSIX compliant.

PowerShell is most definitely not a POSIX shell. zsh/bash/ksh/dash can all run shell scripts written for the Bourne shell, sh. The only thing PowerShell has in common with UNIX shells is it has "Shell" in the name.

tats-u commented 3 months ago

GNU libraries and programs that make Linux distributions POSIX compliant.

Alpine Linux does not use GNU tools by default. GNU tools can be run in macOS (maybe not all though) and Hurd. You cannot overturn the fact that Android and Linux distros use the POSIX-compatible component as their important and essential part.

PowerShell is most definitely not a POSIX shell. zsh/bash/ksh/dash can all run shell scripts written for the Bourne shell, sh. The only thing PowerShell has in common with UNIX shells is it has "Shell" in the name.

This does not mean PowerShell is allowed to deviate from the norm among POSIX shells. Fish, a very popular non-POSIX-compliant shell, still converts the leading ~ to $HOME. The behavior in PowerShell 7.4 for Windows is really confusing. I have not heard Windows-native tools that use ~ as the prefix of names of files that can be edited by users. You can just quote the string with leading ~ with '. It requires 2x extra types and the shift key in the US keyboard layout to replace ~ with $HOME.

rhubarb-geek-nz commented 3 months ago

This does not mean PowerShell is allowed to deviate from the norm among POSIX shells.

This is some law I am unfamiliar with. PowerShell is not really a shell, it just has shell in the name. It is much more like Python or Perl, as scripting language with class library and packaged modules.

tats-u commented 3 months ago

This is some law I am unfamiliar with.

This is not a real law, but just common sense. If breaks, Windows' reputation among developers will be further damaged.

PowerShell is not really a shell, it just has shell in the name.

Both are the face of the CLI in each OS. Users who want to operate OSes via terminal have to use them. The current face of Windows CLI looks ugly compared to those of macOS or Linux.

It is much more like Python or Perl, as scripting language with class library and packaged modules.

PowerShell does not accept the following, so it is not concerned:

$foo = ~\bar\baz

Bash accepts:

FOO=~/bar/baz
tats-u commented 3 months ago

If you claim that PowerShell does not have to be POSIX-shell like, it has the consequence that many cmdlet aliases like ls, rm, and pwd would be erased from the Windows edition. dir Only those came from the trashy CMD, i.e. dir and del, would remain.