7.4.0 Breaking Change: Tab expansion no longer converts tildes to full paths, breaking path passing to executables on Windows

bradwilson commented 5 months ago

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

With versions prior to 7.4.0, pressing Tab to get completion on a path with ~ converted that path into the full path for my home folder.

Type dir ~\down and press Tab:

In 7.3.9, it will be completed as dir C:\Users\bradwilson\Downloads\
In 7.4.0, it will be completed as dir ~\Downloads\

For built-in commands (like dir), this is fine. For executables, this passes a path with ~ in it, which makes using such paths impossible unless the executable has its own support for mapping ~ to the home folder. For example, if I try to run notepad ~\.config\git\config, it will tell me the path isn't found; if I run notepad C:\Users\bradwilson\.config\git\config, it will open the file appropriately.

This breaks more than a decade of muscle memory expecting ~ to be translated into the correct path, as I've been using PowerShell as my shell since it was called Monad.

This appears to have been purposefully introduced in #19489. I cannot find any way to restore the old behavior short of sticking w/ version 7.3.9.

Expected behavior

PS> TabExpansion2 '~\Downloads' | select -ExpandProperty CompletionMatches

CompletionText                ListItemText        ResultType ToolTip
--------------                ------------        ---------- -------
C:\Users\bradwilson\Downloads Downloads    ProviderContainer C:\Users\bradwilson\Downloads

Actual behavior

PS> TabExpansion2 '~\Downloads' | select -ExpandProperty CompletionMatches

CompletionText ListItemText        ResultType ToolTip
-------------- ------------        ---------- -------
~\Downloads    Downloads    ProviderContainer C:\Users\bradwilson\Downloads

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.9
PSEdition                      Core
GitCommitId                    7.3.9
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Name                           Value
----                           -----
PSVersion                      7.4.0
PSEdition                      Core
GitCommitId                    7.4.0
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

237dmitry commented 5 months ago

In fact, when this behavior finally appeared in 7.4.0-rc.1, I was glad about it. But perhaps this is not as relevant in Windows as it is in Linux.

bradwilson commented 5 months ago

It's very counterproductive in Windows.

~You could argue it's somewhat counterproductive in Linux as well unless every single application you use from the command line already maps ~. Many of the built-in utilities probably already have to do this, so it's less obvious when failures will materialize.~

PowerShell follows POSIX (partially) and translates ~ before calling executables on Linux/macOS.

237dmitry commented 5 months ago

On Linux, all native applications interpret the tilde as a synonym for the user's home directory. It’s just that the command line is shorter due to the fact that it does not expand the path to the full one. The same cannot be said about Windows; most utilities probably simply will not understand what this path is.

bradwilson commented 5 months ago

It's worth pointing out that the Unix-y apps on Windows (for example, all the tools that ship with Git) do map ~ but will only do it if you pass Unix-y separators.

Using less version 563 (shipped with Git 2.29.2.windows.2), these work:

7.3.9: less C:\Users\bradwilson\.config\git\config And crafted by hand: less ~/.config/git/config

But this doesn't:

7.4.0: less ~\.config\git\config results in ~.configgitconfig: No such file or directory

Since PowerShell uses the backslash when tab completing, it's creating command lines that will be broken with these Unix-y tools.

bradwilson commented 5 months ago

To be clear, I don't think the use of the backslash is wrong for PowerShell Core on Windows, as the Unix-y tools do work as expected with backslashes but only with fully qualified paths (which Tab completion no longer generates in 7.4.0).

237dmitry commented 5 months ago

I was talking specifically about Linux as the actual installed OS. Not WSL, not ported utilities, but about my main OS, which is the only one on the computer (Windows on the older notebook).

Therefore, I would like everything to remain as it is, at least on Linux.

bradwilson commented 5 months ago

I'm fine with the new behavior being the default, as long as I have a trivial way to opt out.

MartinGC94 commented 5 months ago

Yes, as you've discovered this was an intentional change and there's currently no way to change the behavior. A quick workaround you can implement on your own machine is to update the tabexpansion2 function to replace the ~ character in the completion results with your home path.

Assuming this is something that people want fixed, I can think of the following solutions:

Permanently revert this change
Add a command to control the behavior, like in: https://github.com/PowerShell/PowerShell/pull/19518 or if that will take too long, revert the change until such a command is added
Make PS resolve ~ before passing it in as an argument to native commands.

Quick and dirty tabexpansion2 workaround:

``` function TabExpansion2 { <# Options include: RelativeFilePaths - [bool] Always resolve file paths using Resolve-Path -Relative. The default is to use some heuristics to guess if relative or absolute is better. To customize your own custom options, pass a hashtable to CompleteInput, e.g. return [System.Management.Automation.CommandCompletion]::CompleteInput($inputScript, $cursorColumn, @{ RelativeFilePaths=$false } #> [CmdletBinding(DefaultParameterSetName = 'ScriptInputSet')] [OutputType([System.Management.Automation.CommandCompletion])] Param( [Parameter(ParameterSetName = 'ScriptInputSet', Mandatory = $true, Position = 0)] [AllowEmptyString()] [string] $inputScript, [Parameter(ParameterSetName = 'ScriptInputSet', Position = 1)] [int] $cursorColumn = $inputScript.Length, [Parameter(ParameterSetName = 'AstInputSet', Mandatory = $true, Position = 0)] [System.Management.Automation.Language.Ast] $ast, [Parameter(ParameterSetName = 'AstInputSet', Mandatory = $true, Position = 1)] [System.Management.Automation.Language.Token[]] $tokens, [Parameter(ParameterSetName = 'AstInputSet', Mandatory = $true, Position = 2)] [System.Management.Automation.Language.IScriptPosition] $positionOfCursor, [Parameter(ParameterSetName = 'ScriptInputSet', Position = 2)] [Parameter(ParameterSetName = 'AstInputSet', Position = 3)] [Hashtable] $options = $null ) End { $TempRes = if ($psCmdlet.ParameterSetName -eq 'ScriptInputSet') { [System.Management.Automation.CommandCompletion]::CompleteInput( <#inputScript#> $inputScript, <#cursorColumn#> $cursorColumn, <#options#> $options) } else { [System.Management.Automation.CommandCompletion]::CompleteInput( <#ast#> $ast, <#tokens#> $tokens, <#positionOfCursor#> $positionOfCursor, <#options#> $options) } [System.Management.Automation.CompletionResult[]]$NewCompletionList = foreach ($Item in $TempRes.CompletionMatches) { if ($Item.CompletionText -like "*~*") { [System.Management.Automation.CompletionResult]::new( $Item.CompletionText.Replace('~', $HOME), $Item.ListItemText, $Item.ResultType, $Item.ToolTip ) } else { $Item } } $TempRes.CompletionMatches = $NewCompletionList $TempRes } } ```

mklement0 commented 5 months ago

Let me provide some context:

On Unix-like platforms it is the (POSIX-compatible) shell (such as bash) that performs tilde (~) expansion, not the standard utilities (such as as ls) themselves; the same applies to globbing (wildcard expansion in filenames), among other such so-called shell expansions. You can verify this with /bin/ls --% ~ from PowerShell.

On Unix-like platforms only, PowerShell emulates this behavior (as well as globbing) when calling external programs:
- That is, it expands an unquoted ~ - either in isolation of if followed by/ (but not \) - to the value of $HOME.
- As an aside: This emulation is incomplete:
- Syntactically: Use of ` to escape ~ - /bin/echo `~- should work like /bin/echo '~' and /bin/echo "~", i.e. should suppress expansion, but currently doesn't; the inverse is true for /bin/echo ~/'foo bar', which should expand -d due to the ~ itself being unquoted, but doesn't (if you remove the space, it works): #20754
- Lack of support for other users' home directories: ~user1 works in POSIX-compatible shells to refer to the home directory of user1 (verify from PowerShell with sh -c "echo ~$env:USER"), but not in PowerShell: #12387
On Windows, where the native shell (cmd.exe) has no such expansion features, PowerShell does not perform them.
- Individual cross-platform utilities, especially those with a Unix heritage, may compensate for this by implementing these expansions themselves, though possibly only globbing.

Given the above, I propose the following variation of @MartinGC94's proposal 1:

Revert the change on Windows only, i.e. go back to making ~ expand to the value of $HOME only there - conceivably, it could even expand to verbatim $HOME, so as to preserve the intent to express the path abstractly.

While making PowerShell emulate literal ~ expansion on argument-passing to external programs on Windows also may seem appealing, there's a greater risk of breaking existing Windows-only code that never anticipated ~ as having special meaning and therefore requiring quoting. Also, emulating globbing too is definitely not an option, as a program expecting, say, file*.txt verbatim, then receiving multiple arguments (file1.txt file2.txt ...) could break.

mklement0 commented 5 months ago

Taking a step back, re proposal 3 (unquoted ~ expansion on external-program argument-passing also on Windows): It may be an option IF:

We're not concerned about the resulting asymmetry (tilde expansion then works everywhere, globbing (filename expansion) only on Unix).
More importantly, that the technically breaking change resulting from expanding a token-initial, unquoted ~ (either in isolation or if followed by \ or /) to the current user's home directory falls into bucket 3, i.e. is unlikely to break existing code.

Jaykul commented 5 months ago

For what it's worth, the problem I have is that I don't think you went far enough. I really don't want TabCompletion to replace generic values like ~ with the specific value on my computer. The same goes for or $home and $pwd and any other variable that I might have a partial path in.

How about instead of reverting a good feature, we replace the literal ~ at the front of the path with $HOME and then ... leave all the variables as variables in the path?

Since PowerShell expands variables when passing arguments to native apps, this gets us the best of all worlds:

Tab-completed paths work with native apps, like they used to
The shortest syntax like ~/Doc{Tab} still tab completes
Tab-completion will not make my carefully generic "$HOME/Docu{Tab}" into a script that only works on my computer.
Does not require potentially breaking changes to argument passing

NOTE: the catch to this is that it doesn't work if they're single-quoting the path they are tab-completing, in which case we might want to resort to the old behavior...

MartinGC94 commented 5 months ago

I am planning on creating a PR to stop it from replacing variables with their resolved values but that's a separate issue from this.
I don't like the idea of replacing ~ with $home when tab completing because it would feel weird and confusing for anyone outside of this discussion. It also wouldn't be accurate because the home location that ~ refers to is set in the provider, while $home just points to the same static location always. See:

PS C:\Users\Martin> $Provider=Get-PSProvider FileSystem
PS C:\Users\Martin> $Provider.Home
C:\Users\Martin
PS C:\Users\Martin> $Provider.Home = 'C:\Windows'
PS C:\Users\Martin> TabExpansion2 'ls ~\' | select -ExpandProperty CompletionMatches | select -First 1

CompletionText ListItemText        ResultType ToolTip
-------------- ------------        ---------- -------
~\appcompat    appcompat    ProviderContainer C:\Windows\appcompat

PS C:\Users\Martin> $HOME
C:\Users\Martin

rhubarb-geek-nz commented 5 months ago

On Linux, all native applications interpret the tilde as a synonym for the user's home directory

Not exactly true. The tilde is expanded by the shell (bash, sh, zsh etc). The applications don't need to map the tilde in command line arguments. Just like on UNIX the shell expands file wildcards, so the application does not need to enumerate directories, the shell did it for them.

$ cat test.c
#include <unistd.h>
#include <string.h>

int main(int argc,char **argv)
{
        write(1,argv[1],strlen(argv[1]));
        return 0;
}
$ cc test.c
$ ./a.out ~
/home/bythesea

No interpretation by the native app.

The shell does not expand if quoted

$ ./a.out '~'
~

and

$ ./a.out "~"
~

Now that is a gotcha, because shell does expand environment variables within double quotes but not within single quotes. So shell is doing special treatment for tilde, it is not treating it simply as $HOME.

Bash Manual - Tilde-Expansion

mklement0 commented 5 months ago

Thanks, @rhubarb-geek-nz - this has all been covered by my previous comment.

@MartinGC94: good point about the provider-specific nature of ~.

However, tab-completion was previously already provider-aware and would not expand ~ while the current location was, say, the Env: drive (for which a home location isn't defined) - that shouldn't change (or, rather: categorically leave ~ alone for all other providers).

(On a side note: On Unix-like system, when tilde expansion happens in the context of argument-passing to external programs, the current location's provider does not matter; it is always the value of $HOME that is used - which makes sense, given that the outside world only knows the file-system, and nothing about PowerShell's other providers.)

Given that among the built-in providers only the FileSystem one has a home location predefined, the slight loss of level of abstraction that comes from translating ~ to $HOME - if the FileSystem provider underlies the current location (which, however, is almost always the case, if I were to guess) - is still preferable to expanding to the full, literal home directory path.

To me, translating ~ to verbatim $HOME would be no more confusing than the old behavior of expanding to, say, C:\Users\jdoe.

That said, we could avoid this problem if we went with proposal 3, which is the cleanest solution. Personally, that is my preference, but the question is whether the concerns raised previously prevent it.

bradwilson commented 5 months ago

I'm surprised by the number of people who are suggesting that I'm asking for a new feature rather than lamenting of the removal of an old feature. The conversations about "confusions for users" is exactly why we're here: a "breaking" change was made and me (a user) is confused about why it was done and how I can undo it. This was clearly a feature I counted on and would like to have back, even if it is no longer the default behavior.

I would be fine with tab completion of ~ into $HOME (under the assumption that it would be evaluated before being sent to executables). It's also worth noting that in both 7.3.9 and 7.4.0 if I type dir $HOME/Down and press Tab it gets expanded to C:\Users\bradwilson\Downloads\ or /home/bradwilson/Downloads/ (depending on OS).

(I just deleted a bunch of stuff because I realized that echo in my pwsh examples was using the echo alias and not /usr/bin/echo, so my examples were incorrect.)

mklement0 commented 5 months ago

I'm surprised by the number of people who are suggesting that I'm asking for a new feature

I don't see anyone suggesting that.

The - intentional - change that was introduced in 7.4.0 discussed here has a real downside on Windows (as you've discovered) - lack of ~ resolution when calling external programs - and solving that is what the discussion is about.

And it sounds like you'd be happy with the new behavior as long as the latter problem is solved.

As for variable references in paths: as @MartinGC94 has stated, the plan is to also change the behavior in order to preserve variable references during tab-completion in a future PR.

However, if we go with the replace-~-with-verbatim-$HOME approach, that change would already need to include the preservation of variables, so as to prevent the scenario where iterative tab-completion then results in expansion of $HOME to its value.

jhoneill commented 5 months ago

@MartinGC94 I think this really needs to be rolled back to previous behaviour. But thanks for sharing the modified tabexpansion2

I've created a gist https://gist.github.com/jhoneill/322a77199350c76a5785f5406ea97bac with what I think is a more effective version although I'm not sure about modifying $inputScript and then changing theReplacementlength after calling CompleteInput()

However it does some things which I had in an argument completer previously and had to be bound to whatever cmdlet/function arguments I thought would benefit - a bit tedious

~ [tab] expands ~ to $userProfile and tab expands as if I had typed c:\users\james\ or whatever
~~ [tab] cycles through AdminTools, ApplicationData, CDBurning, CommonAdminTools, CommonApplicationData, CommonDesktopDirectory, CommonDocuments, CommonMusic, CommonOemLinks, CommonPictures, CommonProgramFiles, CommonProgramFilesX86, CommonPrograms, CommonStartMenu, CommonStartup, CommonTemplates, CommonVideos, Cookies, Desktop, DesktopDirectory, Favorites, Fonts, History, InternetCache, LocalApplicationData, LocalizedResources, MyComputer, MyDocuments, MyMusic, MyPictures, MyVideos, NetworkShortcuts, Personal, PrinterShortcuts, ProgramFiles, ProgramFilesX86, Programs, Recent, Resources, SendTo, StartMenu, Startup, System, SystemX86, Templates, UserProfile, Windows
~~My [tab] cycles through MyDocuments, MyMusic, MyPictures, MyVideos
~~MyDocuments\ [tab] with the trailing \ expands ~~MyDocuments to C:\users\james\documents
~~MyDocuments\p [tab] with the trailing \ expands ~~MyDocuments to C:\users\james\documents\p and cycles through things beginning with P
.... [tab] expands every . after 2 to ..\ so cd ... [tab] cycles through directories 2 levels up.
. [tab] and .. [tab] no longer need the trailing \
^ [tab] expands ^ to the PowerShell profile directory

I saw a way to fix #20765 at the same time so I've put that in at the end

Gist
tabExpansion2.Ps1
GitHub Gist: instantly share code, notes, and snippets.

mklement0 commented 5 months ago

To provide what I think is the proper framing for this issue:

Preserving abstractions in the user input - i.e. an initial ~, as well as variable references such as $HOME or $env:SystemRoot - is definitely the way forward, both for concision and portability and, secondarily and debatably, to minimize visual disruption.
- If there are associated challenges, let's try to address those - the ~-related one is the only one I'm aware of, and concrete solutions have been proposed - let's try to come to a shared understanding of what the best way forward is.
Separately - if there's a need - provide the old resolve-everything-to-literals behavior as an opt-in.
- A secondary challenge is that user will tend to look for such configurability to PSReadLine, even though it is technically related to the PowerShell engine itself - the same goes for #20765

237dmitry commented 5 months ago

this really needs to be rolled back to previous behaviour

On Windows

kilasuit commented 5 months ago

Can the change that caused this issue to become present please get linked to for traceability and also so that we can then also look to update changelogs, docs and any other supporting material as needed.

jhoneill commented 5 months ago

this really needs to be rolled back to previous behaviour

On Windows

I prefixed that with "I think", I'm a very infrequent user of Linux and and non Windows users may have a different view to users of Windows Given there is another issue about the behaviour of ~/path with native Linux tools I can see possible advantages to rolling back there too but that's not where I do my work so others are better places to say.

I will try frame something as "O'Neill's nth law" along the lines of "things which get into a product because they seemed like a good idea at the time to a developer are a disproportionate cause of problems". 7.4 seems to have a few of those, and I'd put colourization in the same group.

rhubarb-geek-nz commented 5 months ago

I will try frame something as "O'Neill's nth law" along the lines of …

Also known as The road to hell is paved with good intentions

237dmitry commented 5 months ago

non Windows users may have a different view to users of Windows Given there is another issue about the behaviour of ~/path with native Linux tools I can see possible advantages to rolling back there too

As a Linux and Windows user, I believe that using shortcuts like ~/path on Linux makes sense, as any utility or application will interpret it correctly as a system-wide designation for the user's home directory. The same cannot be said for Windows, so here we definitely need to return the previous behavior.

This is my personal opinion, like everything else I write here, because I'm just an user who cares about the future of PowerShell. Sorry that sometimes my statements seem unclear or even rude - poor knowledge of the language and often incorrect translation are the reason for this. I respect you all and value your opinion.

mklement0 commented 5 months ago

@kilasuit:

The PR is already linked in the initial post (#19489)
The original issue by @lzybkr that inspired the ~ part of the PR (with the variable-preservation part yet to come) is: #5350
The release notes and changelog only contain the following abstract description: "Improve path completion (#19489)"
It looks like about_Tab_Expansion wasn't updated to mention the change.

@237dmitry, thank you for your clarification that there may be a language barrier and for your expression of respect.

Just to recap the previous clarification: it isn't utilities (external programs) that interpret ~, it is the (POSIX-compatible) shell that up front resolves ~ to the value of $HOME and passes the result to utilities.

PowerShell (sensibly) chose to emulate this behavior - selectively, for calls to utilities (external programs) only - but currently only on Unix-like platforms. It could equally choose to do that on Windows too, which would solve the issue at hand. Again, it comes down to whether we think this would break anything.

To resume the framing debate:

It is clear that a change must be made.
- Concrete solutions have been proposed that preserve the spirit of the new behavior, namely here and here.
Reverting to the old behavior is one option - possibly selectively on Windows.
- It's important not to mistake a personal preference for the old behavior for a compelling reason that reverting is the only solution.
- If there are reasons (other than habit) to oppose the new behavior separately from the fixable problem it introduced, please state them as such.
- If the new behavior is kept and fixed, offering the old behavior as an opt-in is then an option.

237dmitry commented 5 months ago

Again, it comes down to whether we think this would break anything

I think this dilemma will be successfully resolved without affecting the already running version on the Unix platform. As you noted in another thread, there are problems with quotation marks, but quoting only part of the path is not that common in everyday interactive practice.

archer884 commented 5 months ago

As a Linux and Windows user, I believe that using shortcuts like ~/path on Linux makes sense, as any utility or application will interpret it correctly as a system-wide designation for the user's home directory. The same cannot be said for Windows, so here we definitely need to return the previous behavior.

This marks the second time in this discussion that the behavior of Linux shells has been misdescribed. To be clear, Linux programs DO NOT interpret ~/path. Period. They never even SEE the ~ part. The shell itself replaces ~ with a reference to the user's home directory, and it is the lack of this behavior on Windows that has rendered practically every utility I use non-functional in the past several days.

This is a serious breaking change, and the damage needs to be undone asap. If I enter a path beginning with ~ and press tab, I expect to see a real path. Period.

Edit: as far as HOW the damage is undone, I don't care. I primarily use Linux. If you make it work like Linux, that's fine. If you make it work like Windows, that is also fine. The problem right now is that it doesn't work at all. Whatever you do, however, do not pretend it's reasonable to expect me to type $HOME instead of ~. One of them is a little longer than the other, you see. :)

rhubarb-geek-nz commented 5 months ago

If I enter a path beginning with ~ and press tab, I expect to see a real path. Period.

The expansion of tilde to the equivalent of $HOME has nothing to do with "tab expansion" on Linux. I interpret "tab expansion" as part of the interactive user experience. In UNIX shells, sh, bash etc the tilde expansion is part of the script expansion when executing the command.

I am using "bash 5.1-6ubuntu1" on ubuntu 22.04, if I enter

$ echo ~/.ssh/auth

and then press tab, I get

$ echo ~/.ssh/authorized_keys

There is no need to replace the tilde with $HOME because it will already be substituted when I press enter.

It is also disingenuous to claim that PowerShell is treating tilde just like UNIX shells, it only handles one case, the simple "~" when calling external programs and ignores all the other cases, including when the tilde prefixes a user name.

Bash

$ echo ~
/home/bythesea
$ /usr/bin/echo ~
/home/bythesea
$ /usr/bin/echo ~root
/root

PowerShell

PS> echo ~
~
PS> /usr/bin/echo ~
/home/bythesea
PS> /usr/bin/echo ~root
~root

archer884 commented 5 months ago

If I enter a path beginning with ~ and press tab, I expect to see a real path. Period.

The expansion of tilde to the equivalent of $HOME has nothing to do with "tab expansion" on Linux. I interpret "tab expansion" as part of the interactive user experience. In UNIX shells, sh, bash etc the tilde expansion is part of the script expansion when executing the command.

Yes, thanks, but this issue has nothing whatsoever to do with UNIX shells. This issue relates to the interactive user experience on Windows, which is entirely broken as of version 7.4.

bradwilson commented 5 months ago

@archer884 I've updated the title to say "on Windows" so perhaps the discussion about Linux behavior (right or wrong) can end up in a new issue (if, for example, people want to file a bug against the fact that ~user doesn't work correctly).

mklement0 commented 5 months ago

Thanks, @bradwilson; note that such issues (pertaining to PowerShell's incomplete emulation of Unix-style tilde expansion on Unix-like platforms) not only have already been filed, but were previously linked to above.

From what I can tell, this comment still summarizes what's needed to resolve the issue at hand, and nothing new was added since.

Diogo-Rossi commented 5 months ago

I came here after some googling. This should be reported in the release notes What's New in PowerShell 7.4

To summarize: this works fine:

notepad $HOME/Documents/mydoc.txt

But this doesn't:

notepad ~/Documents/mydoc.txt

Maybe there is a way to treat ~ like $HOME

But @mklement0 already said all that in his complete comment.

jhoneill commented 5 months ago

To summarize: this works fine:
notepad $HOME/Documents/mydoc.txt

That evaluates the variable before launching the executable with the resulting string

But this doesn't:
notepad ~/Documents/mydoc.txt

That leaves ~ as a meta character which the executable may or may not understand, if you're calling notepad this is the same doing notepad ~/Documents/mydoc.txt from the Run box , or in cmd.

Maybe there is a way to treat ~ like $HOME

Set-PSReadlineKeyHandler    -Key Enter -ScriptBlock {
    try {
        $line   =  $cursor = $null
        [Microsoft.PowerShell.PSConsoleReadline]::GetBufferState([ref]$line, [ref]$cursor)
        if($line -match '(?<= | ''| ")~(?=[\\/]?)') { 
            [Microsoft.PowerShell.PSConsoleReadline]::RevertLine()
            [Microsoft.PowerShell.PSConsoleReadline]::Insert(($line -replace '(?<= | ''| ")~(?=[\\/]?)', $HOME))
        }
    }
    finally {[Microsoft.PowerShell.PSConsoleReadLine]::AcceptLine()}
}

Transforms ~ to home when you hit enter. (With a simple check to see if it looks like the start of a path)

rhubarb-geek-nz commented 5 months ago

Transforms ~ to home when you hit enter. (With a simple check to see if it looks like the start of a path)

I dislike the idea of invisible translation that occurs when in an interactive session that is different from what happens when it is part of a script.

Interpretation of ~ and $HOME should be consistent whether entered in by hand or part of a script.

This script expands the ~

#!/usr/bin/env pwsh

/usr/bin/echo ~

Which then leads to why do we need to invent new rules for the Windows handling of ~, why can't we use the same rules as on Linux?

jhoneill commented 5 months ago

Transforms ~ to home when you hit enter. (With a simple check to see if it looks like the start of a path)

I dislike the idea of invisible translation that occurs when in an interactive session that is different from what happens when it is part of a script.

Interpretation of ~ and $HOME should be consistent whether entered in by hand or part of a script.

This script expands the ~
#!/usr/bin/env pwsh

/usr/bin/echo ~
Which then leads to why do we need to invent new rules for the Windows handling of ~, why can't we use the same rules as on Linux?

I don't think we can because (AIUI) cat ~/temp.txt on linux doesn't need anything to be done to what is passed to cat for ~ to mean home directory. Windows Apps / APIs don't understand ~ : explorer et al - including file dialog boxes understand %UserProfile% but in in a limited way (on win 11 [win]+ r %USERPROFILE%\temp.txt opens the file in notepad
[win]+ r notepad %USERPROFILE%\temp.txt doesn't).

On Windows, therefore, it needs the shell to translate ~ I can't remember who told me years ago to keep it for the command line and not use it in scripts. And the Readline fix is just for people to have as a preference to say don't break if I use ~ as a path instead of $home

That's different from the bit of developer-driven-development which "broke" tab-expansion for many users' definition of break. (If I type start ~\pictures\slideshow [tab] at a prompt I want ~ to expand to C:\users\james and the slideshow to become slideshow.ppt. But if I type cd ~\p [tab] in a script I want the ~ to remain as ~ and p to expand to pictures so other people can use my script. unless I have ~\... it might better to repace ~\ with the the literal $home which will get expanded later. Certainly if I'm writing a script I don't wan tabbing to replace $home with my directory and break for anyone else who runs the script. At the prompt I don't care.

rhubarb-geek-nz commented 5 months ago

I don't think we can because (AIUI) cat ~/temp.txt on linux doesn't need anything to be done to what is passed to cat for ~ to mean home directory.

PowerShell on Linux has to do the translation. The operating system "exec" call does not expand variables or interpret ~, it is done by the shell.

not use it in scripts.

That would be the worst of all worlds, ~ working in Linux scripts, ~ working in Linux command line, ~ working in Windows command line but not in scripts.

How could you interactively test commands if the evaluation of a command is different depending on whether it was typed or from a script?

jhoneill commented 5 months ago

I don't think we can because (AIUI) cat ~/temp.txt on linux doesn't need anything to be done to what is passed to cat for ~ to mean home directory.

PowerShell on Linux has to do the translation. The operating system "exec" call does not expand variables or interpret ~, it is done by the shell.

I may be misinformed then. In any event prior to PowerShell, everything on linux seemed to work with ~ and nothing on Windows did. There are / were different expectations on the the different OSes.

Same with / on windows and linux PowerShell try cd Temp: ; cd / one moves you to the root of the "temp" drive and one to the root of everything. Making cd / change to the root of Temp: on linux would upset a good number of Linux users who expect it to go to / not temp:\ - but changing to C: on Windows would seem even more perverse.

not use it in scripts.

That would be the worst of all worlds, ~ working in Linux scripts, ~ working in Linux command line, ~ working in Windows command line but not in scripts.

Not quite what I said. We tell people not to write aliases in scripts and ~ is an alias for $Home, so the advice don't write "~/Documents" but "$home/Documents" seems fine to me. I'm sure someone gave me that advice 10 or 15 years ago :-)

How could you interactively test commands if the evaluation of a command is different depending on whether it was typed or from a script?

I'm not proposing that. I'm saying

For people who want the tab expansion behaviour they've had for long time, I have a gist which will undo the current aberration (as I see it)
For people who want to use ~ as alias on the command line (without the tab 'fix' and hitting tab) there is the option to have psreadline switch it for them

rhubarb-geek-nz commented 5 months ago

I'm not proposing that. I'm saying

For people who want to use ~ as alias on the command line (without the tab 'fix' and hitting tab) there is the option to have psreadline switch it for them

You may not be proposing that, but that is the effect. Interactive interpretation would be different to that from a script, so ~ will have one interpretation when typed, and a different one when in a script.

bradwilson commented 5 months ago

I may be misinformed then. In any event prior to PowerShell, everything on linux seemed to work with ~ and nothing on Windows did.

Which was fine, because you could rely on tab completion of ~ to get a full path for the executable. Hence why I opened this issue.

rhubarb-geek-nz commented 5 months ago

I am surprised that no one has mentioned the traditional use of tilde on windows as prefix for a temporary file

Why are hidden files with a leading tilde treated as super-hidden?

So not only is tilde a valid filename character, it already has precedence in Windows regarding its purpose

domsleee commented 5 months ago

For people who want to use ~ as alias on the command line (without the tab 'fix' and hitting tab) there is the option to have psreadline switch it for them

I would be one of these people 💯 #20402 Although I would only expect tilde expansion to only occur without quotes, so the proposed flag PSNativePSPathResolution would not be considered a solution to the tab completion issues described here 👍

I think the current completion doesn't make sense on nix either, because you can't reuse the completions in native executions. Consider this case, from the completion where I'm trying to open ~/my folder/a.txt, using code ~/my<tab> (on mac)

code '~/my folder/a.txt' # this won't work

The old completion from 7.3.9 (code '/Users/user/my folder/$') was reasonable behaviour, 7.4.0 isn't 👍 I'm not saying reverting is the only way, just saying this is a reason to change to the 7.4.0 unix behaviour as well.

rhubarb-geek-nz commented 5 months ago

Consider this case, from the completion where I'm trying to open ~/my folder/a.txt, using code ~/my<tab> (on mac)

code '~/my folder/a.txt' # this won't work

Of course it shouldn't, you quoted the tilde. Quoting tilde prevents expansion in bash

$ echo ~
/home/bythesea
$ echo '~'
~

domsleee commented 5 months ago

Of course it shouldn't, you quoted the tilde. Quoting tilde prevents expansion in bash

Yeah, I have no issue with the tilde expansion behaviour on mac, only the tab expansion. The problem is the tab expansion adds the single quotes around the tilde for folders with spaces, so code ~/my<tab> turns into code '~/my folder/'.

To be clear I'm suggesting a reason the new behaviour is undesirable for mac, from https://github.com/PowerShell/PowerShell/issues/20750#issuecomment-1825707096:

If there are reasons (other than habit) to oppose the new behavior separately from the fixable problem it introduced, please state them as such.

pwsh 7.4.0 (mac):

PS /Users/user> (TabExpansion2 'code ~/my').CompletionMatches[0].CompletionText
'~/my folder'

pwsh 7.3.9 (mac):

PS /Users/user> (TabExpansion2 'code ~/my').CompletionMatches[0].CompletionText
'/Users/user/my folder'

rhubarb-geek-nz commented 5 months ago

I have no problem with the 7.3.9 version, it is absolutely unambiguous.

When bash tab-expands a filename with spaces it does not quote the entire string, but replace the space with backslash-space.

$ echo > "my file.txt"
$ cat ~/my\ file.txt

mklement0 commented 5 months ago

@domsleee, good find on the broken tab-completion of ~/my to '~/my folder', and thanks for tackling a PR for ~ expansion on Windows too (https://github.com/PowerShell/PowerShell/pull/20402).

The broken tab-completion is conceptually related to #20754 (the fact that even a properly specified ~/'my folder' doesn't shell-expand to the equivalent of $HOME/my folder); I've created a separate issue for it:

20851

mklement0 commented 5 months ago

@rhubarb-geek-nz, ~ as a verbatim character at the start of the name of a file to signal its temporary nature on Windows is a convention (except that it impacts File Explorer's display behavior, selectively) that is conceptually unrelated to treating ~ at the start of a path as the user's home directory.

The latter POSIX-shell shell expansion doesn't exist in cmd.exe, which, in fact, doesn't support any such expansions (except a limited form of parameter/variable expansion), and - with the exception of the equivalents of command substitution and arithmetic substitution - neither does PowerShell, except for the emulations discussed below.

However, in the context of PowerShell, ~ does have long-established analogous meaning, but with crucial differences:

Since there is no shell expansion taking place (i.e. up-front replacement of ~ with the value it represents), it is the target command that sees the ~, and as noted, PowerShell provider cmdlets interpret a path-initial ~ as referring to the their home location, if defined. For the FileSystem provider, it is defined, namely as the value of $HOME, i.e. the user's home directory, as on Unix.
This has two implications:
- When targeting a provider cmdlet, it doesn't matter whether the ~ is quoted or not; e.g., Get-Item ~, Get-Item '~', Get-Item `~, and Get-Item "~" are equivalent.
- Other PowerShell commands, i.e. those that aren't provider cmdlets receive ~ verbatim (too); so that Write-Output ~ echoes ~ verbatim.

As noted, on Unix-like platforms PowerShell chose to emulate two frequently used shell expansions, tilde expansion and filename expansion (globbing) - of necessity, for compatibility, the POSIX-shells-only distinction between unquoted (do expand) and quoted use of ~ (do not expand) had to be implemented too, even though it is foreign to PowerShell.

With no analogous features existing in cmd.exe, no emulations were implemented on Windows:

Implementing filename expansion on Windows too is not an option, as it would break backward compatibility in a manner that is likely to break existing code.
By contrast, implementing tilde expansion on Windows too - see #20402 - makes perfect sense, given the long-established meaning of ~ in PowerShell, and for cross-platform consistency.
- The only caveat is that it too is technically a breaking change, as it changes the meaning of unquoted arguments that start with ~\ or ~/ in existing code when passed to native programs on Windows; e.g. cmd /c echo ~\foo currently echoes verbatim ~\foo but would then echo, say, verbatim C:\Users\jdoe\foo.
- As noted, the hope is that this is unlikely enough to exist in the real world and therefore constitutes an acceptable bucket 3 change.

Once #20402 is implemented, the problem that prompted creation of this issue will go away.

jhoneill commented 5 months ago

I'm not proposing that. I'm saying

For people who want to use ~ as alias on the command line (without the tab 'fix' and hitting tab) there is the option to have psreadline switch it for them

You may not be proposing that, but that is the effect. Interactive interpretation would be different to that from a script, so ~ will have one interpretation when typed, and a different one when in a script.

So, people should be forbidden from customizing their environments to work that way ?

Notepad ~/myfile fails in a script, so why should someone not create an autocorrect rule which fixes that ? Scrolling up will show the corrected version, so will the history. They can copy the command line version into a script it will work. I really don't see anything wrong with autocorrect in psreadline: the ability to run something when enter is pressed has been there for ages (I had it trap a pseudo command hh and display a picklist of history items - hh wouldn't work in a script) so it's a fair bet I'm not the only person who's thought of using it for autocorrect.

My view might be a bit "fringe" but if I type cd ~~[tab] my environment is set to cycle through well know directories and cd ~~\commonDocuments\[tab] will cycle through what's in c:\users\public\Documents. cd ...[tab] expands each extra . after two to ..\ but cd ... will work (I have cd as a function with custom transformer and completer for the path). This is emphatically not script-safe (It will fail on anyone else's system), but I can live with some command lines being that way. Other views are available - what's efficient for me might horrify you :-)

Of course the cd you do at a prompt by have a different effect in a script that runs from a different directory, so most of us learn that a script shouldn't make assumptions about the current directory :-)

We have a proposal to make it work by expanding ~/ when notepad is called. Yipee! Except when someone writes ~/ in a future script and someone else runs that script on Windows PowerShell it breaks. So now we have to tell people if they want their code to run everywhere use $home instead. {No, that's not a reason to leave it broken, and my autocorrect is just a band-aid until it is fixed - for those who really want it}

rhubarb-geek-nz commented 5 months ago

I really don't see anything wrong with autocorrect in psreadline: the ability to run something when enter is pressed has been there for ages

The problem with ~ being invisibly interpreted as $HOME in PSReadLine but not when executing a script leads to the problem where you can't see why your script doesn't work. You type the line at the command prompt and it works, you run the script and it doesn't work even though the text is identical. So in effect you would have two different interpreters with different rules both claiming to be PowerShell.

jhoneill commented 5 months ago

I really don't see anything wrong with autocorrect in psreadline: the ability to run something when enter is pressed has been there for ages

The problem with ~ being invisibly interpreted as $HOME in PSReadLine

And if it were invisible this would be a problem, but what happens is the text between the prompt and the cursor changes

I hit enter and

but not when executing a script leads to the problem where you can't see why your script doesn't work.

Well, you can't see run notepad ~/myfile to find out why it doesn't work because every time you try the line changes before your eyes, giving a you a strong hint to change what is in the script. And because this is a personal choice for users who want it, if they regularly use ~ in scripts (which I'm saying "don't do") they need to take responsibility for handling the difference.

in effect you would have two different interpreters with different rules both claiming to be PowerShell.

No it's an autocorrect. imagine if some of the light bulbs in VS code just changed sort to Sort-object (to deal with sort not being an alias on linux and running the external executable), or there was one for ~/ paths which didn't say "don't use this" but changed it. I really don't see it as any different to word converting (c) to a copyright symbol. Bottom line its there if people want it as a workaround, but it's not the universal answer.

rhubarb-geek-nz commented 5 months ago

The problem with ~ being invisibly interpreted as $HOME in PSReadLine

And if it were invisible this would be a problem, but what happens is the text between the prompt and the cursor changes

But it is too late. You type X and press enter, it changes it to Y, prints and executes Y even if that is not what you wanted.

The tab-expansion allows you to see the replacement before you commit with enter. Your suggestion is to bypass that validation and execute something different to what the user sees before they press enter.

You are adding a third level of translation that no-one is expecting and is not implemented any where else. I am happy with code-completion and tab-expansion but they all allow you to see what will be executed before you commit.

This last minute tilde swap does not meet the "least astonishment" principle.

domsleee commented 5 months ago

@mklement0, thanks for raising that issue.

For people who want to use ~ as alias on the command line (without the tab 'fix' and hitting tab) there is the option to have psreadline switch it for them

@jhoneill, is your proposal to make the "enter autocorrect" a built-in option is PSReadLine? I would much prefer tilde expansion described in #20402, since it would allow tilde expansion in scripts. A lot of bash scripts use it, so it would be nice for linux users coming from bash as a default in a future release of powershell (assuming all goes well and there isn't some legacy windows code that relies on the tilde not expanding).

If "enter autocomplete" was implemented as an option in PSReadLine, I wouldn't expect it to be enabled by default, since it would surprise users when their prompt changed after pressing enter (I believe @rhubarb-geek-nz was saying this).

Question - Would the enter autocorrect be useful if #20402 was implemented? Any advantages compared to #20402?

PowerShell / PowerShell