installation fails if PATH is too long

phanirithvij commented 3 years ago

Purpose of the issue

[ ] Bug report (encountered problems/errors)
[ ] Feature request (request for new functionality)
[ ] Question

Version Information

1.3.17 (2020-12-23)
191012

Edition Windows 10 Home Single Language
Version 20H2
Installed on    ‎8/‎7/‎2020
OS build    19042.804
Experience  Windows Feature Experience Pack 120.2212.551.0

Description of the issue

When installing for the first time if PATH is too long it fails to run somewhere in the middle with Input like is too long, the syntax is incorrect error before it reaches the git-for-windows initial setup

My PATH has this

``` C:\Users\Rithvij\scoop\apps\python27\current\scripts;C:\Users\Rithvij\scoop\apps\adoptopenjdk-hotspot-jre\current\bin;C:\Users\Rithvij\scoop\shims;D:\Software\mpv-x86_64-20200315-git-281f5c6;C:\Users\Rithvij\AppData\Local\Programs\Python\Python38\Scripts\;C:\Users\Rithvij\AppData\Local\Programs\Python\Python38\;C:\Users\Rithvij\.cargo\bin;C:\Users\Rithvij\AppData\Local\Microsoft\WindowsApps;D:\Software\flutter\flutter\bin;D:\Software\cmder;D:\Software\flutter\flutterm\bin;C:\Program Files (x86)\Microsoft Visual Studio\Installer;D:\Software\ZIPS and Installers\weebp-0.6.1-windows-x64;D:\Software\mpv-x86_64-20181002;C:\Program Files\Docker Toolbox;C:\Users\Rithvij\AppData\Roaming\Pub\Cache\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build;D:\Software\ZIPS and Installers\nim-1.0.0\bin;C:\Users\Rithvij\.nimble\bin;D:\Software\node;D:\Software\micro-1.4.1;C:\xampp\php;D:\Software\scrcpy-win64;C:\Users\Rithvij\AppData\Local\Programs\Microsoft VS Code\bin;D:\Software\LOVE;C:\Program Files\Mercurial\;C:\Users\Rithvij\AppData\Roaming\Python\Python38\Scripts;C:\Users\Rithvij\AppData\Local\bin\NASM;C:\libjpeg-turbo-gcc\bin;C:\msys64\mingw64\bin;D:\Projects\golang;C:\tools\dart-sdk\bin;D:\Software\depot_tools;D:\Software\sqlite\sqlite-tools-win32-x86-3300100;D:\Software\processing-3.5.4;D:\Software\flutter\flutterm\.pub-cache\bin;D:\Software\apktool;C:\Users\Rithvij\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Rithvij\AppData\Roaming\npm;C:\Users\Rithvij\.local\bin;C:\Users\Rithvij\go\bin;C:\Users\Rithvij\.dotnet\tools;D:\Software\ZIPS and Installers\upx-3.96-win64;D:\Software\LOVE2D;D:\Software\vips-dev-w64-all-8.10.2\vips-dev-8.10\bin;C:\msys64\usr\bin;D:\Software\PostgreSQL\13\bin;D:\Software\navidrome_0.38.0;D:\Downloads\prometheus-2.23.0.windows-amd64;D:\Software\bin;C:\Program Files\VMD;D:\Software\Fiddler;%USERPROFILE%\.dotnet\tools;D:\Downloads\memcached-1.6.9\cygwin\x64;D:\Games\MineKraft;D:\Downloads\gohls-windows-amd64-snapshot.tar;D:\Software\ffmpeg-1\ffmpeg\bin;D:\Software\Google\Cloud SDK\google-cloud-sdk\bin ```

Also my system PATH is long as well but not sure if this causes any issues

``` D:\Software\ImageMagick-7.0.9-Q16;C:\Program Files\Microsoft MPI\Bin\;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp;C:\Program Files\AdoptOpenJDK\jdk-11.0.5.10-hotspot\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\ProgramData\chocolatey\bin;D:\Software\Lua\5.1;D:\Software\Lua\5.1\clibs;C:\tools\dart-sdk\dart-sdk\bin;C:\Program Files\Calibre2\;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.4.0\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\TDM-GCC-64\bin;D:\Projects\fun\lua\lua-5.3.5\src;D:\Software\Erlang\bin;D:\Software\Elixr\bin;C:\Users\Rithvij\.mix\escripts;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;D:\Software\node14\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;D:\Software\vcpkg\installed\x64-windows\bin;D:\Software\vips-dev-w64-all-8.10.2\vips-dev-8.10\bin;D:\Software\vcpkg;C:\Users\Rithvij\go\src\github.com\tidbyt\go-libwebp\libwebp-1.1.0-windows-x64\bin;C:\Users\Rithvij\go\src\github.com\tidbyt\go-libwebp\libwebp-1.1.0\output\release-dynamic\x64\bin;C:\Program Files\filebrowser;D:\Software\Heroku\bin;C:\Program Files\MongoDB\Server\4.2\bin;D:\Downloads\PSTools;C:\Program Files\dotnet\;D:\Software\Oracle\VirtualBox; ```

Additional Notes:

I'm not sure where the issue occurs I tried debugging with @echo on in init.bat and following files and found it failed when it printed a lot of PATH variables like if NOT``
I'm not sure how long PATH should be for it to occur but I've added my PATH when it occurred
My workaround was to remove users PATH then run cmder, close, then fill the user's PATH back, then subsequent runs worked normally

daxgames commented 3 years ago

Can you provide a screenshot of the exact problem?

chrisant996 commented 3 years ago

Also, does this happen while running an installer pegram, or does it happen while starting Cmder for the first time after it's installed?

phanirithvij commented 3 years ago

I don't know if there's any installers but I got the cmder.7z with git-for-windows from the official site which serves from GitHub releases.

Then I added the extracted folder to path.

It occurs after opening cmder.exe for the very first time and it occurs in the subsequent runs as well until it's resolved.

I didn't take a screenshot before I fixed it but it just says

The input line is too long
The syntax of the command is incorrect

Above the prompt.

I will try to re-create the issue and get a screenshot if its really needed.

On Sun, 28 Feb 2021, 00:07 Chris Antos, notifications@github.com wrote:

Also, does this happen while running an installer pegram, or does it happen while starting Cmder for the first time after it's installed?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cmderdev/cmder/issues/2485#issuecomment-787116766, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHCBL6X7ATLMMX7MJH4JZJ3TBE3YLANCNFSM4YJ5SSSQ .

phanirithvij commented 3 years ago

Screenshot 2021-02-28 111210

chrisant996 commented 3 years ago

The input line is too long followed by The syntax of the command is incorrect happens when CMD is asked to run a command line that exceeds 8191 characters (which is its maximum allowed line length).

The cited user PATH is 2090 characters.
The cited system PATH is 1959 characters.
The PATH env var is the concatenation of the user PATH and the system PATH, so the PATH variable is 4050 characters (2090 + semicolon + 1959).

I'm pretty sure the error The input line is too long is happening on this line: https://github.com/cmderdev/cmder/blob/a66b8b30355230b51c383765376869623521bcc9/vendor/lib/lib_path.cmd#L115

That line has two copies of the PATH variable on the same command line. So CMD will fail to run the line if the PATHs are over around 4040 characters or so.

One way the script could compare them could be to write the OLD_PATH and PATH variables to separate temporary files, and then use fc to compare the temporary files. That's the first way I thought of, but I'm sure there are many other alternatives as well. For example fc /b tmpfileA tmpfileB 2>nul 1>nul & if errorlevel 1 echo Files are different.

However, it looks like enhancepath is used several times every time a new tab is opened that uses init.bat. So for speed it's probably important to avoid writing to temp files unless the PATH is long. So maybe first test the length, and only use the temp file approach if the PATH length is over some threshold. E.g. if "%VAR:~0,4000%" == "%VAR:~0,4001%" ( echo var is short ) else ( echo var is long )

daxgames commented 3 years ago

@phanirithvij

I have doubts that this is the cause though because if it were it would probably happen on every launch after the users path were added back in following the initial launch.

When Cmder full first starts after extraction it runs a script included with git for windows exactly one time that deletes itself. I wonder if the error is coming from the %cmder_root%\vendor\git for windows\post-install.bat script.

You could prove this by downloading cmder_mini, extracting and launching cmder.exe without removing your user path as it does not contain nor run this script.

If @chrisant996 is correct you could add a /f to the Conemu task that launches Cmder. This setting will cause Cmder skip the logic @chrisant996 pointed to.

The only way to fix this on initial run would be to use /f as a default setting and I can 't see that happening because it might cause more issue reports of changed functionality that it would fix.

I just don't see that many people having a path statement so large that when doubled is longer than 8191 characters.

chrisant996 commented 3 years ago

The only way to fix this on initial run would be to use /f as a default setting

Just to double check:

Was my suggested fix unclear, or does it seem not worth implementing?

It would fix the issue, so it seems like another way besides making /f a default setting. But I can understand postponing it as low priority.

I just don't see that many people having a path statement so large that when doubled is longer than 8191 characters.

I think it's pretty uncommon, yes. It can also slow down launching programs, since the longer the PATH, the longer it can take the OS to search all the folders trying to find the program being launched.

However, I can also understand wanting to be able to have a long PATH.

I have a lot of other things in my work queue, but maybe if I get a chance I might submit a pull request for the suggested fix. Not anytime in the near future, though, sorry (too many of my own projects are wanting my time).

daxgames commented 3 years ago

@chrisant996 we are both writing and re-editing and missing things. See my latest and I will look at your message again.

daxgames commented 3 years ago

@chrisant996 yes it looks like your suggestion would solve the issue assuming your initial diagnoses is correct.

chrisant996 commented 3 years ago

I'll take a look at the git for windows post_install.bat script. Maybe it has a similar issue.

Also, the line I cited from lib_path.cmd only gets executed if the script modifies the PATH, right? Maybe somehow the PATH only needs to be modified the first time.

chrisant996 commented 3 years ago

When Cmder full first starts after extraction it runs a script included with git for windows exactly one time that deletes itself. I wonder if the error is coming from the %cmder_root%\vendor\git for windows\post-install.bat script.

I don't think so. That script essentially just launches bash to run some bash shell scripts. It doesn't reference %PATH% and all the real work is done inside bash shell scripts, which will not produce the cited error message.

I think the reason the error only happens the first time is because:

Although git for windows may have already run, it updated the PATH in the registry, and the Cmder process was launched before the PATH got updated in the registry, and therefore in Cmder git for windows is still not yet on the PATH.
So the init.bat script tries to add it to the path via enhancepath, when then hits the error due to the long PATH variable.
The next time Cmder is launched, it picks up the current env vars from the registry, which now include git for windows in the PATH, and thus init.bat doesn't need to add it anymore, and thus the error isn't encountered.

So, another possible workaround/fix might be to explicitly refresh PATH from the registry. If that approach is used then here are two gotchas to consider:

The PATH variable is the concatenation of the system PATH + ; + the user PATH.
The PATH variable in the registry can contain embedded environment variable references if the type is REG_EXPAND_SZ (which is the default), and they need to be expanded.

phanirithvij commented 3 years ago

Awesome, seems like you guys were able to reproduce this. I'll do anything I can to help (just letting you know I don't know batch file debugging)

daxgames commented 3 years ago

@phanirithvij I have not reproduced anything. I have not really been in a place to try. I have been responding to this thread from my phone with no access to a computer to try anythung. Could you try the cmder_mini suggestion ans reporr back?

@chrisant996 Neither the git for windows embedded in cmder nor cmder edits the path in the registry that I am aware of so the same path changes/adsitions would occure on eaxh cmder session launch.

chrisant996 commented 3 years ago

The git for windows installer indeed updates the PATH in the registry. That's the only way to make it take effect permanently, and it certainly did that on my computer.

But perhaps a different install operation occurs when Cmder triggers the install.

daxgames commented 3 years ago

@chrisant996 I am not truing to argue with you or the end user and as I said earlier I have not had the opportunity to REALLY look at it.

You are 100% correct the Git installer does indeed edit the path. The Git for Windows in Cmder full as I stated in my reply does not because it is a portable install embedded in Cmder.

The user said nothing about installing Git for Windows after the first Cmder run SO I never even considered that to be a factor in the issue.

If the only change the user makes to make this work is to remove his user path, launch cmder and allow initial run to complete, then reapply his original user path as stated in the original report the the issue SHOULD reoccur on each subsequent Cmder launch because Cmder makes zero changes to the system path that are retauned between runs.

The only difference at this point between the initial launch and subsequent launches is the post-install.bat is only executes on a completly succesful initial launch.

chrisant996 commented 3 years ago

@chrisant996 I am not truing to argue with you or the end user

I think we're all just trying to figure out what's going wrong and where. 🙃

The Git for Windows in Cmder full as I stated in my reply does not because it is a portable install embedded in Cmder.

Ah so there is indeed a separate method of installation.

The only difference at this point between the initial launch and subsequent launches is the post-install.bat is only executes on a completly succesful initial launch.

Maybe. Experimentation will be necessary to finish tracking it down, I think.

daxgames commented 3 years ago

I think we're all just trying to figure out what's going wrong and where. 🙃

I know I just wanted to clarify my continued responses.

It is REALLY wierd if the end user truly did exactly what he said because Cmder calls itself a portable app in that it makes no changes to the system. You download a zip, extract it and launch cmder.exe. There is no install process executed for any component of Cmder.

The session init process should be the same for every run whether from the cmder.exe and for each session started via the Conemu session menu because Conemu has the env that the cmder.exe launcher had when it was launched and it should not contain any prior modifications from any previous Cmder sessions.

Maybe. Experimentation will be necessary to finish tracking it down, I think.

Yup! I am at a computer now so I will play with it.

daxgames commented 3 years ago

I added the majority of the users user path and system path to the coresponding paths in my current system. Total combined length 4505 characters. This is longer than the concatenated path provided by the user.

Edited:

My initial test of this was not a good test. I already had a populated %cmder_root%\config folder or I did not restart cmder so my path was still too short. I did not reproduce the error in this situation.

I recreated the %cmder_root%\config folder as it would be on initial launch and now I have reproduced the error:

Investigating now.

daxgames commented 3 years ago

The problem is not just the comparison that @chrisant996 pointed out earlier but it is the entire if block since it is interpreted as one command by cmd.exe.

Even with the solutions suggested the problem still exists, removing theif and leaving the statements that are inside the if () has the same issue. Commenting the entire if block including its content still has the same issue

Removing the entire if block including its content or exiting the function before getting to the block are the only ways to resolve the issue that I have found OR a shorter path variable.

I am unsure the benefit of fixing this outweighs the benefit of having the debug statements given the unlikely event a user has a path this long and how often we ask for debug output to help solve user issues.

chrisant996 commented 3 years ago

The problem is not just the comparison that @chrisant996 pointed out earlier but it is the entire if block since it is interpreted as one command by cmd.exe.

CMD does not treat an entire if block as a single command line. If you're seeing it seem to behave as though they're interpreted as a single command line, then something else is going wrong.

The following is treated as at least 5 separate different command lines, each of which can use up to 8191 characters without encountering an error:

if "%ABC%" == "%DEF%" (
    echo { insert 6000 characters here }
    echo { insert 6000 characters here }
    echo { insert 6000 characters here }
    echo { insert 6000 characters here }
)

When I run that script (after inserting 6000 characters as stated) it works correctly, without any errors.

Even with the solutions suggested the problem still exists, removing theif and leaving the statements that are inside the if () has the same issue. Commenting the entire if block including its content still has the same issue

How were the lines commented out?

If it was done by inserting rem at the beginning of each line, that won't successfully comment out the line:

Inserting rem at the beginning of each command line would make each command line even longer, further exacerbating the problem.
CMD can't recognize that it's a comment until after it parses the command line, and CMD will not even begin to parse a command line that is longer than 8191 characters.

Removing the entire if block including its content or exiting the function before getting to the block are the only ways to resolve the issue that I have found OR a shorter path variable.

The following technique solves the problem in my local testing. For illustration purposes, this proof of concept uses hard-coded tempfileA and tempfileB temp file paths, but that part could benefit from being cleaned up a bit.

rem -- Test whether OLD_PATH or PATH are long.
if not "%OLD_PATH:~0,3000%" == "%OLD_PATH:~0,3001%" goto :toolong
if not "%PATH:~0,3000%" == "%PATH:~0,3001%" goto :toolong
if not "%OLD_PATH%" == "%PATH%" goto :changed
goto :notchanged

:toolong
echo %OLD_PATH%>tempfileA
echo %PATH%>tempfileB
fc /b tempfileA tempfileB 2>nul 1>nul
if errorlevel 1 ( del tempfileA & del tempfileB & goto :changed )
del tempfileA & del tempfileB
goto :notchanged

:changed
echo OLDPATH not equals PATH
goto :eof

:notchanged
echo OLDPATH equals PATH
goto :eof

I am unsure the benefit of fixing this outweighs the benefit of having the debug statements given the unlikely event a user has a path this long and how often we ask for debug output to help solve user issues.

The debug statements are worth keeping, for sure. It's definitely possible to keep them and also fix the issue, and it doesn't involve very much code. I apologize that the example code I've been sharing hasn't been clear! Hopefully the suggestion above helps clarify my suggestion.

daxgames commented 3 years ago

I retested again and somehow got different results, I don't know why. Initially I was even getting the error when leaving just the debug statements.

I implemented the code from @chrisant996 above and it seems to have resolved the error.

New build here.

Initial launch example and current path length:

daxgames commented 3 years ago

Something is still wierd though. Still working on it.

daxgames commented 3 years ago

This build is good.

phanirithvij commented 3 years ago

I have doubts that this is the cause though because if it were it would probably happen on every launch after the users path were added back in following the initial launch.

Yeah, weirdly I didn't get it after I added the path back for the very first time. ~I remember rebooting before testing it, does it change things? I'll try rebooting again and edit this.~ rebooting does nothing issue still occurs

I got it again after I ran cmder subsequently as you said.

When Cmder full first starts after extraction it runs a script included with git for windows exactly one time that deletes itself. I wonder if the error is coming from the %cmder_root%\vendor\git for windows\post-install.bat script.

You could prove this by downloading cmder_mini, extracting and launching cmder.exe without removing your user path as it does not contain nor run this script.

I did as you said and it error does occur with cmder_mini

Generating clink initial settings in "D:\Downloads\cmder_mini\config\settings"
        1 file(s) copied.
Additional *.lua files in "D:\Downloads\cmder_mini\config" are loaded on startup.
The input line is too long.
The syntax of the command is incorrect.

D:\Downloads\cmder_mini
λ

Sorry for responding late, after you've possibly fixed it 😢 .

This build is good.

~I'm gonna test it and let you know if it works in my system.~

Works!!

Generating clink initial settings in "D:\Downloads\cmder\config\clink_settings"
        1 file(s) copied.
Additional *.lua files in "D:\Downloads\cmder\config" are loaded on startup.
Creating initial user_aliases store in "D:\Downloads\cmder\config\user_aliases.cmd"...
        1 file(s) copied.
Running Git for Windows one time Post Install....
"running post-install"
'C:\WINDOWS\system32\drivers\etc\hosts' -> '/etc/hosts'
'C:\WINDOWS\system32\drivers\etc\protocol' -> '/etc/protocols'
'C:\WINDOWS\system32\drivers\etc\services' -> '/etc/services'
'C:\WINDOWS\system32\drivers\etc\networks' -> '/etc/networks'
The batch file cannot be found.
Creating user startup file: "D:\Downloads\cmder\config\user_profile.cmd"
        1 file(s) copied.

D:\Downloads\cmder
λ

cmderdev / cmder