microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.08k stars 797 forks source link

access Linux symlinks from \\wsl$ #5118

Open trentis opened 4 years ago

trentis commented 4 years ago

Is your feature request related to a problem? Please describe. I want to be able to open a symlink from the '\wsl$\Ubuntu' directory. The reason for this Is I'm writing a program that would read an elf binary and import the required objects to run it inside the directory that the binary is in.

This is from a virtualization perspective the kernel we use is linux based and Im purely looking at a way to improve development of apps on Windows. The only thing I want to do with the files is be able to read them and copy to a location without any changes. But most elf objects link to shared libraries that are symlinks but Windows doesn't seem to know what to do with that file. Any sort of open on that file results in system could not find the name specified. I assume this is because the explorer is attempting to open the symlink but its at a path it doesn't understand like '/lib/64' instead of '\wsl$\Ubuntu\lib\64'.

Describe the solution you'd like Maybe on the file open in the windows explorer prefix it with the location of the symlink. '/lib/64' turns into opening '\wsl$\Ubuntu\lib\64' I imagine this would be a big task probably.

Describe alternatives you've considered I could probably resolve this from the perspective of installing another daemon on my subsystem and just do the calls there. I'm mainly querying about this so I can improve UX of our application and the user doesn't have to install two of the same thing.

Is there a use case where we could set the LD_LIBRARY_PATH on our Windows machine to point to one of the sub system shares?

Additional context Basically want to read the ldd of an elf binary copy the objects into a windows directory. Proceed to do that recursively for everything that binary requires.

dslijepcevic commented 2 years ago

You can create a Samba share inside WSL2 and mount it as a network drive on Windows side and then symlinks will work fine. Still waiting for 9p server to support this under \wsl$, though.

audrenbdb commented 2 years ago

Has anyone found a temporary solution to this? Should I "manually" sync those folders?

I did.

My problem was the following: my IDE (jetbrain) opened in windows was not able to process properly symlinks in ubuntu.

What I did was installing my IDE in ubuntu directly and open it thanks to the native wsl gui

After you install your favorite editor, you can create a .desktop entry as described here: https://youtrack.jetbrains.com/issue/DOC-13496/Documentation-about-wsl2-gui-integration

And that's it, everything is running flawlessly :)

Yolo-plop commented 2 years ago

Has anyone found a temporary solution to this? Should I "manually" sync those folders?

I wrote a script 3 lines bash script that watches for changes in a folder and does a rsync to where i use the libraries.

What I did was installing my IDE in ubuntu directly and open it thanks to the native wsl gui

It does seem like a good idea but requires Windows 11 and my work computer is not compatible. I'm hoping to get a new machine in a near future and try this solution.

GrzegorzKlimek commented 2 years ago

I just use bind mount, instead of symlinks and it works fine. What is bind mount -> link

IlanVivanco commented 2 years ago

I wrote a script 3 lines bash script that watches for changes in a folder and does a rsync to where i use the libraries.

@Yolo-plop Could you share this script?

What I did was installing my IDE in ubuntu directly and open it thanks to the native wsl gui

@dnfclas I'm running VSCode and they already run a VSCode server from the Ubuntu side, but the app I'm having issues with, needs to be on the Windows side, so I cannot use it as you mentioned.

I just use bind mount, instead of symlinks and it works fine. What is bind mount -> link

@GrzegorzKlimek Could you elaborate on how you do this? When I tried to mount it, I got no errors, but no files were mounted either.

Yolo-plop commented 2 years ago

@Yolo-plop Could you share this script?

# apt install rsync
# apt install inotify-tools

while inotifywait -r -e modify,create,delete,move /path/to/lib/dist ; do
    rsync -r /path/to/lib/dist   /path/to/project/node_modules
done
tomty89 commented 2 years ago

Probably you want to at least add -t? (Although I'm not sure how well that works when a Windows filesystem is involved. Is it though? Or are you guys talking about syncing within the ext4 vhdx?) There's also the alternative -c, if you can afford it.

Also see --delete and friends.

IlanVivanco commented 2 years ago

You can create a Samba share inside WSL2 and mount it as a network drive on Windows side and then symlinks will work fine. Still waiting for 9p server to support this under \wsl$, though.

@dslijepcevic I was able to do this, and it worked, but now the WSL IP is constantly changing upon restart, and I cannot have a reliable drive mounted on the windows side because of it.

dslijepcevic commented 2 years ago

You can create a Samba share inside WSL2 and mount it as a network drive on Windows side and then symlinks will work fine. Still waiting for 9p server to support this under \wsl$, though.

@dslijepcevic I was able to do this, and it worked, but now the WSL IP is constantly changing upon restart, and I cannot have a reliable drive mounted on the windows side because of it.

@IlanVivanco Yes, that is unfortunate, but here is how you can obtain the IPs on each side:

Then after you start the Samba service on WSL2 side you can map a drive letter using net use <drive-letter>: \\<wsl2-ip>\<path> from, say, a scheduled task/.cmd script on the Windows side or from a WSL2 shell initialization script.

lanoux22 commented 2 years ago

First you say yeah they have done something cool then you say, okay, windaube, like always...

going back to debian with dualboot for adobe

badreddineelouarigli commented 2 years ago

+1 need this issue fixed

ghost commented 2 years ago

+1. I'm using PNPM package manager for Node.js which creates symlinks in node_modules/. WebStorm, which is configured to run prettier and eslint every time the file is saved, fails to do it as it can't find the installed package through the symlinks.

ncwp commented 2 years ago

+1 How is this still not fixed 2 years later?

DariuszLuber commented 2 years ago

+1 - Problem running Jest tests - it cannot find binary for jest

rdemicheli commented 2 years ago

+1 need this issue fixed

debel27 commented 2 years ago

Can you please upvote the issue rather than commenting it with a +1 ?

I know this is frustrating that we don't get any pulse from the maintainers. But it's also frustrating to receive notifications on this issue and then discover that the comment does not add anything new.

maicol07 commented 2 years ago

@OneBlue can you look at this really long-awaited feature?

cleonard-godaddy commented 1 year ago

My problem is not related to development per se, as vscode and Pycharm work fine for me. However, I want to use robocopy to pull a mirror of my home directory to Windows, where I can use my preferred backup solution so that all of my backups are in the same (three) place(s). It's not very useful if I'm missing symlinked files. I'm going to upvote, but I wanted to take time to say that this seems like a fundamental piece of missing functionality.

elliason commented 1 year ago

I also struggle with this issue. My intellij IDE cant follow symlinks created via yarn workspaces (node.js monorepo tool). Very disappointing experience.

ymolchaniuk commented 1 year ago

How long we should wait for these feature?

ganey commented 1 year ago

How long we should wait for these feature?

If it's anything like Google products, at least 2 years from when the issue was started. So maybe soon?

tomty89 commented 1 year ago

I'd say it's more likely that MS drops WSL eventually than fixing this.

ltomov commented 1 year ago

How long we should wait for these feature?

If it's anything like Google products, at least 2 years from when the issue was started. So maybe soon?

Actually lack of symlinks support has been a well known windows problem for more than 20 years :)

There's probably a very good reason for that, so I wouldn't hold my breath to see it fixed anytime soon.

Best option I've found is to use vscode and open the project in WSL mode - symlinks work perfectly this way (wsl mode is different from opening a wsl folder - it makes vscode run some WSL server that takes care of symlinks and others).

culyun commented 1 year ago

@Itomov Windows NTFS supports symlinks and hardlinks, eg.

C:\work\cool_project\ mklink /D cool_lib C:\work\libs\cool_lib

culyun commented 1 year ago

I've given up waiting on this. The requirements are clear enough, which leads me to think that either MS doesn't care or there is some gnarly technical reason why supporting WSL symlinks is dead in the water.

tkelly-mgm commented 1 year ago

It looks like they're resolving as junctions with no target?

~/tmp$ ls -la
total 16
drwxr-xr-x  3 tkelly tkelly 4096 Sep 14 16:36 .
drwxr-xr-x 22 tkelly tkelly 4096 Sep 14 16:28 ..
drwxr-xr-x  2 tkelly tkelly 4096 Sep 14 16:29 a
lrwxrwxrwx  1 tkelly tkelly    1 Sep 14 16:25 b -> a
lrwxrwxrwx  1 tkelly tkelly   18 Sep 14 16:35 b_abs -> /home/tkelly/tmp/a
-rw-r--r--  1 tkelly tkelly    4 Sep 14 16:28 hi
lrwxrwxrwx  1 tkelly tkelly    2 Sep 14 16:34 hi_s -> hi
> cmd /c dir "\\wsl$\Ubuntu-20.04\home\tkelly\tmp"
 Volume in drive \\wsl$\Ubuntu-20.04 has no label.
 Volume Serial Number is 0000-0000

 Directory of \\wsl$\Ubuntu-20.04\home\tkelly\tmp

09/14/2022  04:29 PM    <DIR>          a
09/14/2022  04:28 PM                 4 hi
09/14/2022  04:25 PM    <JUNCTION>     b [...]
09/14/2022  04:35 PM    <JUNCTION>     b_abs [...]
09/14/2022  04:28 PM                 4 hi2
09/14/2022  04:35 PM    <DIR>          ..
09/14/2022  04:34 PM    <JUNCTION>     hi_s [...]
09/14/2022  04:35 PM    <DIR>          .
               5 File(s)             29 bytes
               3 Dir(s)  222,915,723,264 bytes free
ltomov commented 1 year ago

@itomov Windows NTFS supports symlinks and hardlinks, eg.

C:\work\cool_project\ mklink /D cool_lib C:\work\libs\cool_lib

yeah, but as far as I'm aware they are not 100% the same, and perhaps for that reason can't be used to emulate WSL symlinks on NTFS side. I tried that about a year ago and couldn't get it to work, but unfortunately don't remember the details.

florianajir commented 1 year ago

+1 this bug is really annoying when working with pnpm package manager

throwable-one commented 1 year ago

Since 20, Ubuntu uses symlinks bin -> usr/bin, so you can open anything in via \\wsl$\Ubuntu\bin\.. Those are reported as reparse point, at least in .NET API

foreach (var filePath in Directory.GetFileSystemEntries(@"\\wsl$\Ubuntu\"))
            {
                var attrs = File.GetAttributes(filePath);
                Console.Write("{0} is ", filePath);
                if ((attrs & FileAttributes.Directory) != 0)
                {
                    Console.Write("Directory ");
                }
                else
                {
                    Console.Write("File. Exists: {0}. Attrs: {1} ", File.Exists(filePath), attrs);

                    try
                    {
                        using (var file = File.Open(filePath, FileMode.Open))
                        {
                            file.ReadByte();
                            Console.Write("Read");
                        }
                    }
                    catch (Exception e)
                    {
                        Console.Write("Can't open {0}", filePath);
                    }
                }

                Console.WriteLine();
            }

And the result is:

\\wsl$\Ubuntu\sbin is File. Exists: True. Attrs: ReparsePoint Can't open \\wsl$\Ubuntu\sbin
\\wsl$\Ubuntu\boot is Directory
\\wsl$\Ubuntu\bin is File. Exists: True. Attrs: ReparsePoint Can't open \\wsl$\Ubuntu\bin
Anatolych97 commented 1 year ago

@craigloewen-msft @benhillis Any updates on this issue? If the company doesn't want to fix it, then just give us an official response and close this issue. We need answers! 😂

throwable-one commented 1 year ago

Everything is even worse: CreateFile works on Win10, but not on Win11 for file symbolic links.

~$ ls -lh /usr/bin/python3
lrwxrwxrwx 1 root root 9 Apr  5  2021 /usr/bin/python3 -> python3.9

Now, lets check this link from Windows:

#include <Windows.h>
#include <stdio.h>

int main()
{
    const char* path = "\\\\wsl$\\ubuntu\\usr\\bin\\python3";

    HANDLE h = CreateFileA(path,
                           FILE_READ_ATTRIBUTES,
                           FILE_SHARE_DELETE |
                           FILE_SHARE_READ | FILE_SHARE_WRITE,
                           NULL,
                           OPEN_EXISTING,
                           FILE_FLAG_BACKUP_SEMANTICS,
                           NULL);
    if (h == INVALID_HANDLE_VALUE)
    {
        const DWORD err = GetLastError();
        fprintf(stderr, "Can't open file: %ld", err);
        return -1;
    }
    BY_HANDLE_FILE_INFORMATION finfo;
    const BOOL result = GetFileInformationByHandle(h, &finfo);
    if (result == 0)
    {
        const DWORD err = GetLastError();
        fprintf(stderr, "Can't get file by handle: %ld", err);
        return -5;
    }
    printf("File attributes: 0x%lX", finfo.dwFileAttributes);

    return 0;
}

Win10: File attributes: 0x400. Ok. that means reparse point. Win11: "Can't open file: 3" which means ERROR_PATH_NOT_FOUND

elliason commented 1 year ago

For anyone struggling with this issue when using node.js monorepo tools like pnpm or yarn workspaces, the solution may be to use IDE via remote development (VSCode has https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack and now with new beta release PHPStorm can do the same https://blog.jetbrains.com/phpstorm/2022/11/phpstorm-2022-3-beta-2/ ) @florianajir

debel27 commented 1 year ago

... and now with new beta release PHPStorm can do the same https://blog.jetbrains.com/phpstorm/2022/11/phpstorm-2022-3-beta-2/ )

Great news! For Jetbrains users working on monorepos, this is finally a step in the right direction.

I still hope WSL2 will eventually support simlinks on \\wsl, but having the IDE back-end run on WSL is the way to go for the monorepo use case.

AndrewBogdanovTSS commented 1 year ago

If only it worked more stably. For me Remote Development launches in Webstorm 1 out of 10 times

Anatolych97 commented 1 year ago

For anyone struggling with this issue when using node.js monorepo tools like pnpm or yarn workspaces, the solution may be to use IDE via remote development (VSCode has https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack and now with new beta release PHPStorm can do the same https://blog.jetbrains.com/phpstorm/2022/11/phpstorm-2022-3-beta-2/ ) @florianajir

Remote development is an excellent first step, but at this moment, it works awfully. For me, it reloads the WE client on each launch, and in general, performance is not good.

debel27 commented 1 year ago

Remote development is an excellent first step, but at this moment, it works awfully. For me, it reloads the WE client on each launch [...]

I have the same problem when I launch remote development from Webstorm. If I go through Jetbrains Gateway, it doesn't download the client every time.

Regarding usability issues with remote development, I think it's important we report them while the topic is still hot.

tkelly-mgm commented 1 year ago

I could generally get remote development in Webstorm to work, but it had high latency and used considerably more RAM than otherwise. I eventually just started running VcXsrv and running my IDE entirely in the VM.

emeGuan commented 1 year ago

Hello. In case it is of interest to you. How to access the WSL2 file system from Windows with Samba. Samba transforms symlinks into Windows folders, so symlinks don't present any problem for filesystem use from Windows. The article is written in Spanish, it is very easy to follow using the translator integrated into linkedin. Best regards https://www.linkedin.com/pulse/acceso-wsl2-desde-windows-con-samba-manuel-nicol%C3%A1s-ortu%C3%B1o/

Anatolych97 commented 1 year ago

Hello. In case it is of interest to you. How to access the WSL2 file system from Windows with Samba. Samba transforms symlinks into Windows folders, so symlinks don't present any problem for filesystem use from Windows. The article is written in Spanish, it is very easy to follow using the translator integrated into linkedin. Best regards https://www.linkedin.com/pulse/acceso-wsl2-desde-windows-con-samba-manuel-nicol%C3%A1s-ortu%C3%B1o/

Wow, not bad. Looks much better than the current implementation of remote development. Thanks!

emeGuan commented 1 year ago

Hello. In case it is of interest to you. How to access the WSL2 file system from Windows with Samba. Samba transforms symlinks into Windows folders, so symlinks don't present any problem for filesystem use from Windows. The article is written in Spanish, it is very easy to follow using the translator integrated into linkedin. Best regards https://www.linkedin.com/pulse/acceso-wsl2-desde-windows-con-samba-manuel-nicol%C3%A1s-ortu%C3%B1o/

Wow, not bad. Looks much better than the current implementation of remote development. Thanks!

The performance is good. I don't see any difference between working with WSL+SAMBA and working with the local file system. As I comment at the end of the article, WSL does not allow setting a static IP. If you use WSL for javascript development (for example vuejs), there is no problem starting the development server, you can access via localhost, but if you need to have multiple virtualservers (for example Apache with multiple virtualservers) you need to know the WSL IP. Soon I'm going to write another article describing the problem and the workaround I use

emeGuan commented 1 year ago

What I was saying about the workaround to solve that WSL2 not allow static IP https://www.linkedin.com/pulse/wsl2-doesnt-support-static-ip-manuel-nicol%25C3%25A1s-ortu%25C3%25B1o/

cryptiklemur commented 1 year ago

For some reason, i can't connect to the samba share, it just hangs.

toyi commented 1 year ago

This issue has been here for a while and as a Jetbrains user working on WSL, I feel like I'm stuck between the silence of Microsoft here and the "we can't do anything unless this is solved in WSL" of Jetbrains.

Their remote IDEs (Webstorm / PHPStorm) are still in their early stages, they lack a lot of features and eat an absurde amount of resources.

I'm telling myself... "what now?". I don't feel confortable with any of the choices I have.

  1. Developing on Windows (not WSL)
  2. Developing on Linux (seriously, switching the OS? I seriously consider this option)
  3. Developing with VSCode (I love VSCode, but I love the DX of Jetbrains way more and the database tools are amazing, miles ahead of the SQLTools experience)
  4. Continue to work without autocompletion. You guessed it, this is not a good DX.

If it's not going to be fixed, a simple "this will not be fixed" would be appreciated.

tkelly-mgm commented 1 year ago

To run WebStorm with VcXSrv, I did the following:

  1. Install VcXSrv, which is an X Server that runs in Windows.
  2. Start VcXSrv. While starting it, I need to check "Disable access control".
  3. Set my environment variables in WSL to connect to VcXSrv:
    export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2; exit;}'):0.0
    export LIBGL_ALWAYS_INDIRECT=1
  4. Install the Linux version of WebStorm inside of WSL.
  5. Start WebStorm using the WSL terminal.

And it mostly just works. Sometimes subwindows and menus end up on the wrong monitor, but it generally acts pretty natively.

GimpMaster commented 1 year ago

This issue has been here for a while and as a Jetbrains user working on WSL, I feel like I'm stuck between the silence of Microsoft here and the "we can't do anything unless this is solved in WSL" of Jetbrains.

Their remote IDEs (Webstorm / PHPStorm) are still in their early stages, they lack a lot of features and eat an absurde amount of resources.

I'm telling myself... "what now?". I don't feel confortable with any of the choices I have.

  1. Developing on Windows (not WSL)
  2. Developing on Linux (seriously, switching the OS? I seriously consider this option)
  3. Developing with VSCode (I love VSCode, but I love the DX of Jetbrains way more and the database tools are amazing, miles ahead of the SQLTools experience)
  4. Continue to work without autocompletion. You guessed it, this is not a good DX.

If it's not going to be fixed, a simple "this will not be fixed" would be appreciated.

We basically decided to stick with Yarn 1.X for these reasons. I'm just hoping that Jetbrains will keep improving the remote IDE so then it won't matter anymore and we can finally switch to PNPM.

oldium commented 1 year ago

To run WebStorm with VcXSrv, I did the following:

  1. Install VcXSrv, which is an X Server that runs in Windows.
  2. Start VcXSrv. While starting it, I need to check "Disable access control".
  3. Set my environment variables in WSL to connect to VcXSrv:
    export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2; exit;}'):0.0
    export LIBGL_ALWAYS_INDIRECT=1
  4. Install the Linux version of WebStorm inside of WSL.
  5. Start WebStorm using the WSL terminal.

And it mostly just works. Sometimes subwindows and menus end up on the wrong monitor, but it generally acts pretty natively.

Use WSLg in that case. This should work. I even was able to run GUI application from SSH connection started from WSL with X11 forwarding enabled.

debel27 commented 1 year ago

I'm telling myself... "what now?". I don't feel confortable with any of the choices I have.

  1. Developing on Windows (not WSL)
  2. Developing on Linux (seriously, switching the OS? I seriously consider this option)
  3. Developing with VSCode (I love VSCode, but I love the DX of Jetbrains way more and the database tools are amazing, miles ahead of the SQLTools experience)
  4. Continue to work without autocompletion. You guessed it, this is not a good DX.

I ended up going for 2. It's the better compromise at the moment

ganey commented 1 year ago

@toyi i did the same as @debel27 and changed OS, got a Macbook and run docker or just install the packages. Tried a more powerful Windows laptop but this was a real blocker for continuing with Windows as part of a team.

Made this decision last year and as this issue is still ongoing, really glad to have not stuck with Windows.

toyi commented 1 year ago

@tkelly-mgm @GimpMaster @oldium @debel27 Thank you for your feedbacks. Sorry, I was a little bit upset when I wrote my comment! I just find the situation to be very frustrating.

Using VcXSrv / WSLg is indeed an alternative I didn't mentionned, it might be ok for some people but to me, it seems too hacky / not optimized for my daily workflow. I'll switch to Linux and I'll wait until a propre solution is found.

Again, if someone in charge of WSL read this, any feedbacks on this issue would be greatly appreciated.

A lot of people use WSL for development, a lot of people use Jetbrains products and a lot of people works with the JS ecosystem. Therefore, I guess a lot of people is impacted by this issue.

emeGuan commented 1 year ago

I use WSL2+Samba by the problem with symlinks. Developer in vuejs, PHP and Java. All the code in WSL and access to the code (with samba) from Windows with PhpStorm(vue and php) and Eclipse (java). The performance is good and for me all ok.