package-build--create-tar: don't fiddle with windows filenames

[juergenhoetzel]

Windows 10 ships a native version of tar: https://techcommunity.microsoft.com/t5/Containers/Tar-and-Curl-Come-to-Windows/ba-p/382409

Also GNU tar on Windows uses native filenames.

I'm not able to build packages on Windows without this commit:

a fsharp-mode-20191130.1857
a fsharp-mode-20191130.1857/eglot-fsharp.el
a fsharp-mode-20191130.1857/fsharp-mode-font.el
a fsharp-mode-20191130.1857/fsharp-mode-pkg.el
a fsharp-mode-20191130.1857/fsharp-mode-structure.el
a fsharp-mode-20191130.1857/fsharp-mode-util.el
a fsharp-mode-20191130.1857/fsharp-mode.el
a fsharp-mode-20191130.1857/inf-fsharp-mode.el
Fetching origin
HEAD is now at 4a1df33 Merge pull request #230 from juergenhoetzel/changelog-update
tar.exe: Failed to open '/c/Users/pidsleywin/.emacs.d/.cask/26.3/bootstrap/packages/fsharp-mode-20191130.1857.tar'

[tarsius]

We don't fiddle with filenames because doing so is fun but because it appeared that doing so was necessary. If we applied this, then that would most likely break things for other users. If you can figure out how to determine whether we have to fiddle with the path or not, then that would be best. Failing that, an option could be added.

The fiddle was added in 38e4eca50086a783761b3eaa91b62950cd76c357. But no detailed explanation was added in the commit message. @steckerhalter do you have any recollections?

The difference has to do with what kind of Emacs (cygwin, windows-nt, ???) is being combined with what kind of tar (cygwin, window-nt, ???). And then cygwin can be configured to deal with paths one way or another. And then applications may fail to respect those settings...

[juergenhoetzel]

We don't fiddle with filenames because doing so is fun but because it appeared that doing so was necessary. If we applied this, then that would most likely break things for other users. If you can figure

that's what I also thought, but i found no tar implementation to point out these peculiarities.

And then cygwin can be configured to deal with paths one way or another. And then applications may fail to respect those settings...

The removed conditional only checked for (eq system-type 'windows-nt) , not cygwin.

[steckerhalter]

I think tar had to be GNU for some reasons. But it was necessary to fiddle with the path to make it work even there. I don't know know now though :)

[juergenhoetzel]

I think tar had to be GNU for some reasons. But it was necessary to fiddle with the path to make it work even there. I don't know know now though :)

You have a good memory. I tried it:

PS C:\> C:\Users\pidsleywin\scoop\shims\tar.exe cvf c:/foo.tar foo.dat
"C:/Users/pidsleywin/scoop/apps/tar/current/bin/tar.exe": Cannot connect to c: resolve failed

But using the /c/foo.tar doesn't create a archive (silently fails)

[steckerhalter]

I found this here: https://www.drupal.org/project/drush/issues/483110#comment-1994938

The errors you see with GNU tar are a result of passing it a path with the drive label included (e.g. c:\foo\bar). I suspect it's calling a Windows change-directory function to cd to the directory and failing on the "c:". It appears that the alternate tar utility knows how to handle this. I played around with removing the drive label from the file path in Windows and it works in most situations. It has the side effect of limiting the utility to working on the currently active drive.

the fiddling seems to have been exactly this: removing the drive e.g. c: on windows and replacing it with / so you would have to try with /foo.tar

[steckerhalter]

there's a PR from @kiennq for quelpa which should add windows bsdtar support. maybe you can use some of the code from there: https://github.com/quelpa/quelpa/pull/151/files

[juergenhoetzel]

there's a PR from @kiennq for quelpa which should add windows bsdtar support. maybe you can

I'm sure you meant the GNU variant. bsdtar doesn't have the GNU extension/bloat :laughing:

use some of the code from there: https://github.com/quelpa/quelpa/pull/151/files

BTW, there is a command line option to disable the GNU extension:

       --force-local
              Archive file is local even if it has a colon.

so you don't have to use the regex-replace.

To be honest, I don't think this additional code is worth the trouble, since Windows 10 now comes with BSD tar (as mentioned above) and Windows 7 will reach End Of Life (EOL) on 14 January 2020 :thinking:

[steckerhalter]

I'm sure you meant the GNU variant. bsdtar doesn't have the GNU extension/bloat

No, I did mean bsdtar. I'm not sure that disabling the GNU extensions solves the path issue. It could be a general GNU tar issue on Windows (not specific to the format). Anyway, what happens here in package-build is not for me to be judged. I guess @tarsius is in charge for that :smile:

[juergenhoetzel]

I'm sure you meant the GNU variant. bsdtar doesn't have the GNU extension/bloat

No, I did mean bsdtar. I'm not sure that disabling the GNU extensions solves the path issue. It could be a general GNU tar issue on Windows (not specific to the format). Anyway, what happens here in

BSD tar (windows 10 tar):

PS C:\Users\pidsleywin> C:\Windows\System32\tar.exe --version
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.5.f-ipp

works fine:

PS C:\Users\pidsleywin> C:\Windows\System32\tar.exe cf c:\tmp\test.tar -- test.py
PS C:\Users\pidsleywin>

VS GNU tar:

PS C:\Users\pidsleywin> C:\Users\pidsleywin\scoop\shims\tar.exe --version
tar (GNU tar) 1.23
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

tries to network connect:

PS C:\Users\pidsleywin> C:\Users\pidsleywin\scoop\shims\tar.exe cf c:\tmp\test.tar -- test.py
"C:/Users/pidsleywin/scoop/apps/tar/current/bin/tar.exe": Cannot connect to c: resolve failed

--force-local works:

PS C:\Users\pidsleywin> C:\Users\pidsleywin\scoop\shims\tar.exe cf c:\tmp\test.tar --force-local -- test.py
PS C:\Users\pidsleywin>

[steckerhalter]

It works :smiley: So is this "feature" a GNU extension? I can't find anything being mentioned about it. Can't bsdtar handle things like sftp:// etc. in this case? Anyway, it's one way to do it on Windows without the path regex.

[juergenhoetzel]

So is this "feature" a GNU extension? I can't find anything being mentioned about it.

It's documented in the man-page (man tar(1)):

       --force-local
              Archive file is local even if it has a colon.

Can't bsdtar handle things like sftp:// etc. in this case?

I have never understood the benefits of this extension. When you can just pipe (in the sense of the Unix philosophy):

tar cf - -- test.py|ssh pidsley "cat >/tmp/test.tar"

instead of using this non-portable extension:

tar cf pidsley:/tmp/test.tar --rsh-command=/usr/bin/ssh -- test.py

[steckerhalter]

It's documented in the man-page (man tar(1))

yeah I saw this but it doesn't mention it's a specific extension/feature of GNU tar

I have never understood the benefits of this extension.

it is questionable why they did that, right...

[tarsius]

To be honest, I don't think this additional code is worth the trouble, since Windows 10 now comes with BSD tar (as mentioned above) and Windows 7 will reach End Of Life (EOL) on 14 January 2020

So... I am closing this.

[jcs090218]

Hmmm... Can't we simply check things like?

     (when (and (eq system-type 'windows-nt)
+               (not (string-prefix-p "bsdtar" (shell-command-to-string "tar --version"))))
       (setq tar (replace-regexp-in-string "^\\([a-z]\\):" "/\\1" tar)))

so the package can still works in the native Windows system? 😕

[tarsius]

Doing it like this would mean starting a second subprocess every time we need to start one subprocess, and since Windows is slow at starting subprocesses that doesn't seem like such a good idea.

[tarsius]

Also,

To be honest, I don't think this additional code is worth the trouble, since Windows 10 now comes with BSD tar (as mentioned above) and Windows 7 will reach End Of Life (EOL) on 14 January 2020

So... I am closing this.

should we really support legacy versions of Windows longer than Microsoft?

[jcs090218]

Doing it like this would mean starting a second subprocess every time we need to start one subprocess, and since Windows is slow at starting subprocesses that doesn't seem like such a good idea.

Oh, yes. It was just an example. Probably better to do it this way.

+ (defconst package-build-tar-string (shell-command-to-string "tar --version")
+   "...")

      (when (and (eq system-type 'windows-nt)
+                (not (string-prefix-p "bsdtar" package-build-tar-string)))
        (setq tar (replace-regexp-in-string "^\\([a-z]\\):" "/\\1" tar)))

Or similar to quelpa's solution, see https://github.com/quelpa/quelpa/pull/151/files (much more complete).

should we really support legacy versions of Windows longer than Microsoft?

I am using Windows and probably won't be able to move out completely in the next decades. 😭 It would be nice if this package package-build does support it, so I can continue to build Emacs packages on Windows without needing to switch over to Linux or macOS.

Would it hurt if it did support legacy versions of Windows? Just curious. 😅

[jcs090218]

@tarsius Friendly ping. :)

[akirak]

@jcs090218 Why don't you switch to Windows 10 or later? Are you going to stick with an old version of Windows for more than a decade?

Windows 10 now comes with BSD tar (as mentioned above) and Windows 7 will reach End Of Life (EOL) on 14 January 2020

[jcs090218]

Why don't you switch to Windows 10 or later?

I'm using windows 10, and the issue is from the bsd tar.

Are you going to stick with an old version of Windows for more than a decade?

Many still use the old system, Windows XP, 7, Vista, etc. But I'm not answering that since I think that is people's preferences and irrelevant.

[tarsius]

@tarsius Friendly ping. :)

I was busy with #67. It's a big change and there is a risk of breakage. I don't want to complicate that by merging unrelated changes right now. I am also super busy until the end of the year at least. If I don't get back to this on my own, then please ping me again sometime next year.

:fireworks: Happy new year!

[jcs090218]

@tarsius Friendly ping. ❤️

If you have time to take a look at this, will be greatly appreciate it!