Create Bash For Windows based on Git 4 Windows SDK

[SteveALee]

Idea

Provided a modified version of the Git for WIndows SDK that gives a similar environment to Git for windows bash but allows additional packages lik jq to be installed by the user using the pacman package manger..

Background

Git for WIndows provides a mature and solid Win32 native bash shell and enviironrment that is perfect for many dev tasks using git and the other bundled *nix syle tools. If npm would only run bash rather than only cmd it would be perfect for npm script porability too. It's easy to install and is based on MinGW native ports and MSYS2 POSIX emulator dll supported exes with a preference for the Ming ports where possible.

This compares well with WSL - Microsoft 3rd and most successful attempt to provide a full POSIX based linux environment on Windows. WSL runs real userland linux tools on a kernel than maps to WIN32 system calls. While WSL is the real deal thing, are speed isssues and often Git 4 Windows is a light weight 'leand and mean' alternative.

Git for WIndows SDK is prmarilary designed to allow work on the WIndws git package so it includes various dev tools like the C tool chain which are not normally wanted. Thus, the idea is to modify the bundled packages to provide a version of the sdk suitable to more general developers .

Thanks to @dscho for suggesting this.

[SteveALee]

@dscho There are quite a few moving parts here and I'm still getting to grips. :)

But here is an idea: why don't you work on a minimal Git for Windows-flavored MSYS2? I.e. a subset of Git for Windows' SDK that comes with Git preinstalled, and with pacman preinstalled, but not with gcc and friends?

Are we creating a new version of the SDK or using the SDK to build a new deployment of G4W?

thanks

[dscho]

There are quite a few moving parts here and I'm still getting to grips. :)

No worries ;-)

But here is an idea: why don't you work on a minimal Git for Windows-flavored MSYS2? I.e. a subset of Git for Windows' SDK that comes with Git preinstalled, and with pacman preinstalled, but not with gcc and friends?

Are we creating a new version of the SDK or using the SDK to build a new deployment of G4W?

What I had in mind was to have a new "flavor" of the SDK, much like the portable Git is a flavor of Git for Windows: built from the same environment, but it looks different, and it caters to a different (although overlapping) audience.

Are you stuck somewhere?

[SteveALee]

Are you stuck somewhere?

Thanks for asking. I actually have had less time than I expected.

I was exploring adding pacman to the portable build of G4W as that seemed relatively easy to do and test. That seemed a quicker route to what I Imagined (eg is already missing C toolchain).

Out of interest why did you think the SDK is a better starting point? Does it have other useful packages not in the user installer? I can certainly explore that approach. Especially now I understand more about how this all works.

It's trivial to add a package to the main build script (either as a script parameter or by editing) but pacman has quite a few other parts to add, like the packages and the daemon. Was also thinking could posssbly add an option to the build script rather than a separate file. That would add pacman and change target zip name.

For the packages I wonder if the user could fetch those when 1st using pacman rather than we add them to the zip)

[dscho]

I was exploring adding pacman to the portable build of G4W as that seemed relatively easy to do and test. That seemed a quicker route to what I Imagined (eg is already missing C toolchain).

Careful: it is not enough to just include the files of the pacman package; you also have to skip the excludes like var/lib/pacman/local/.

Out of interest why did you think the SDK is a better starting point?

Because it already has all the files you need. And essentially you are doing that: by extending the portable Git (that is generated in an SDK). And that is really similar to what my idea was: to add another release.sh file (just like the one generating the portable Gits) that also uses make-file-list.sh to generate the list of files, and will need to be extended to allow for starting from a fresh list of packages to include, and also a flag to add Pacman's package information for those packages that lives in subdirectories of var/lib/pacman/local/).

pacman has quite a few other parts to add, like the packages and the daemon

We don't need no daemon, do we?

Was also thinking could posssbly add an option to the build script rather than a separate file

I'd rather have a separate release.sh script in a separate subdirectory of /usr/src/build-extra/. After all, what you have in mind does not even need to include Git. So it has pretty much nothing to do with Git for Windows, apart from the fact that it can install Git for Windows' Pacman packages.

For the packages I wonder if the user could fetch those when 1st using pacman rather than we add them to the zip)

Actually, what I had in mind would start from a really minimal set of included files/packages. Like, a MinTTY, Bash, filesystem, the base system, like. Wasn't there a meta package for that? I thought it was something like msys2-base or something.

clicketyclick

Ah, it was base:

$ pacman -Qi msys2-runtime | grep Groups
Groups          : base

[dscho]

[...] it was base:

$ pacman -Qi msys2-runtime | grep Groups
Groups          : base

And here is another very interesting command for your purposes:

$ pacman -Qg | grep '^base '
base bash
base bash-completion
base bsdcpio
base bsdtar
base bzip2
base coreutils
base crypt
base curl
base dash
base file
base filesystem
base findutils
base flex
base gawk
base gcc-libs
base git-for-windows-keyring
base grep
base gzip
base inetutils
base info
base less
base lndir
base mintty
base msys2-keyring
base msys2-runtime
base ncurses
base pacman
base pacman-mirrors
base pactoys-git
base pax-git
base pkgfile
base rebase
base sed
base tftp-hpa
base time
base ttyrec
base tzcode
base util-linux
base which

[SteveALee]

Thanks for those tips. I make the PR and then we'll have somehting concrete to discuss :)

Careful: it is not enough to just include the files of the pacman package; you also have to skip the excludes like var/lib/pacman/local/.

Yeah, I discovered that and also mentioned it below (daemon)

Because it already has all the files you need. And essentially you are doing that: by extending the portable Git (that is generated in an SDK). And that is really similar to what my idea was:

I was obviously overthinking what you were saying. And I see the release distributions are really just subsets of the SDK, plus a bit of install stuff. We appear to be on the same page after all. :)

We don't need no daemon, do we?

Oops no, I meant /etc/pacman.d

I'd rather have a separate release.sh script in a separate subdirectory of /usr/src/build-extra/. After all, what you have in mind does not even need to include Git. So it has pretty much nothing to do with Git for Windows, apart from the fact that it can install Git for Windows' Pacman packages.

And that is exactly what I've done so far, partially as not sure if the other files like those in ...root/ might need tweaking. :)

Actually, what I had in mind would start from a really minimal set of included files/packages. Like, a MinTTY, Bash, filesystem, the base system, like.

Sounds good. I was thinking of git but then saw would need perl etc etc which It would be good to avoid in the distro. Thought I'd start with g4w and prune from there. Unless we can get a nice list together - as I think your other comment does.

clicketyclick

heh

[SteveALee]

I feel we really do need a basic git. Is the minimal one good enough for most users?

[dscho]

I feel we really do need a basic git. Is the minimal one good enough for most users?

What do you mean by "basic git"?

The part of Git that does not require Perl? Or shell? If so, this is not (yet) split off into its own package, so that would have to be done first (otherwise we would have to ship only partial contents of the mingw-w64-git package).

[SteveALee]

Yes I was Wondering if that might be possible but I do not know which parts require Perl. I also see the MINIMAL__Git in the release scripts, but that seems to mean got and not much else,

The bottom line for my basic msys2 I'd be happy with the current setup plus Pacman. Making it more minimal as you suggest is fine, as long as common shell scripting and git are there. Other packages can then be easily added on demand via pacman.

Currently I have more files than portable but am not sure what I could sensibly prune from the base group of packages.

[dscho]

I do not know which parts require Perl.

That one's easy: git ls-files git-*.perl | sed 's/\.perl$//' in the git.git checkout (which mingw-w64-git/PKGBUILD has access to, when packaging files).

The not so easy part is to determine what else is required. For example, I think that git send-email requires a Perl module to be able to talk to IMAP.

clicketyclick

So it does use Net::SMTP, and that package is provided by /usr/share/perl5/core_perl/Net/SMTP.pm, which is owned by the perl package itself:

$ pacman -Qo /usr/share/perl5/core_perl/Net/SMTP.pm
/usr/share/perl5/core_perl/Net/SMTP.pm is owned by perl 5.26.2-1

Let's see, what else is required by that script...

clicketyclick

Hmm. All of those use lines that I looked at seem to correspond to Perl packages that are included in perl. Let's look at another of my favorite Perl scripts: git-add--interactive.perl.

clicketyclick

Ahhh! Term::ReadKey is provided by the perl-TermReadKey package. So that would have to be hard-coded, I guess.

And of course, we would have to remove that hard-coded dependency again once I finish my work to convert that Perl script to C...

I also see the MINIMAL__Git in the release scripts, but that seems to mean got and not much else

Indeed. That refers to the MinGit editions of Git for Windows, i.e. Git for Windows intended for 3rd-party applications (where we explicitly do not include Perl, or any interactive functionality, in order to save on space).

The bottom line for my basic msys2 I'd be happy with the current setup plus Pacman. Making it more minimal as you suggest is fine, as long as common shell scripting and git are there. Other packages can then be easily added on demand via pacman.

Sure.

What I would suggest, then, is that you start out with a really minimal one that really only provides pacman itself, and installs mingw-w64-git as well as other dependencies listed in make-file-list.sh via pacman.exe first thing after unpacking. It can use the update-via-pacman.bat script, I guess, but it would need to use another name because that script can be overwritten by git-extra while the script is running, and that would cause the script to exit with an exit code indicating failure, I assume.

Currently I have more files than portable but am not sure what I could sensibly prune from the base group of packages.

The thing is: if you ship pacman's files, and if your intention is to allow the user to use pacman.exe to install packages, there is no reason why the initial script itself should not use pacman.exe to download the current, say, mingw-w64-git as well. Those files do not need to come pre-bundled.

[SteveALee]

Thanks for the ideas. A very good point about post unbundle install of packages!

Time to rethink....

I haven;t quite got pacman working after unbundle -

warning: Public keyring not found; have you run 'pacman-key --init'?
error: git-for-windows: key "57759F748D223F034D8BE870BB3AA74136C569BB" is unknown
...

I guess I need to bundle more files yet.

This is kinda fun :)

[dscho]

warning: Public keyring not found; have you run 'pacman-key --init'? error: git-for-windows: key "57759F748D223F034D8BE870BB3AA74136C569BB" is unknown

Right, this is what those lines in git-extra's post-install script are all about:

    # Ensure that Git for Windows' GPG keyring is still imported
    pacman-key --list-keys BB3AA74136C569BB >/dev/null ||
    pacman-key --populate git-for-windows

The actual files of that keyring are installed via git-for-windows-keyring: https://github.com/git-for-windows/build-extra/tree/master/git-for-windows-keyring