Building from source

[deevus]

Proposal

We add the ability to compile programs from source.

scoop install hello --source

The app manifest has a formula key which points to a ps1 file containing instructions on how to build from source. It is expected to be alongside the json manifest.

Why?

• Nothing like this exists for Windows
• If you don't trust the binary distribution you can compile from source instead
• It's cool!
• Other reasons?

  Why not?

• Complexity (Scoop has a simple well-defined role)
• Windows users don't want to compile from source (?)
• No functional benefit over using binaries (?)

[lukesampson]

What sort of build tool chain would you use, or would it just be "anything goes" with the PS script doing whatever it wants to?

And do you have a specific program in mind?

I agree this would be cool. Personally, I would probably always prefer a pre-built binary over building from source, and I tend to agree with the other why-nots too.

[deevus]

I like the idea of having a set tool chain, however I don't have a particular one in mind. I just really like how Homebrew builds everything from source. I understand that Windows is pretty horrible for that sort of thing... but there's no shame in working to change that.

[lukesampson]

Yeah it could be that one of the reasons no-one builds from source on Windows is because it's really hard. I've tried to build a few MinGW things and failed miserably. I got rust to build but only because I used MSYS which is kind of unsatisfying.

I'm not against the idea though. Do you think it's worth trying to put together a standalone script that can build something, as a proof of concept? Or do you already have one?

[deevus]

Yeah I was going to look into getting GNU Hello building. That project is built to test your build environment. I think if I can get something that can build that... most of the hard work will be done.

[scottwillmoore]

I would be interested! I have always considered building a system similar to homebrew for windows, however the amount of effort required as well as decided what tools to develop it with has always put me off. I have always been jealous of homebrew and other OS X tools which Windows never seem to have; as well as bad C support and even bad C++ support for crossplatform development which Microsoft appears not to really care about. :cry: Anyway, I have used and studied homebrew quite a bit and I believe an effective system could also be built for Windows.

First of all I believe it may be a better idea to make this project separate from scoop, because I like the simplicity of how scoop works at the moment and I don't want to pollute it with an entire building from source system.

Also while I enjoy PowerShell for some reason I find it incredibly slow! I don't know whether this is my machine or whether PowerShell is just extremely unresponsive as a command line tool. I often find that I am waiting for commands to execute. While saying this I don't see many great alternatives to PowerShell that will run natively on windows.

My only other popular option is using Go possibly with a lightweight JavaScript engine (otto built in Go). Another idea could be possibly with lua, which can actually be made into an effective task runner as seen with the project premake. Just look at some of the lua syntax examples, it hardly looks like lua!

I believe a pretty awesome PowerShell script wrapper could be created similar to something that psake achieves. It hardly looks like PowerShell and is simple for people who are not familiar with PowerShell (which I believe will be many, considering my familiarity with it when I first came across scoop).

Homebrew also has an extremely interesting system which they call turning brews into bottles which are effectively prebuilt binarys which are built by homebrew. These binarys are stored in a bintray repository here. This process is automatically executed by their automatic bots when a pull request is accepted, however I believe when they first started this process was manually completed.

Anyway first of all I believe figuring whether it is a viable solution after building some projects manually and assessing whether it would be possible to automate the process of building these projects on a Windows machine. Sorry for making this so long, just wanted to write my thoughts down.

[deevus]

No apologies needed! I'm happy to see some discussion about it. I feel exactly the same way as you regarding Homebrew/Linuxbrew.

I don't really find Powershell too slow once it's running but the start-up time is really bad. That being said I have no problem with choosing another language. I do like that you don't need to install anything to get Scoop running, which is a major plus since Windows people generally cry about more than 1 step. However, Powershell could just be used as a bootstrap to get the system installed.

Theres https://github.com/mfichman/winbrew written in Python but you need to have VS 2013 installed, and it's quite new. It could be a workable project to fork if we could factor out the VS requirement. As far as bottles go, Scoop is kind of a bottle repository already. We could use that to Scoop's advantage if this theoretical system was built.

[deevus]

Perhaps something to consider is that Windows apps being built from source would need to support a range of different build systems. That would probably be a major requirement to get this working for a great catalogue of apps.

[scottwillmoore]

Interesting. Yeah I was going to mention PowerShell bootstrap could be used to get the system installed if we required a language such as ruby or python. The winbrew project is pretty awesome, while the Visual Studio requirement could be annoying for some (especially with the size of the current community edition which is in the gigabytes) I believe that would be neccessary dependency for a windows build system, as XCode is a neccessary dependency for building on a Mac.

The project only has a few formula, but enough to show promise to creating a windows source building system. I do not particularly enjoy python (although I can't say I have worked with it much), but the syntax is nice and simple. It also has a bit of the infrastructure and code in place as an example to a windows source building system.

I also belive the system should not be a direct clone of hombrew as the windows buildsystem may need to be different in several areas such as being able to target multiple build systems, and stealing the homebrew name and naming conventions doesn't seem unique.

Also the system will probably need a better dependency management system than scoop as it will require applications such as cmake to be installed in order to build programs on top of that. Almost creating a tree of applications which depend on others to be installed. Homebrew has a clever dependecy system, but I guess other package managers such as nodeJS may also have some interested things we can learn from, provided it is simple to implement and maintain.

Homebrew also has a nice optional step way of building scripts which you can also see here.

[deevus]

@scottwillmoore I wonder if using docker would be viable. Compiling on Windows can be a bitch, so why not set up a docker image and using Docker for Windows, cross compile in the docker container, returning the installation.

[deevus]

Here's a start: https://github.com/deevus/docker-mxe

[deevus]

I got GNU Hello building using MXE and Docker: https://gist.github.com/deevus/40e223b1794017316900

You will need boot2docker and tar installed

[scottwillmoore]

Awesome, I'm not an expert on docker, would you mind explaining how this works? Isn't this creating a Linux container with an cross platform environment/application (not really sure what this is) called MXE which allows you to build software on a linux system for Windows?

I still don't fully understand what MXE does, but do you think there will be complications when building larger applications using it. It also feels counterintuitive to build a Windows application on Linux in a virtualised environment (could be slow, take more resources, requires extra dependencies), but it could also have benefits such as making the system independent from the users configurations and installed programs (probably more benefits).

I remember trying to build Lua on windows a while ago to see how difficult it would be and discovered they use this tool for building Lua on Windows. It's called TDM-GCC and is basically a prebundled MinGW, GCC, Windows APIs that can create 32bit or 64bit binaries. This may be another idea to consider, although I have never really used it so I do not know how effective it is.

Also, is there any benefits of using Microsoft Visual Studios compiler, because that is about as native Windows as you can get, but I understand many UNIX applications will not build correctly with MSVS. Is it also true that many Windows only applications will not build with MinGW?

Many users will probably want C and C++ building tools as well, because this is usually the first applications that users will want to install, so that may be another advantage to having the system setup a building environment on Windows.

[deevus]

You're pretty much spot on there about docker. MXE is just a bundled mingw tool chain that builds a lot of stuff itself. The container just gets it to build gcc and that's it.

I went for docker because it's relatively quick, easy to setup and it's not cygwin. Docker will probably have native windows support (or close to it) in future since they've already announced support in the new Windows Server. So despite it being Linux it should get close to native speeds in future. Plus VMs are much faster these days.

In the case of GNU Hello (and many other projects) it uses GNU Configure which needs a supported shell (bash or cygwin etc). Docker gives a pretty sure fire way of building them without much hassle. The container is essentially a Windows build machine. Pass it the code and it returns the binaries.

Perhaps you can look at TDM-GCC to see how easy it is to build GNU Hello? Maybe we look at msys and msvc as well? Are we avoiding cygwin? On 21/04/2015 6:17 PM, "Scott Will Moore" notifications@github.com wrote:

Awesome, I'm not an expert on docker, would you mind explaining how this works? Isn't this creating a Linux container with an cross platform environment/application (not really sure what this is) called MXE which allows you to build software on a linux system for Windows?

I still don't fully understand what MXE does, but do you think there will be complications when building larger applications using it. It also feels counterintuitive to build a Windows application on Linux in a virtualised environment (could be slow, take more resources, requires extra dependencies), but it could also have benefits such as making the system independent from the users configurations and installed programs (probably more benefits).

I remember trying to build Lua on windows a while ago to see how difficult it would be and discovered they use this tool for building Lua on Windows. It's called TDM-GCC http://tdm-gcc.tdragon.net/ and is basically a prebundled MinGW, GCC, Windows APIs that can create 32bit or 64bit binaries. This may be another idea to consider, although I have never really used it so I do not know how effective it is.

Also, is there any benefits of using Microsoft Visual Studios compiler, because that is about as native Windows as you can get, but I understand many UNIX applications will not build correctly with MSVS. Is it also true that many Windows only applications will not build with MinGW?

Many users will probably want C and C++ building tools as well, because this is usually the first applications that users will want to install, so that may be another advantage to having the system setup a building environment on Windows.

— Reply to this email directly or view it on GitHub https://github.com/lukesampson/scoop/issues/340#issuecomment-94699413.

[lukesampson]

This isn't a bug report, but I couldn't get boot2docker to work, sorry. I've never been able to get VirtualBox running inside the Parallels VM I use for Windows.

[deevus]

You can install boot2docker in OSX then configure the docker client in Windows to talk to it via ip though I'm not sure how the shared folders will work in that case. On 22/04/2015 6:17 AM, "Luke Sampson" notifications@github.com wrote:

This isn't a bug report, but I couldn't get boot2docker to work, sorry. I've never been able to get VirtualBox running inside the Parallels VM I use for Windows.

— Reply to this email directly or view it on GitHub https://github.com/lukesampson/scoop/issues/340#issuecomment-94925171.

[scottwillmoore]

@lukesampson develops from a Mac through a VM! Thats perfectly fine don't worry :stuck_out_tongue:, for some reason it just made me laugh.

I'll see if I can check out TDM-GCC, MSYS and MSVC this weekend, but I assume MSVC will be awesome if we get it working, but we will not get it working due to lack of support by developers. Yeah for now I don't think Cygwin is a good idea, it just sounds like your docker setup (building on Linux), but instead of building on Linux in Linux, you are building on Linux in Windows through a weird emulation layer.

Also been thinking about the docker idea, and the term, "it's basically a Windows build machine" sounds pretty cool. It may be a pretty good solution after all, especially if its performance isn't too bad.

[vanblaze]

Is this still something being thought about?

[iipamonkey]

This, please! Another reason "why": The default options for some pre-built apps do not meet a user's needs. Example: The pre-built 'ffmpeg' is currently distributed without the most performant (and non-free) AAC codec library, 'libfdk_aac'. All it would take to resolve that is appending [--enable-gpl --enable-nonfree --enable-libfdk-aac] to the options used for pre-built binary and we'd be in business.

On another note - scoop might very well be the best bit of software that runs on Windows (including Windows itself). Well done, Mr. Sampson.

[deevus]

Revisiting this... I found https://github.com/dockcross/dockcross

And now that Docker on Windows is better supported, this could be a possibility

[masaeedu]

@deevus Cross compiling is overall slightly trickier than just using a local toolchain I think. Maybe a source package for scoop could have some notion of abstracting over a "toolchain" that needs to be present on the system (maybe installed via another scoop package?), e.g. env vars that are passed to the build script telling it where everything is. This is similar to how nixpkg works.

Anyway I want this feature badly enough to set aside some time to work on it; if you have some general ideas on how you want this structured I can get started.

[the-nurk]

frankly this is wild that this isn't a built in feature already. this is like the bread and butter of installation and the whole purpose of package management. i'm not a contributor, but why wasn't this a part of the original feature set. there appears to be no real difference between downloading a zip and doing to pre/post install scripts vs downloading a repo and having pre/post install scripts for compiling.

alls well, nix will come to windows eventually and invalidate the need for scoop/chocolatey/winget/etc... all together.

what work needs to be done to get this working before then?

i'm going to experiment with just hacking it and hosting a repo clone locally as a zip file and pulling it in from there.

this worked

{
    "version": "2.32.3",
    "description": "direnv compiled from source so powershell hook works properly",
    "homepage": "https://direnv.net",
    "license": "MIT",
    "notes": "direnv should be on your path with the powershell hook appended to your powershell $PROFILE",
    "architecture": {
        "64bit": {
            "url": "\\path\\to\\direnv-source.zip",
            "hash": "b4fb33abdbda9cf1b8f65567402639c0b6a6c19fb0d7416aa599f30b64fe16bc",
            "extract_dir": "direnv"
        }
    },
    "depends": [
      "go",
      "make"
    ],
    "installer": {
        "script": [
          "cd $dir",
          "make install"
        ]
    },
    "bin": "direnv.exe"
}

[gitolicious]

For Go apps I am using a pretty straightforward way of building from source in a Docker container with a docker run installer script. Here is my example for k9s:

docker run
  --rm
  -v ${dir}:/app/out
  -w /app
  -e version=$version
  golang:1.22-alpine
  sh -c '
    apk --no-cache add --update make libx11-dev git gcc libc-dev &&
    git clone https://github.com/derailed/k9s.git &&
    cd k9s &&
    GOOS=windows GOARCH=amd64 make build &&
    mv execs/k9s /app/out/k9s.exe
  '

This is in a separate manifest with a -source suffix at the moment. https://github.com/gitolicious/scoopy-doo/blob/master/bucket/k9s-source.json

[bgshacklett]

Regarding use cases, the biggest I see at the moment is applications which don't have pre-built binaries available. This is one area where Homebrew has a major benefit over Scoop. For example, the aws-console program, which is part of the Rain repository, does not appear to be available as a pre-built standalone package. Homebrew's formula for the program uses Go to build and caches a "bottle": https://github.com/Homebrew/homebrew-core/blob/950ccca70ead2459946cdf14fedfc1fcf9119598/Formula/a/aws-console.rb#L25

As much as I love containers, they're not a viable option for many Windows users who don't have the ability to install a container runtime. In particular, in highly regulated environments, I've seen virtualization of any sort disabled and forbidden on end-user laptops. IMO, anything which requires administrative privileges at its core should be considered a last resort. The ability to install software without administrative privileges is one of the most useful things about Scoop.

Unfortunately, Make has its own set of problems on Windows. Makefiles are often written assuming a *nix environment and commands are available. This could probably be worked around by using msys2, or even a set of PowerShell functions which could serve as a compatibility layer, though I wish we could just stop assuming what the build userland looked like.

[W1M0R]

@nicksettoon Thanks for your tip! I managed to do something similar for meteor.

{
  "homepage": "https://github.com/stefanlogue/meteor",
  "description": "A highly configurable CLI tool for writing conventional commits",
  "version": "0.22.0",
  "license": "MIT",
  "architecture": {
    "64bit": {
      "url": "https://github.com/stefanlogue/meteor/archive/refs/tags/v0.22.0.zip",
      "hash": "ac34d37169c65d0dc26817c656f1d5a1d0a0c2a89bf69b2707b6b6d19b93e35b",
      "extract_dir": "meteor-0.22.0",
      "extract_to": "src"
    }
  },
  "bin": [ "meteor.exe" ],
  "depends": [
    "go"
  ],
  "pre_install": [
    "pushd \"$dir/src\"; go build -o ../meteor.exe; popd; [void](rd \"$dir/src\" -ea Continue -Recurse -Force)"
  ],
  "checkver": {
    "github": "https://github.com/stefanlogue/meteor"
  },
  "autoupdate": {
    "architecture": {
      "64bit": {
        "url": "https://github.com/stefanlogue/meteor/archive/refs/tags/v$version.zip"
      }
    }
  }
}

[yashamon]

This could be very nice for arm64 builds since some apps don't provide binaries, but they are relatively easy to build from source. Also I guess compiling on windows is hard is no longer true. At least for modern projects (fully compatible with llvm) compilation from command line on windows is sometimes easier than linux. Easier, because you can fine tune the development environmental variables to do anything you want without messing with the PATH variable.