search

elm-lang / elm-platform

Bundle of all core development tools for Elm
BSD 3-Clause "New" or "Revised" License
762 stars 125 forks source link

Why not distribute statically linked binaries? #193

Closed naddeoa closed 7 years ago

naddeoa commented 7 years ago

I've just spent a few days beating my head against a wall trying to get the elm suite running in an environment that wasn't considered that I have very little control over. In the end, I got it working by building the elm platform from source and statically linking the binaries that were produced, which was a pretty small update to the cabal file:

ghc-options:
    -threaded -O2 -W -static -optl-static -optl-pthread

What are the reasons that they're distributed dynamically linked? Were there licensing issues, or was the binary size a concern?

process-bot commented 7 years ago

Thanks for the issue! Make sure it satisfies this checklist. My human colleagues will appreciate it!

Here is what to expect next, and if anyone wants to comment, keep these things in mind.

jvoigtlaender commented 7 years ago

I'm pretty sure I remember that the reduction in binary size was celebrated.

naddeoa commented 7 years ago

It is nice that the binaries are smaller, but it comes at the expense of them running at all in some Linux environments. It also lead to documentation that mentions which Ubuntu packages to install to make it work, which is great for Ubuntu users but leaves other people hanging.

It wouldn't really have to be all the binaries that get statically linked, just the ones distributed via npm. The Mac and Windows installers can naturally install whatever they want. There is definitely more predictability in MacOS environments.

evancz commented 7 years ago

@jvoigtlaender is referring to the use of split-objs: True which is unrelated to linking.

I didn't know there were flags for these sorts of things. I have some questions for you:

naddeoa commented 7 years ago

Yep, they're there! And they're all documented in the man page for ghc too. Here is an online version for reference.

https://linux.die.net/man/1/ghc

The optl-static is the way that ghc allows users to pass options through a build and to other programs. It is really opt<program-letter>-<program-arg>, where l stands for linker, which you'll remember needing to run when you compile C files. It's the same thing as specifying -static when linking binaries on the command line. gcc does a similar thing IIRC with options like -lstatic.

The -static option on the other hand is specific to ghc. The man pages are light on this but I have to assume that its doing the same thing the linker is doing but for Haskell libraries. That probably involves downloading static versions of some libraries during the build. I'm definitely not an expert on ghc by any means; I pretty much did what I had to do to get it up and running on Ubuntu (not a lot of fun).

The -pthread option is more mysterious to me. It Adds support for multithreading with the pthreads library according to the gcc man pages. I can't tell you why we need that for static linking. It might be the case that we can just ignore it for some or all of the Elm binaries.

Here are the sizes for the major binaries. It's actually better than I thought. I actually only need elm-make for my stuff but I compiled them all to see.

             dynamic  static
elm          1.3M     2.5M 
elm-make     9.6M      11M 
elm-package  6.5M     7.9M 
elm-repl     4.7M     6.0M

I'm having some issues running running some of them. elm-make seems to work just fine. elm-package dies when it attempts to download packages. elm-repl keeps complaining about single quotes no matter what I do. I'm sure there are solutions though.

I'd be willing to look into it more if no one is against distributing the binaries statically. I mean, I'm going to be recompiling elm-make statically anyway so I can use it in my own build environment.

evancz commented 7 years ago

Thanks for explaining all that and running the numbers!

Yeah, I'm interested in making our linux binaries more reliable, and I think it'd be really valuable to incorporate the things you learn in the official releases. So go for it!

I have not heard of folks having linking issues on Windows or Mac, so one option is to only do this additional linking for the linux binaries. (If we can find evidence that this is a problem on other platforms though, I have no objection! Just don't want to solve problems that are only theorized.)

Note: make sure to bring up your learnings on the elm-dev mailing list when a new release of Elm is in the pipeline. It will be way more efficient to batch the changes you are suggesting with the normal testing phase.

naddeoa commented 7 years ago

Will do.

It's probably best to just do Linux at first. I don't know much about how any of this works in the Windows world and I haven't used a Windows machine in a while.

Are there any docs or wikis about the release process? I didn't know there was a normal testing phase, or what that involves. And should I close this issue and move it entirely to elm-dev?

naddeoa commented 7 years ago

Just to update; I came across a similar issue on github: https://github.com/leanprover/lean/issues/557

The seg faults have to do with threading. If I remove the -threaded then the static binaries work fine. I kept the linker threading options though, so that tells me that it could be a bug in our version of haskell, which seems to be true in at least one other case: http://stackoverflow.com/questions/25386520/haskell-compiled-program-giving-a-segfault.

Logical questions from here are:

naddeoa commented 7 years ago

Just started a thread up on elm-dev. I'll close this out and talk there.

https://groups.google.com/forum/#!topic/elm-dev/Q8ZjKRpKX1g

domenkozar commented 7 years ago

As I'm working on another Haskell project doing static OSX and Linux builds, I'll weight in a bit.

For -threaded try using musl instead of gcc. See this post for more info.

Instead of -split-objs, use -split-sections. See this commit for more details.

For OSX I'm currently fiddling with how to compile everything statically except m, libSystem and other OSX specific libraries. It's tricky to get right.

PS: I suggest using nsis package for windows: https://github.com/input-output-hk/pos-haskell-prototype/blob/master/installers/WindowsInstaller.hs

naddeoa commented 7 years ago

Thanks a ton! Those are great resources, that definitely saves me a ton of time. It looks like there is even hope of reducing the binary sizes, which will make @rtfelmdan super happy.

For the OSX build,j did you want to leave the OSX libraries dynamic because its the "right" thing to do, or because the binary actually won't work if you make them static in the future?

On Wed, Jan 4, 2017 at 9:36 AM Domen Kožar notifications@github.com wrote:

As I'm working on another Haskell project doing static OSX and Linux builds, I'll weight in a bit.

For -threaded try using musl instead of gcc. See this post https://www.fpcomplete.com/blog/2016/10/static-compilation-with-stack for more info.

Instead of -split-objs, use -split-sections. See this commit https://github.com/NixOS/nixpkgs/pull/21589/commits/29277f1c4735280228801405e7ba7b6f4b977121 for more details.

For OSX I'm currently fiddling with how to compile everything statically except m, libSystem and other OSX specific libraries. It's tricky to get right.

PS: I suggest using nsis package for windows: https://github.com/input-output-hk/pos-haskell-prototype/blob/master/installers/WindowsInstaller.hs

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/elm-lang/elm-platform/issues/193#issuecomment-270433609, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLTLbnmGWNbFY9Cq6OvJc78KGhA4FEGks5rO9iQgaJpZM4LYgey .

domenkozar commented 7 years ago

Because some of the OSX libs don't ship with static version. At least I couldn't find one.