search

sharplispers / cormanlisp

Corman Lisp
MIT License
570 stars 78 forks source link

Consider static linking of Visual C++ runtime to the executables #56

Closed ninejaguar closed 5 years ago

ninejaguar commented 5 years ago

Environment: Windows XP 32-bit OS and Windows 10 64-bit OS

Version: CormanLisp-3.1.2b.zip

Issue: Missing mfc140.dll error message appears after extracting Corman Lisp from CormanLisp-3.1.2b.zip and double-clicking on CormanLisp.exe.

Error Message: "CormanLisp.exe - Unable To Locate Component: This application has failed to start because mfc140.dll was not found. Re-installing the application may fix this problem."

Workaround: The workaround is for the user to locate and manually install the "Visual C++ Redistributable for Visual Studio 2015" as discussed here: https://social.msdn.microsoft.com/Forums/vstudio/en-US/ff26dc80-0fbc-4391-8460-880574e67e75/mfc140dll-missing-error-while-running-exe-file?forum=vcgeneral

arbv commented 5 years ago

Not an issue. The "Visual C++ Redistributable for Visual Studio 2015" is included into the installer package (MSI). The ZIP file (CormanLisp-3.1.2b.zip) is provided only as a fallback option.

ninejaguar commented 5 years ago

Not an issue. The "Visual C++ Redistributable for Visual Studio 2015" is included into the installer package (MSI). The ZIP file (CormanLisp-3.1.2b.zip) is provided only as a fallback option.

While that may be the intent, it can reduce exposure for Corman Common Lisp in corporate environments. SBCL is already blocked by corporations that forbid running setup/installers without having jealously guarded Admin rights. Luckily, Armed Bear Common Lisp and Clozure Common Lisp offer simple portable no-install versions. It just happened that I had "Visual C++ Redistributable for Visual Studio 2015" already installed on my workstation and was able to run Corman CL using the Zip file version.

However, a typical corporate user who doesn't already have the "Visual C++ Redistributable for Visual Studio 2015" installed will be unlikely to install it without having Admin rights, thereby restricting access to Corman CL.

The Zip file version shouldn't be looked on as a second class citizen, but rather as the only likely way to allow Corman CL on corporate workstations as restrictions continue to tighten over time.

arbv commented 5 years ago

@ninejaguar

The DLLs from the "Visual C++ Redistributable for Visual Studio 2015" are not meant to be distributed alongside the application (e.g. in a ZIP) so the only solution to this issue is to switch to static linking with "Visual C++ Redistributable".

It has its downsides though:

The benefit is that it will make the executables more portable (hopefully).

It is an option, I just haven't evaluated it thoroughly yet.

So, I hear you and am aware that there are users with different needs, but, considering the count of active contributors to the project, it is not possible to support all possible options, especially from the very beginning.

arbv commented 5 years ago

I have reopened the issue.

BTW, if this issue is really important for you and you know how to use Visual Studio 2015, you might consider grabbing the VS 2015 Community Edition installer and help me to evaluate the possibility of static linking (it is easy, no understanding of codebase is actually required).

I would be very appreciated.

binghe commented 5 years ago

It's possible to statically link MFC but impossible [?] to statically link MSVCRT. Either the end user needs to install Visual C++ Redistributable, or manually copy the needed 2 DLLs into the same folder with Corman Lisp.

arbv commented 5 years ago

@binghe

As far as I know, MSVCRT (the C/C++ runtime itself) is statically linkable as well. The /MT compiler's switch instructs the compiler to generate the code which is suitable for what we are talking about.

https://docs.microsoft.com/en-us/cpp/build/reference/md-mt-ld-use-run-time-library?view=vs-2017

In fact, I was not sure about the ability to statically link MFC :)

binghe commented 5 years ago

@arbv, ok, maybe I'm wrong. For statically linking MFC, in VS 2015 project settings -> General, I see the property "Use MFC" has two options: Use MFC in a Shared DLL and Use MFC in a Static Library, isn't this what you wanted? (however, to make it really work one also needs to use /MT instead of /MD.

arbv commented 5 years ago

@binghe Yes, it seems that it is what is needed.

Zulu-Inuoe commented 5 years ago

I think it makes more sense to always link dynamically and instead distribute the necessary dll's with the zip package (not the installer). This is listed as preferred method 3 in https://docs.microsoft.com/en-us/cpp/ide/choosing-a-deployment-method?view=vs-2017

Additionally for MFC dll's, see this following page: https://docs.microsoft.com/en-us/cpp/ide/redistributing-the-mfc-library?view=vs-2017

In particular, the MFC dll's were unintentionally missing from the redist directory in 2015, so they explicitly mention to use the ones installed to system32.

The advantage of keeping it as dynamically linked is that it keeps the build process simpler (single version), less chance of weirdness when calling other libraries that have linked dynamically due to split heaps, and it gives the user the ability to use the system provided libraries if they are available by just deleting the dll's from Corman's bin directory

arbv commented 5 years ago

@Zulu-Inuoe

Thanks for the valuable information.

The approach which you suggested seems like a compromise.

Unfortunately, MFC version which is used by Corman Lisp relies on Universal C Runtime, which is problematic to distribute in the form of DLL files alongside the application in a ZIP archive as there are a lot of them (see the attachment, the list is not complete).

You can read about it here (Local deployment):

https://docs.microsoft.com/en-us/cpp/ide/universal-crt-deployment?view=vs-2017

Using static linking seems like a lesser evil, but it has its downsides too (one of which you have mentioned). On the positive side, it could make the applications created by Corman Lisp more portable as well.

So, it seems, there is no ultimate solution to the problem.

ucrt

arbv commented 5 years ago

@Zulu-Inuoe

Not all of the listed libraries are actually used, but it is safer to distribute all of them (See the CormanLispServer.dll imports table). It is actually mentioned in the documentation.

imports

arbv commented 5 years ago

Here is another possibility to use DLLs without relying on api-ms-win-*.dll: https://stackoverflow.com/a/45984650

This solution is hacky, to be honest.

A more detailed discussion: https://social.msdn.microsoft.com/Forums/en-US/a28331ae-19a3-4a34-b3ba-1e8fd4430375/missing-apimswincore-dlls

So yeah, there are a lot of options to investigate.

UPD: I was wrong, this option does not solve the problem of distributing 40-odd DLLs.

Zulu-Inuoe commented 5 years ago

For Vista+ I believe it's safe to assume the UCRT is present (on computers updated after 2015). Windows 10+ has it as a core system component.

But if you want to be able to run on XP, then it is an issue indeed. I don't know what's -commonly- done, but I've certainly seen other applications locally distribute the UCRT as well. Though I agree it's very ugly.

Can I make the suggestion then that since you're already considering differing builds (dynamic vs static) you instead consider two build options:

1. XP - Either statically link, or otherwise bundle all the dll's 2. Vista+ - Redistribute vc runtime and mfc, but don't distribute UCRT (assume they're already on the system)

2 of course relies on the assumption that the UCRT is already installed on those platforms. I'll need to do more reading before I can swear to that.

arbv commented 5 years ago

@Zulu-Inuoe

No, I am not considering multiple builds. This is out of the question. It is too much effort. There is going to be only one build distributed in the different forms (zip file and installer).

Considering the number of active contributors it makes more sense to drop some options, rather than to add new ones.

So no, no mutiple configurations, sorry.

  1. Vista+ - Redistribute vc runtime and mfc, but don't distribute UCRT (assume they're already on the system)

2 of course relies on the assumption that the UCRT is already installed on those platforms. I'll need to do more reading before I can swear to that.

It is installed on Vista+ only if the system was updated via Windows Update (or UCRT was installed manually, which is unlikely).

Zulu-Inuoe commented 5 years ago

Apologies. I misunderstood and thought you were considering making a statically linked version vs a dynamically linked one.

rgcorman commented 5 years ago

It’s great to see all the interest in Corman Lisp, and ongoing support.

I believe that it’s better to link dynamically to the MSVCRT dll. This is because the dynamic library can be easily accessed from Lisp, that is code written in Lisp can easily call any of its functions via the FFI. I believe that some parts of the system make use of this and thus need the dll to be present. I am not sure if you would run into problems if you linked statically and then also had the dynamic library, it could be a problem having both static and dynamic libraries (or possibly two different versions) if those libraries in turn are calling some other dynamic library in the OS. You really can’t get away from dlls, the whole OS is built on them, and they are great for programming against.

I am not sure the source of the concern that you can’t redistribute those DLLs, I always thought they were redistributable. Earlier versions of Corman Lisp always included them in the install package.

Roger

From: Wilfredo Velázquez-Rodríguez notifications@github.com Sent: Wednesday, March 20, 2019 7:35 AM To: sharplispers/cormanlisp cormanlisp@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [sharplispers/cormanlisp] Consider static linking of Visual C++ runtime to the executables (#56)

I think it makes more sense to always link dynamically and instead distribute the necessary dll's with the zip package (not the installer). This is listed as preferred method 3 in https://docs.microsoft.com/en-us/cpp/ide/choosing-a-deployment-method?view=vs-2017

Additionally for MFC dll's, see this following page: https://docs.microsoft.com/en-us/cpp/ide/redistributing-the-mfc-library?view=vs-2017

In particular, the MFC dll's were unintentionally missing from the redist directory in 2015, so they explicitly mention to use the ones installed to system32.

The advantage of keeping it as dynamically linked is that it keeps the build process simpler (single version), less chance of weirdness when calling other libraries that have linked dynamically due to split heaps, and it gives the user the ability to use the system provided libraries if they are available by just deleting the dll's from Corman's bin directory

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/sharplispers/cormanlisp/issues/56#issuecomment-474857972, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAOZ4zFs3-nFAgA3yjx7XAw0uS603hUUks5vYkcCgaJpZM4b6-mz.

arbv commented 5 years ago

@Zulu-Inuoe

My apologies, probably I misunderstood you. I thought you suggested to make and support two builds - a dynamically linked one and a statically linked one and release both of them. Sorry, my native language is not English, reading and writing in it is still like through a filter for me.

Indeed, I was considering static build vs dynamic build (with a bunch of DLLs in a ZIP file). It seems that it boils down to these two options. And I am not really sure which one is better.

arbv commented 5 years ago

@rgcorman

Thanks for the valuable information, Roger! The facts which you mentioned are the ones which I completely lost out of sight.

OK then, I will consider the possibility of distributing 40+ dynamic load libraries inside the ZIP archive, if Microsoft really wants it that way.

ninejaguar commented 5 years ago

OK then, I will consider the possibility of distributing 40+ dynamic load libraries inside the ZIP archive, if Microsoft really wants it that way.

I've uninstalled the "Visual C++ Redistributable for Visual Studio 2015" from the Windows XP 32 Bit and Windows 10 64 bit laptops, and confirmed that the error in attempting to run the zipped version of Corman CL returns on both machines. If you decide to make a new Zip with the dlls available, then I can test it on those two environments now that they're missing the dlls.

arbv commented 5 years ago

As it turned out, obtaining required DLL files is easier to do than I thought - it is possible to get the necessary DLL files directly from the unpacked MSI. As the ZIP archive is created from the MSI installer, it helps a bit to automatise the ZIP archive creation.

Thanks to everyone for brainstorming the issue!

@ninejaguar May I ask you to test the attached builds?

UPD: I believe it is enough to test it on Windows XP.

CormanLisp-master.zip CormanLisp-3.1.2b-with-crt.zip

Zulu-Inuoe commented 5 years ago

@arbv Ah I see. No worries, I don't think I was clear enough to begin with.

I agree that having two separate builds is far too much work. And to be clear, if there is only one build (which again, I support), I do not believe it would be a viable option to statically link.

Depending on what you wish to target, I think it could be reasonable to only ship the necessary C++ dependencies (vcruntime, mfc, etc.) without the UCRT. Doing so prevents portable usage of Corman on XP and non-updated machines, but I believe that is a non-issue in practice since:

This would prevent needing to incorporating those libraries into the build, as well as saving the Corman bin directory from needing 40+ dll files.

All that that said, I'm not against including them, either. I'm just highlighting options.

Also to elaborate on what I mentioned earlier, with regards other applications which take the approach of shipping the UCRT, here's a list of some applications on my PC that do so:

So it's not unheard of to do this, as unaesthetic as it looks.

Apologies for the wall of text. Thanks!

arbv commented 5 years ago

@Zulu-Inuoe I think we have settled this out - we will distribute the (almost) full Visual C++ Runtime (~60 DLLs) in the ZIP archive in the following releases. It is ugly, but it seems to be the best/most compatible option.

By the way, thanks for the provided links, I have found very valuable information there - how to distribute the local copy of CRT and where to look for the necessary files. We might need this in the future (in particular if/when we migrate to VS2017 - there might be some problems when including Visual C++ Redistributable into the MSI installer).

ninejaguar commented 5 years ago

@arbv

As it turned out, obtaining required DLL files is easier to do than I thought - it is possible to get the necessary DLL files directly from the unpacked MSI. As the ZIP archive is created from the MSI installer, it helps a bit to automatise the ZIP archive creation.

Thanks to everyone for brainstorming the issue!

@ninejaguar May I ask you to test the attached builds?

UPD: I believe it is enough to test it on Windows XP.

CormanLisp-master.zip CormanLisp-3.1.2b-with-crt.zip

They both work on Windows XP 32-bit!

What a quick solution to ensure that Corman CL is available to the widest possible audience for the longest possible period. Great job!

UPD: After confirming that the new Zips containing the dlls worked, I then reinstalled the "Visual C++ Redistributable for Visual Studio 2015" on Windows XP 32-bit to check if there might be any issues with the new Zips that already contained the dlls. There were no apparent issues. They worked just as expected.

arbv commented 5 years ago

@ninejaguar

Thanks! I have added the ZIP file with DLLs to the project's download section.