search

processone / ejabberd

Robust, Ubiquitous and Massively Scalable Messaging Platform (XMPP, MQTT, SIP Server)
https://www.process-one.net/ejabberd/
Other
6.11k stars 1.51k forks source link

Could not compile dependency :fast_tls, fatal error: 'openssl/err.h' file not found #include <openssl/err.h> #3259

Closed anildigital closed 4 years ago

anildigital commented 4 years ago

Environment

Errors


user@MacBookPro ~/C/e/p/phoenix_ejabberd (master)> echo $PKG_CONFIG_PATH
/usr/local/opt/openssl@1.1/lib/pkgconfig
user@MacBookPro ~/C/e/p/phoenix_ejabberd (master)> echo $LDFLAGS
-L/usr/local/opt/openssl@1.1/lib
user@MacBookPro ~/C/e/p/phoenix_ejabberd (master)> echo $CPPFLAGS
-I/usr/local/opt/openssl@1.1/include
user@MacBookPro ~/C/e/p/phoenix_ejabberd (master)> mix compile
===> Compiling fast_tls
===> Compiling /Users/user/Code/elixir/phoenix/phoenix_ejabberd/deps/fast_tls/c_src/fast_tls_drv.c
===> /Users/user/Code/elixir/phoenix/phoenix_ejabberd/deps/fast_tls/c_src/fast_tls_drv.c:21:10: fatal error: 'openssl/err.h' file not found
#include <openssl/err.h>
         ^~~~~~~~~~~~~~~
1 error generated.

** (Mix) Could not compile dependency :fast_tls, "/Users/user/.mix/rebar3 bare compile --paths="/Users/user/Code/elixir/phoenix/phoenix_ejabberd/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile fast_tls", update it with "mix deps.update fast_tls" or clean it with "mix deps.clean fast_tls"
user@MacBookPro ~/C/e/p/phoenix_ejabberd (master) [1]> elixir --version
Erlang/OTP 22 [erts-10.7.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.10.3 (compiled with Erlang/OTP 22)
user@MacBookPro ~/C/e/p/phoenix_ejabberd (master)>

Bug description

Not able to compile ejabberd hex package on macOS with provided details above.

      {:ejabberd, "~> 20.4"},
Freyskeyd commented 4 years ago

Did you try to export the related flags describe in the fast_tls repository?

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CFLAGS="-I/usr/local/opt/openssl/include/"
export CPPFLAGS="-I/usr/local/opt/openssl/include/"

https://github.com/processone/fast_tls#osx-build-example

JohnB commented 4 years ago

Those exports worked for me - thanks!

luzaranza commented 2 years ago

There appears to be no support whatsoever for this on Windows 10.

luzaranza commented 2 years ago

Attempting to run a mix deps.get dumps you out to a missing cl.exe message --> then installing some of the visual studio MSVC core will not get you any further, since clearly the compiler env has no idea about the configuration of this package and spits out a host of warnings and errors out 1) because the c++ compiler complains about memory allocation for dynamic arrays 2) then has no idea how to run any makefiles. Trying to install a pseudo bash eg git bash appears to get you further if you reference a pre-installed msys ming64 environment with the openssl dev files already pre-referenced, and appears to write some config files (although they look minimal / as good as empty - indicating no useful config written) and trying to then run a "make" command (I tried chocolatey make which doesn't know how to pick up the config - the make in msys ming64 doesn't appear to undersand the local directory) simply returns no target or makefile found. If anyone has any ideas on how to somehow get these erlang libraries compiled for Windows, I'd be very grateful for any grains of wisdom here ...

licaon-kter commented 2 years ago

Did you read https://www.process-one.net/blog/install-ejabberd-on-windows-10-using-docker-desktop/ ?

luzaranza commented 2 years ago

Dear Licaon_Kter, No I haven't. I'll pass this information to my dev team (We are a team of 2 - a small struggling fledgling company in latin america trying to provide interactive language learning using a Phoenix-Elixir/react Hooks platform. To be perfectly honest with you, we are more than willing to try out possibilities to enable a devoted stun server - up to now we have to use the Zoom app with our own website - our company director now wants us to be "Zoom-independent" so the need to install our own stun/turn server. I apologise if I sound a bit dumb here on not being on the "up-and-up" regarding your fantastic product, but .. [we are too damn busy with our day-to-day rollouts for our English course units ...], so please bear with us if we don't know anything about Ejabberd for windows desktop. We simply came across this upon reading alternatives for an inhouse stun server (the other one we are reading about is mongooseice - although this one appears too heavy duty for our needs). Yes, we would like to have pure linux machines (I have more than 20 years experience as a solutions developer in investment banking using various flavours of unix tools and perl-based transaction processing sytems for critical apps), but alas we have to use Windows machines. Best regards, Paul. On Thursday, 7 July 2022 at 12:02:21 GMT-5, Licaon_Kter @.***> wrote:

Did you read https://www.process-one.net/blog/install-ejabberd-on-windows-10-using-docker-desktop/ ?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

luzaranza commented 2 years ago

Did you actually read and try to understand our issue??????? We perused and attempted to run the instructions contained in your link. After a few aborted/botched attempts by your installer script, we managed to get an "ejabberd dockerised folder" installed using Windows 10.But our objective was: have the required fast_tls/stun source files compiled and installed using Elixir mix.How would this be possible using your "container" load that we now see here?

On Thursday, 7 July 2022 at 12:02:21 GMT-5, Licaon_Kter ***@***.***> wrote:

Did you read https://www.process-one.net/blog/install-ejabberd-on-windows-10-using-docker-desktop/ ?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

licaon-kter commented 2 years ago

What that link says is this: "On Windows ejabberd supports this method of install"

Any other? You might be on your own. :(

I'm not a dev here so maybe wait for them to chime in.

luzaranza commented 2 years ago

Sir, The problem in Windows 10 mix compile appears to be with fetching a certain "pc v1.14.0" - called by p1_utils ?? maybe this is an Erlang dependency which somewhere along the way is being mis-interpreted (we can't determine at this point if it is being called by Elixir mix or Erlang rebar or another child program called in chain of events). We will continue investigating - but clearly it appears that this install chain is not Windows 10 ready. ;( (We ran mix compile through wsl fine with no problems - but we can't use wsl for production installs - just dev for time being - so unfortunately we are stuck on Windows 10 ...). If we extract the fast_tsl master and manually attempt a compile then we are confronted with the MSVC compiler ... ;{ haha ... life goes on - there will surely be a solution, we realise that one has to have patience. Thanks for your reply - apologies for our frustrations. On Tuesday, 12 July 2022 at 10:21:54 GMT-5, Licaon_Kter @.***> wrote:

What that link says is this: "On Windows ejabberd supports this method of install"

Any other? You might be on your own. :(

I'm not a dev here so maybe wait for them to chime in.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

badlop commented 2 years ago

That pc is port_compiler, used for compiling C code. That is required by several ejabberd dependencies that include C code, all the fast_*. Maybe there are examples out there how to use port_compiler on Microsoft Windows.

luzaranza commented 2 years ago

Yes, this is definitely getting closer. At this point in the chain the process is picking up a default Microsoft Visual Studio batch programcalled cl.exe which doesn't know how to handle this dependency and clearly appears to be the wrong compiler. Out of curiosity, we ran the same Elixir mix compile command in wsl linux and saw that the fast_tls dependency compiled ok - we then went back to Windows 10 and ran the same Elixir mix compile commmand - and low and behold - the fast_tls dependency installed without any problems. On closer investigation, we can see a fast_tls_build\default\plugins\pc\ebin directory containing several Erlang beam files. This dir gets repopulated each time a clean Elixir mix compile commmand is run - funnily enough these beam files are not overwritten when switching back to Windows 10 and re-running - presumably this is controlled by Elixir timestamps. We have installed a Hex package rebar3 port compiler for native code version 1.14.0, but as yet are unable to configure it to run the right compiler for these erlang/Ejabbberd dependencies. We will keep searching for examples and see if we can get this configured. Many thanks for this.

On Tuesday, 12 July 2022 at 13:59:36 GMT-5, badlop ***@***.***> wrote:

That pc is portcompiler, used for compiling C code. That is required by several ejabberd dependencies that include C code, all the fast*. Maybe there are examples out there how to use port_compiler on Microsoft Windows.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

luzaranza commented 2 years ago

When we run the Elixir mix compile in Windows against fast_tls, we can see that no files are written to the c_src sub-directory. We also receive Microsoft C++ error messages indicating that this tool has no understanding of the input parameters given to it. Also, a vc140.pdb (program debug file) is written out - presumably by the Microsoft Visual Studio cl.exe build script.There is also an _ebin folder written out - which one assumes contains an attempt to produce erlang beam files and config. We have a wls (Windows Subsystem for Linux) soft-link pointing at the same fast_tls folder.When we run the Elixir mix compile in wsl, we can see that indeed files are written to the c_src sub-directory (.c and .d files amongothers), and the _ebin folder is updated with what appears to be correctly configured erlang beam files and config.We also receive a success message. Interestingly enough, when we then re-run the same Elixir mix compile, but this time in Windows against fast_tls, the command alsosucceeds. We can see that all files in the fast_tls folder are preserved and no timestamps are updated. The _ebin folder is notupdated and remains intact. We can even install the stun server on top since it appears to accept the fast_tls dependency But we are completely lost as to how to make the windows version compile like the linux version in the first place. We have checked the rebar config - these appear to be files that can't be edited -  and all the environment variables - but we simply cannot see how to tell the windows version to compile fast_tls like the linux one - is it rebar or the port compiler? if so, how do we change the config? We feel it may be an Erlang config thing? We don't have experience in managing Erlang configs, we can handle it in Elixir mix, but that is the limit of our knowledge. On Tuesday, 12 July 2022 at 13:59:36 GMT-5, badlop @.***> wrote:

That pc is portcompiler, used for compiling C code. That is required by several ejabberd dependencies that include C code, all the fast*. Maybe there are examples out there how to use port_compiler on Microsoft Windows.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

weiss commented 2 years ago

There appears to be no support whatsoever for this on Windows 10.

  1. Originally, this issue was about macOS.
  2. Compiling NIFs on Windows is a PITA.
  3. Even more so if they depend on external libraries.
  4. Both the Erlang/OTP include files and the Rebar3 port compiler assume a Microsoft Visual C++ (MSVC) environment on Windows. Trying to use MinGW or WSL to build native Windows binaries will be even more of a PITA.

Attempting to run a mix deps.get dumps you out to a missing cl.exe message --> then installing some of the visual studio MSVC core will not get you any further, since clearly the compiler env has no idea about the configuration of this package and spits out a host of warnings and errors out 1) because the c++ compiler complains about memory allocation for dynamic arrays 2) then has no idea how to run any makefiles.

I looked into compiling fast_yaml and fast_tls on Windows (but didn't check other dependencies). You're right that (at least) those two need adjustments to support an MSVC environment. I submitted PRs with the necessary changes:

I created an example Rebar3 project (not Mix, sorry) that depends on the modified libraries and has a GitHub Actions workflow for building them on Windows Server 2022 (the resulting binary artifacts can be downloaded from the Actions page for a while). As you can see in the workflow file, what it does is:

  1. Set up an MSVC environment.
  2. Build and install (static versions of) LibYAML and OpenSSL using vcpkg.
  3. Point MSVC to those by setting CFLAGS/LDFLAGS appropriately.
  4. Run rebar3 compile as usual.

We simply came across this upon reading alternatives for an inhouse stun server

For what it's worth, there's also a standalone variant of ejabberd's STUN/TURN support called eturnal, which has a proper Windows installer.

Neustradamus commented 2 years ago

@luzaranza: With improvements from @weiss, it is good now?

luzaranza commented 2 years ago

Many thanks for this. And please accept my apologies for not replying sooner. For the past 2 months we have been occupied with rollouts. I will attend to this and certainly will investigate the eturnal alternative. Much appreciated.

On Sunday, 17 July 2022, 06:33:04 GMT-5, Holger Weiß ***@***.***> wrote:

There appears to be no support whatsoever for this on Windows 10.

Attempting to run a mix deps.get dumps you out to a missing cl.exe message --> then installing some of the visual studio MSVC core will not get you any further, since clearly the compiler env has no idea about the configuration of this package and spits out a host of warnings and errors out 1) because the c++ compiler complains about memory allocation for dynamic arrays 2) then has no idea how to run any makefiles.

I looked into compiling fast_yaml and fast_tls on Windows (but didn't check other dependencies). You're right that (at least) those two need adjustments to support an MSVC environment. I submitted PRs with the necessary changes:

I created an example Rebar3 project (not Mix, sorry) that depends on the modified libraries and has a GitHub Actions workflow for building them on Windows Server 2022 (the resulting binary artifacts can be downloaded from the Actions page for a while). As you can see in the workflow file, what it does is:

We simply came across this upon reading alternatives for an inhouse stun server

For what it's worth, there's also a standalone variant of ejabberd's STUN/TURN support called eturnal, which has a proper Windows installer.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>