search

esl / MongooseIM

MongooseIM is Erlang Solutions' robust, scalable and efficient XMPP server, aimed at large installations. Specifically designed for enterprise purposes, it is fault-tolerant and can utilise the resources of multiple clustered machines.
Other
1.64k stars 422 forks source link

Compilation failing on ubuntu 16.04 #1361

Closed shanbhardwaj closed 6 years ago

shanbhardwaj commented 6 years ago

MongooseIM version: (2.0.1) Installed from: (source) Erlang/OTP version: (19.3.6)

Describe the issue. make command fails with the output attached in the file here. 

==> lz4 (compile)
==> semver (compile)
==> pooler (compile)
==> re2 (compile)
Fetching origin
HEAD is now at 09fc9ce... Make SimplifyWalker::SimplifyRepeat() use Regexp::Concat().
make[1]: Entering directory '/home/deploy/MongooseIM/deps/re2/c_src/re2'
c++ -m64 -c -o obj/re2/compile.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/compile.cc

c++ -m64 -c -o obj/re2/filtered_re2.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/filtered_re2.cc

c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report, with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
Makefile:178: recipe for target 'obj/re2/compile.o' failed
make[1]: *** [obj/re2/compile.o] Error 4
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/home/deploy/MongooseIM/deps/re2/c_src/re2'
ERROR: Command [compile] failed!
Makefile:17: recipe for target 'compile' failed

make-log.txt

shanbhardwaj commented 6 years ago

Also linked to this issue https://github.com/google/re2/issues/148

fenek commented 6 years ago

This library compiles 100% fine with gcc 4.x. Can you please try this version?

Arslan007 commented 6 years ago

I have same problem. I tried to compile and got same error and checked my gcc version

/usr/src/MongooseIM# gcc --version 
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

As per your suggestion I downgraded my gcc verson to 4.8 

usr/src/MongooseIM# gcc --version 
gcc (Ubuntu 4.8.5-4ubuntu2) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

But having same issue and unable to compile.

/usr/src/MongooseIM# ./rebar3 compile
===> Verifying dependencies...
===> Skipping goldrush (from {pkg,<<"goldrush">>,<<"0.1.7">>,
                                     <<"349A351D17C71C2FDAA18A6C2697562ABE136FEC945F147B381F0CF313160228">>}) as an app of the same name has already been fetched
===> Compiling re2
Fetching origin
HEAD is now at 09fc9ce... Make SimplifyWalker::SimplifyRepeat() use Regexp::Concat().
make: Entering directory '/usr/src/MongooseIM/_build/default/lib/re2/c_src/re2'
c++ -m64 -c -o obj/re2/dfa.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/dfa.cc
c++ -m64 -c -o obj/re2/filtered_re2.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/filtered_re2.cc
c++ -m64 -c -o obj/re2/mimics_pcre.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/mimics_pcre.cc
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
Makefile:178: recipe for target 'obj/re2/dfa.o' failed
make: *** [obj/re2/dfa.o] Error 4
make: *** Waiting for unfinished jobs....
make: Leaving directory '/usr/src/MongooseIM/_build/default/lib/re2/c_src/re2'
===> Hook for {pc,compile} failed!

@fenek Please suggest!!

Arslan007 commented 6 years ago

Did tried with clean but this time error is differant

g++ -O3 -finline-functions -Wall -fPIC -I /usr/lib/erlang/erts-7.3/include/ -I /usr/lib/erlang/lib/erl_interface-3.8.2/include  -c -o /usr/src/MongooseIM/_build/default/lib/snappy/c_src/snappy/snappy-stubs-internal.o /usr/src/MongooseIM/_build/default/lib/snappy/c_src/snappy/snappy-stubs-internal.cc
cc /usr/src/MongooseIM/_build/default/lib/snappy/c_src/snappy_nif.o /usr/src/MongooseIM/_build/default/lib/snappy/c_src/snappy/snappy-sinksource.o /usr/src/MongooseIM/_build/default/lib/snappy/c_src/snappy/snappy.o /usr/src/MongooseIM/_build/default/lib/snappy/c_src/snappy/snappy-stubs-internal.o -shared -lstdc++ -L /usr/lib/erlang/lib/erl_interface-3.8.2/lib -lerl_interface -lei -o /usr/src/MongooseIM/_build/default/lib/snappy/c_src/../priv/snappy_nif.so
/usr/bin/ld: cannot find -lstdc++
collect2: error: ld returned 1 exit status
Makefile:57: recipe for target '/usr/src/MongooseIM/_build/default/lib/snappy/c_src/../priv/snappy_nif.so' failed
make: *** [/usr/src/MongooseIM/_build/default/lib/snappy/c_src/../priv/snappy_nif.so] Error 1
make: Leaving directory '/usr/src/MongooseIM/_build/default/lib/snappy/c_src'
===> Hook for compile failed!
Arslan007 commented 6 years ago

@fenek Any update on this issue. Thanks

fenek commented 6 years ago

I'm focused on 2.1.0beta2 release at the moment. I'll have some time later to create Ubuntu VM and check this problem. Anyway cannot find -lstdc++ indicates that maybe the dev package of C++ standard library is missing. Please try sudo apt-get install libstdc++-dev

Arslan007 commented 6 years ago

Thanks Fenek!

Take your time. I tried your suggested suggestion but same issue although I already have latest libstdc++

libstdc++-7-dev is already the newest version (7.1.0-5ubuntu2~16.04).

shanbhardwaj commented 6 years ago

@fenek I was able to get around this issue by using gcc-7. You can close this issue if you'd like.

Arslan007 commented 6 years ago

@shanbhardwaj How did you fixed the error. I am having libstdC++-7 and gcc to latest.

Arslan007 commented 6 years ago

@fenek I am able to proceed further by setting gcc version 7 but now I am getting an exception on next step. I have attached crash dump.

===> Compiling idna
eheap_alloc: Cannot allocate 177423032 bytes of memory (of type "old_heap").

Crash dump is being written to: erl_crash.dump...done
Makefile:28: recipe for target 'rel' failed
make: *** [rel] Error 1

erl_crash.dump.zip

Arslan007 commented 6 years ago

@fenek Sorry for being more curious on that since I am struggling with that and I know you are busy. Can you please give me some hint what I am doing wrong!!

shanbhardwaj commented 6 years ago

@Arslan007

i was able to compile this on a 512MB RAM VPS. I used

fenek commented 6 years ago

@Arslan007

It's strange that Erlang can't allocate ~170MB of memory for compiling something. Are you trying to compile on some machine with very limited resources?

Arslan007 commented 6 years ago

@fenek No I am using Azure VM with 512 MB memory and it was failing on that and I don't know why. Then as per @shanbhardwaj I created one droplet of same specs on Digital ocean just for sake of fun and to double checks and I experienced little bit different issue like on first try of compile it gives me error like this KILLED and then I did make clean and its compiled successfully but taking so much time. Since MongooseIM is compiled and I am trying to run MongooseIM server from mongooseimctl commands and its not working. When I do ./mongooseimctl start and there is no error but I want to check the status from mongooseim by this command ./mongooseimctl status then it gives me below error like node is down.

Failed RPC connection to the node mongooseim@localhost: nodedown

Commands to start a MongooseIM node:
  start  Start a MongooseIM node in server mode
  debug  Attach an interactive Erlang shell to a running MongooseIM node
  live   Start MongooseIM node in live (interactive) mode
MongooseIM cluster management commands:
  join_cluster other_node_name                Add current node to cluster
  leave_cluster                               Leave current node from the cluster
  remove_from_cluster other_node_name         Remove dead node from the cluster

I have attached the logs and crash dump as well! Log&Dump.zip

P.S: I didnot changed any configs in bin folder and trying to start with default configurations of MongooseIM

fenek commented 6 years ago

Unfortunately I'm not sure what could be causing the problem in this environment, as I've recently used MongooseIM on Azure VM but with more memory... so it's my best bet - that 512MB is not enough. When I start MIM on Ubuntu 16.04 server edition in VBox on my computer (with 512MB RAM assigned to the VM), MongooseIM occupies ~50% of memory right after start.

Maybe you could try with more powerful VM?

kzemek commented 6 years ago

Just to chip in, C++ compiler being killed is usually a sign of lack of memory, too. Of course, Idna error points in the same direction. Plus the operating systems in use today will dramatically slow down when very low on memory, which would explain why compilation would take much time.

You can sidestep the issue by creating a swap file; the compilation will be slow but should be successful if memory is indeed the problem.

Arslan007 commented 6 years ago

Thanks @fenek If I am understanding correctly you are suggesting me to increase memory in order to run MIM although I am already successfully compiled MIM.

@kzemek Thanks for your insightful chip in. I would love to know how I can create a swapfile and sidestep this issue.

fenek commented 6 years ago

Yes. Maybe Ubuntu image for Azure:

In order to check that, can you please paste the output of cat /proc/meminfo?

Arslan007 commented 6 years ago

Thanks @fenek

Here is results of cat /proc/meminfo

MemTotal:         683176 kB
MemFree:           94808 kB
MemAvailable:     199644 kB
Buffers:           30360 kB
Cached:           166892 kB
SwapCached:            0 kB
Active:           382116 kB
Inactive:          70512 kB
Active(anon):     259824 kB
Inactive(anon):    14532 kB
Active(file):     122292 kB
Inactive(file):    55980 kB
Unevictable:        3660 kB
Mlocked:            3660 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               104 kB
Writeback:             0 kB
AnonPages:        259056 kB
Mapped:            39172 kB
Shmem:             16556 kB
Slab:              62380 kB
SReclaimable:      35388 kB
SUnreclaim:        26992 kB
KernelStack:        4304 kB
PageTables:         6368 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      341588 kB
Committed_AS:    1249600 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    161792 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       65472 kB
DirectMap2M:      720896 kB

I did experiment by increasing size 512 MB to 2 GB on my Azure VM and it works like charm so conclusion is MIM requires minimum 1-2 GB Memory in order to run.

fenek commented 6 years ago

Are these memory stats from a VM where MongooseIM is running? Or is it just a bare OS?

Arslan007 commented 6 years ago

@fenek It's with Azure 512 MB Machine. MIM is compiled successfully but never runs starts on this machine with this specs.

Here is the machine with 2GB Ram and MIM is running on it

MemTotal:        2048204 kB
MemFree:          334208 kB
MemAvailable:    1134552 kB
Buffers:          119952 kB
Cached:           789388 kB
SwapCached:            0 kB
Active:          1149304 kB
Inactive:         410280 kB
Active(anon):     653232 kB
Inactive(anon):    27236 kB
Active(file):     496072 kB
Inactive(file):   383044 kB
Unevictable:        3660 kB
Mlocked:            3660 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                36 kB
Writeback:             0 kB
AnonPages:        653908 kB
Mapped:            75264 kB
Shmem:             27804 kB
Slab:             126972 kB
SReclaimable:     105960 kB
SUnreclaim:        21012 kB
KernelStack:        3936 kB
PageTables:         6008 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1024100 kB
Committed_AS:    1886628 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    583680 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       40940 kB
DirectMap2M:     2056192 kB
DirectMap1G:           0 kB
fenek commented 6 years ago

OK, I think it's clear now. In fact, in order to start, MongooseIM requires ~300MB but it leaves no space for actual users' sessions and message processing. The VM with 512MB reports MemAvailable: 199644 kB, which is too low. These Ubuntu images actually seem to consume quite a lot of RAM for OS, since when I've started Ubuntu server edition in VirtualBox, the OS itself occupied only ~50MB.

MongooseIM clusters are memory-intensive, so it's a good idea to use VM with 2GB RAM as a development machine.

Arslan007 commented 6 years ago

@fenek Thanks.

Last question what is suggested specs for the development machine and production machine.

P.S: You can close the ticket as well

fenek commented 6 years ago

For development, 2GB RAM and 1 or 2-core CPU should be perfectly enough. No particular requirements for disk space. Logs are rotated and have size limit.

For production machines, it very much depends on your predicted traffic and what features are enabled. The best way to learn your perfect setup is to create a cluster with e.g. 2 nodes with 4 cores each, 8GB RAM and see if it's not enough, way to much or ideal.