Pluto in libreswan segfault

Ralim commented 4 years ago

Hey,

I'm trying to use the new IPSec support from libreswan. When starting an IPSec connection to a server setup using instructions from here, pluto is exiting with a segfault.

Is there any way to debug this easily? (Happy to spend some time trying to debug here). Would rather not have to install the entire os-clr-on-clr package as its rather large for my small SSD. Is there a way to get debug symbols for this package to make this more readable ?

Nov 01 10:28:46 Bespoke kernel: pluto[5091]: segfault at fffffffffffffff0 ip 000055e67cc7d417 sp 00007fffb6184640 error 5 cpu 0 in pluto[55e67cc3b000+104000]
Nov 01 10:28:46 Bespoke kernel: Code: 0f 1f 00 48 ba 12 01 2f 25 ff ff ff ff 55 48 89 e5 41 54 53 48 89 fb 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 <48> 8b 47 f0 48 39 d0 0f 84 ea 00 00 00 ba ed fe d0 da 48 39 d0 0f

Nov 01 10:28:54 Bespoke systemd-coredump[4169]: Process 4163 (pluto) of user 0 dumped core.

                                                Stack trace of thread 4163:
                                                #0  0x00005629ec4ca417 n/a (pluto)
                                                #1  0x00005629ec55e82a n/a (pluto)
                                                #2  0x00005629ec4b08a1 n/a (pluto)
                                                #3  0x00005629ec48a27f n/a (pluto)
                                                #4  0x00007f01d90612c3 __libc_start_main (libc.so.6)
                                                #5  0x00005629ec48cb1e n/a (pluto)
Nov 01 10:28:54 Bespoke systemd[1]: systemd-coredump@5-4166-0.service: Succeeded.

OS Version: 31460

Happy to include any log files if you can point me as to where to look for them?

ahkok commented 4 years ago

gdb will try and fetch the right debug files over the network. You may have to run it a few times. You can seed the cache by running gdb /usr/bin/pluto. Once you get all the debugging options loaded, you will see more detailed stack traces.

Also note that turning on telemetry will make it show up at our telemetry server and we can likely find these crashes. We check for crash reports on a weekly basis and if there are many, we will automatically treat them as bugs.

You can also use coredumpctl to examine existing cores (see man coredumptctl).

Please give this a try and see what you can find out - I can't test pluto myself since I have no ipsec testing setup.

Ralim commented 4 years ago

@ahkok I've tried running gdb on pluto (gdb /usr/libexec/ipsec/pluto) But so far it hasn't loaded the debugging symbols. Is there anywhere i can just download these from ? (i'll keep trying later tonight too).

I definitely have telemetry on (according to the installer), as auto submission of faults is a good thing in my book. (For exactly this reason).

Coredump currently doesn't give me a huge amount extra:

sudo coredumpctl dump
Password: 
           PID: 5091 (pluto)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Fri 2019-11-01 10:28:46 AEDT (5h 19min ago)
  Command Line: /usr/libexec/ipsec/pluto --leak-detective --config /etc/ipsec.conf --nofork
    Executable: /usr/libexec/ipsec/pluto
 Control Group: /system.slice/ipsec.service
          Unit: ipsec.service
         Slice: system.slice
       Boot ID: a2300c34fbd54f4abb1d35f13960e4b9
    Machine ID: 57375bebc5f04413988f4d478a1fb821
      Hostname: Bespoke
       Storage: /var/lib/systemd/coredump/core.pluto.0.a2300c34fbd54f4abb1d35f13960e4b9.5091.1572564526000000000000.lz4
       Message: Process 5091 (pluto) of user 0 dumped core.

                Stack trace of thread 5091:
                #0  0x000055e67cc7d417 n/a (pluto)
                #1  0x000055e67cd1182a n/a (pluto)
                #2  0x000055e67cc638a1 n/a (pluto)
                #3  0x000055e67cc3d27f n/a (pluto)
                #4  0x00007ff6c72282c3 __libc_start_main (libc.so.6)
                #5  0x000055e67cc3fb1e n/a (pluto)

I have attached the --output from coredumpctl.

Happy to email you a testing ipsec server you can use if this would be of assistance.

core.zip

bryteise commented 4 years ago

The debug info for 31460 can be found here (you can substitute any release number to that that releases debuginfo). My memory on how to use this is basically to extract the packages debuginfo with rpm2cpio and then place it in the expected path which I'm fuzzy on. @phmccarty probably remembers this better.

phmccarty commented 4 years ago

@bryteise I don't see pluto in Clear, so there won't be any debuginfo available in the repo that will help diagnose this particular issue.

In general though, for packages in the distro, the easiest way to manually install their debuginfo is via the common tooling. For example, for the current m4 package's debuginfo:

curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh
bash user-setup.sh
cd clearlinux
make clone_m4
cd packages/m4
curl -o debug.rpm https://cdn.download.clearlinux.org/releases/current/clear/x86_64/debug/m4-debuginfo-1.4.18-91.x86_64.rpm
make install-debuginfo-local DEBUGINFO_RPM=debug.rpm

ahkok commented 4 years ago

@phmccarty it's in libreswan.

phmccarty commented 4 years ago

Ah, sorry, my repoquery was wrong...

Then you can follow the same steps but with the latest libreswan debuginfo rpm instead. The latest rpm lives here.

Ralim commented 4 years ago

Thank you :) @ahkok I now have the symbols loaded, this is the stack trace that I'm seeing:

Stack trace of thread 35824:
#0  0x000055fe454df417 pfree (pluto)
#1  0x000055fe4557382a pfree (pluto)
#2  0x000055fe454c58a1 crypt_prf_update_symkey (pluto)
#3  0x000055fe4549f27f main (pluto)
#4  0x00007f9f3ab1c2c3 __libc_start_main (libc.so.6)
#5  0x000055fe454a1b1e _start (pluto)

Not sure if this is a clear-linux specific thing or if this should be raised upstream at this point.

I can cause this to occur on my machine by trying to start an IPSec connection to a fake server, using bogus values such as :

Settings -> Network -> Create L2TP connection

Server 8.8.8.8
User : admin
password : admin
ipsec preferences:
Turn on ipsec
Set PSK as admin

vatula commented 4 years ago

I'd like to report the same issue. ipsec fails to start. When i do systemctl start ipsec the service will fail. Checking the logs with journalctl -xe shows

-- Subject: A start job for unit ipsec.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit ipsec.service has begun execution.
-- 
-- The job identifier is 10673.
Mar 19 13:42:49 miniattic clr_debug_daemon[20675]: ./.build-id/dc/
Mar 19 13:42:49 miniattic kernel: calling  padlock_init+0x0/0x1000 [padlock_aes] @ 20682
Mar 19 13:42:49 miniattic kernel: initcall padlock_init+0x0/0x1000 [padlock_aes] returned -19 after 0 usecs
Mar 19 13:42:49 miniattic kernel: calling  padlock_init+0x0/0x1000 [padlock_sha] @ 20684
Mar 19 13:42:49 miniattic kernel: initcall padlock_init+0x0/0x1000 [padlock_sha] returned -19 after 0 usecs
Mar 19 13:42:49 miniattic kernel: calling  padlock_init+0x0/0x1000 [padlock_aes] @ 20703
Mar 19 13:42:49 miniattic kernel: initcall padlock_init+0x0/0x1000 [padlock_aes] returned -19 after 0 usecs
Mar 19 13:42:49 miniattic ipsec[20846]: nflog ipsec capture disabled
Mar 19 13:42:49 miniattic pluto[20857]: NSS DB directory: sql:/etc/ipsec.d
Mar 19 13:42:49 miniattic pluto[20857]: Initializing NSS
Mar 19 13:42:49 miniattic pluto[20857]: Opening NSS database "sql:/etc/ipsec.d" read-only
Mar 19 13:42:49 miniattic pluto[20857]: NSS initialized
Mar 19 13:42:49 miniattic pluto[20857]: NSS crypto library initialized
Mar 19 13:42:49 miniattic pluto[20857]: FIPS HMAC integrity support [disabled]
Mar 19 13:42:49 miniattic pluto[20857]: libcap-ng support [enabled]
Mar 19 13:42:49 miniattic pluto[20857]: Linux audit support [disabled]
Mar 19 13:42:49 miniattic pluto[20857]: Starting Pluto (Libreswan Version 3.31 XFRM(netkey) XFRMI esp-hw-offload FORK PTHREAD_SETSCHEDPRIO GCC_EXCEPTIONS NSS (IPsec profile) (native-PRF) SYSTEMD>
Mar 19 13:42:49 miniattic pluto[20857]: core dump dir: /run/pluto
Mar 19 13:42:49 miniattic pluto[20857]: secrets file: /etc/ipsec.secrets
Mar 19 13:42:49 miniattic pluto[20857]: leak-detective enabled
Mar 19 13:42:49 miniattic pluto[20857]: NSS crypto [enabled]
Mar 19 13:42:49 miniattic pluto[20857]: XAUTH PAM support [enabled]
Mar 19 13:42:49 miniattic pluto[20857]: Initializing libevent in pthreads mode: headers: 2.1.11-stable (2010b00); library: 2.1.11-stable (2010b00)
Mar 19 13:42:49 miniattic pluto[20857]: NAT-Traversal support  [enabled]
Mar 19 13:42:49 miniattic pluto[20857]: Encryption algorithms:
Mar 19 13:42:49 miniattic pluto[20857]:   AES_CCM_16              IKEv1:     ESP     IKEv2:     ESP     FIPS  {256,192,*128}  aes_ccm, aes_ccm_c
Mar 19 13:42:49 miniattic pluto[20857]:   AES_CCM_12              IKEv1:     ESP     IKEv2:     ESP     FIPS  {256,192,*128}  aes_ccm_b
Mar 19 13:42:49 miniattic pluto[20857]:   AES_CCM_8               IKEv1:     ESP     IKEv2:     ESP     FIPS  {256,192,*128}  aes_ccm_a
Mar 19 13:42:49 miniattic pluto[20857]:   3DES_CBC                IKEv1: IKE ESP     IKEv2: IKE ESP     FIPS  [*192]  3des
Mar 19 13:42:49 miniattic pluto[20857]:   CAMELLIA_CTR            IKEv1:     ESP     IKEv2:     ESP           {256,192,*128}
Mar 19 13:42:49 miniattic pluto[20857]:   CAMELLIA_CBC            IKEv1: IKE ESP     IKEv2: IKE ESP           {256,192,*128}  camellia
Mar 19 13:42:49 miniattic pluto[20857]:   AES_GCM_16              IKEv1:     ESP     IKEv2: IKE ESP     FIPS  {256,192,*128}  aes_gcm, aes_gcm_c
Mar 19 13:42:49 miniattic pluto[20857]:   AES_GCM_12              IKEv1:     ESP     IKEv2: IKE ESP     FIPS  {256,192,*128}  aes_gcm_b
Mar 19 13:42:49 miniattic pluto[20857]:   AES_GCM_8               IKEv1:     ESP     IKEv2: IKE ESP     FIPS  {256,192,*128}  aes_gcm_a
Mar 19 13:42:49 miniattic pluto[20857]:   AES_CTR                 IKEv1: IKE ESP     IKEv2: IKE ESP     FIPS  {256,192,*128}  aesctr
Mar 19 13:42:49 miniattic pluto[20857]:   AES_CBC                 IKEv1: IKE ESP     IKEv2: IKE ESP     FIPS  {256,192,*128}  aes
Mar 19 13:42:49 miniattic pluto[20857]:   SERPENT_CBC             IKEv1: IKE ESP     IKEv2: IKE ESP           {256,192,*128}  serpent
Mar 19 13:42:49 miniattic pluto[20857]:   TWOFISH_CBC             IKEv1: IKE ESP     IKEv2: IKE ESP           {256,192,*128}  twofish
Mar 19 13:42:49 miniattic pluto[20857]:   TWOFISH_SSH             IKEv1: IKE         IKEv2: IKE ESP           {256,192,*128}  twofish_cbc_ssh
Mar 19 13:42:49 miniattic pluto[20857]:   NULL_AUTH_AES_GMAC      IKEv1:     ESP     IKEv2:     ESP     FIPS  {256,192,*128}  aes_gmac
Mar 19 13:42:49 miniattic pluto[20857]:   NULL                    IKEv1:     ESP     IKEv2:     ESP           []
Mar 19 13:42:49 miniattic pluto[20857]:   CHACHA20_POLY1305       IKEv1:             IKEv2: IKE ESP           [*256]  chacha20poly1305
Mar 19 13:42:49 miniattic pluto[20857]: Hash algorithms:
Mar 19 13:42:49 miniattic pluto[20857]:   MD5                     IKEv1: IKE         IKEv2:
Mar 19 13:42:49 miniattic pluto[20857]:   SHA1                    IKEv1: IKE         IKEv2:             FIPS  sha
Mar 19 13:42:49 miniattic pluto[20857]:   SHA2_256                IKEv1: IKE         IKEv2:             FIPS  sha2, sha256
Mar 19 13:42:49 miniattic pluto[20857]:   SHA2_384                IKEv1: IKE         IKEv2:             FIPS  sha384
Mar 19 13:42:49 miniattic pluto[20857]:   SHA2_512                IKEv1: IKE         IKEv2:             FIPS  sha512
Mar 19 13:42:49 miniattic pluto[20857]: PRF algorithms:
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_MD5                IKEv1: IKE         IKEv2: IKE               md5
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA1               IKEv1: IKE         IKEv2: IKE         FIPS  sha, sha1
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA2_256           IKEv1: IKE         IKEv2: IKE         FIPS  sha2, sha256, sha2_256
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA2_384           IKEv1: IKE         IKEv2: IKE         FIPS  sha384, sha2_384
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA2_512           IKEv1: IKE         IKEv2: IKE         FIPS  sha512, sha2_512
Mar 19 13:42:49 miniattic pluto[20857]:   AES_XCBC                IKEv1:             IKEv2: IKE               aes128_xcbc
Mar 19 13:42:49 miniattic pluto[20857]: Integrity algorithms:
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_MD5_96             IKEv1: IKE ESP AH  IKEv2: IKE ESP AH        md5, hmac_md5
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA1_96            IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  sha, sha1, sha1_96, hmac_sha1
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA2_512_256       IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  sha512, sha2_512, sha2_512_256, hmac_sha2_512
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA2_384_192       IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  sha384, sha2_384, sha2_384_192, hmac_sha2_384
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA2_256_128       IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  sha2, sha256, sha2_256, sha2_256_128, hmac_sha2_256
Mar 19 13:42:49 miniattic pluto[20857]:   HMAC_SHA2_256_TRUNCBUG  IKEv1:     ESP AH  IKEv2:         AH
Mar 19 13:42:49 miniattic pluto[20857]:   AES_XCBC_96             IKEv1:     ESP AH  IKEv2: IKE ESP AH        aes_xcbc, aes128_xcbc, aes128_xcbc_96
Mar 19 13:42:49 miniattic pluto[20857]:   AES_CMAC_96             IKEv1:     ESP AH  IKEv2:     ESP AH  FIPS  aes_cmac
Mar 19 13:42:49 miniattic pluto[20857]:   NONE                    IKEv1:     ESP     IKEv2: IKE ESP     FIPS  null
Mar 19 13:42:49 miniattic pluto[20857]: DH algorithms:
Mar 19 13:42:49 miniattic pluto[20857]:   NONE                    IKEv1:             IKEv2: IKE ESP AH  FIPS  null, dh0
Mar 19 13:42:49 miniattic pluto[20857]:   MODP1536                IKEv1: IKE ESP AH  IKEv2: IKE ESP AH        dh5
Mar 19 13:42:49 miniattic pluto[20857]:   MODP2048                IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  dh14
Mar 19 13:42:49 miniattic pluto[20857]:   MODP3072                IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  dh15
Mar 19 13:42:49 miniattic pluto[20857]:   MODP4096                IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  dh16
Mar 19 13:42:49 miniattic pluto[20857]:   MODP6144                IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  dh17
Mar 19 13:42:49 miniattic pluto[20857]:   MODP8192                IKEv1: IKE ESP AH  IKEv2: IKE ESP AH  FIPS  dh18
Mar 19 13:42:49 miniattic pluto[20857]:   DH19                    IKEv1: IKE         IKEv2: IKE ESP AH  FIPS  ecp_256, ecp256
Mar 19 13:42:49 miniattic pluto[20857]:   DH20                    IKEv1: IKE         IKEv2: IKE ESP AH  FIPS  ecp_384, ecp384
Mar 19 13:42:49 miniattic pluto[20857]:   DH21                    IKEv1: IKE         IKEv2: IKE ESP AH  FIPS  ecp_521, ecp521
Mar 19 13:42:49 miniattic pluto[20857]:   DH31                    IKEv1: IKE         IKEv2: IKE ESP AH        curve25519
Mar 19 13:42:49 miniattic pluto[20857]: testing CAMELLIA_CBC:
Mar 19 13:42:49 miniattic pluto[20857]:   Camellia: 16 bytes with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Camellia: 16 bytes with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Camellia: 16 bytes with 256-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Camellia: 16 bytes with 256-bit key
Mar 19 13:42:49 miniattic pluto[20857]: testing AES_GCM_16:
Mar 19 13:42:49 miniattic pluto[20857]:   empty string
Mar 19 13:42:49 miniattic pluto[20857]:   one block
Mar 19 13:42:49 miniattic pluto[20857]:   two blocks
Mar 19 13:42:49 miniattic pluto[20857]:   two blocks with associated data
Mar 19 13:42:49 miniattic pluto[20857]: testing AES_CTR:
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 16 octets using AES-CTR with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 32 octets using AES-CTR with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 36 octets using AES-CTR with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 16 octets using AES-CTR with 192-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 32 octets using AES-CTR with 192-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 36 octets using AES-CTR with 192-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 16 octets using AES-CTR with 256-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 32 octets using AES-CTR with 256-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 36 octets using AES-CTR with 256-bit key
Mar 19 13:42:49 miniattic pluto[20857]: testing AES_CBC:
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 16 octets using AES-CTR with 256-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 32 octets using AES-CTR with 256-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 36 octets using AES-CTR with 256-bit key
Mar 19 13:42:49 miniattic pluto[20857]: testing AES_CBC:
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 16 bytes (1 block) using AES-CBC with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 32 bytes (2 blocks) using AES-CBC with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 48 bytes (3 blocks) using AES-CBC with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]:   Encrypting 64 bytes (4 blocks) using AES-CBC with 128-bit key
Mar 19 13:42:49 miniattic pluto[20857]: testing AES_XCBC:
Mar 19 13:42:49 miniattic pluto[20857]:   RFC 3566 Test Case 1: AES-XCBC-MAC-96 with 0-byte input
Mar 19 13:42:49 miniattic pluto[20857]: ABORT: ASSERTION FAILED: ptr != NULL (in pfree() at alloc.c:145)
Mar 19 13:42:49 miniattic systemd[1]: Started Process Core Dump (PID 20860/UID 0).
-- Subject: A start job for unit systemd-coredump@9-20860-0.service has finished successfully
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Ralim commented 4 years ago

I've never resolved this yet either. Only occurs on clear-linux out of all of my dev machines :(

dkosovic commented 4 years ago

As mentioned in issue https://github.com/clearlinux/distribution/issues/1830 , there are two main libreswan segfault reasons:

/usr/bin/certutil from the cryptography bundle is required, otherwise libreswan will segfault as the NSS database isn't initialized (even if you are not using certificates), a segfault might also occur if pluto tries to update the NSS database in /etc/ipsec.d/. For the time being, you need to explicitly install the cryptography bundle for /usr/bin/certutil :
```
sudo swupd bundle-add cryptography
```
The new libreswan-3.32 package from issue https://github.com/clearlinux/distribution/issues/1830 uses /run/pluto/ for the NSS database, but it still needs /usr/bin/certutil from the cryptography bundle.
libreswan <= 3.32 will segfault when built with NSS >= 3.52, an upstream patch is required which the new libreswan-3.32 package from issue https://github.com/clearlinux/distribution/issues/1830 has.

topcat001 commented 4 years ago

Have you tried removing --leak-detective from pluto options in: /usr/lib/systemd/system/ipsec.service ? With this change (and other config) I am successfully using L2TP VPN with AES 256 encryption.

clearlinux / distribution

Pluto in libreswan segfault #1435

Settings -> Network -> Create L2TP connection

ipsec preferences: