problem with libetpan/libsasl builds which link against gnutls

[Ampli-fier]

• Delta Chat Core Version: 0.39.1

• Expected behavior: meson build works

• Actual behavior: meson reports missing header: Has header "CommonCrypto/CommonDigest.h" : NO Has header "direct.h" : NO

• Steps to reproduce the problem:

I followed the current version of the README to build Core in Linux.

When building with meson, I get two missing headers reported (see above). The build process did not interrupt but stated: Subproject netpgp finished

When trying to use delta CLI, I get this error: Trying: <user>@<domain>.de <user>@<domain>.de:***:imap.strato.de:993 <user>@<domain>.de:***:smtp.strato.de:465 AUTH_NORMAL IMAP_SSL SMTP_SSL GnuTLS error: The operation timed out [DC_EVENT_ERROR_NETWORK] first=1, msg=Could not connect to IMAP-server imap.strato.de:993 using SSL. (Error #43)

Browsing through the Issue list made me wonder if #222 is related to my problem.

• Screenshots: NA
• Logs:

$ meson
The Meson build system
Version: 0.49.1
Source dir: /home/dc/DeltaChat/Core/deltachat-core
Build dir: /home/dc/DeltaChat/Core/deltachat-core/builddir
Build type: native build
Project name: deltachat-core
Project version: 0.0.0
Native C compiler: cc (gcc 7.3.0 "cc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0")
Build machine cpu family: x86_64
Build machine cpu: x86_64
Program git found: YES (/usr/bin/git)
Dependency threads found: YES 
Library m found: YES
Found pkg-config: /usr/bin/pkg-config (0.29.1)
Dependency zlib found: YES 1.2.11
Dependency openssl found: YES 1.1.0g
Dependency libsasl2 found: YES 2.1.27
Dependency sqlite3 found: YES 3.22.0
Program libetpan-config found: YES (/usr/bin/libetpan-config)
Message: Dependency libetpan found: 1.8.0. Checking for compatibility.
Message: Check done. Installed libetpan version is compatible.

|
|Executing subproject netpgp 
|
|Project name: netpgp
|Project version: 20140220
|Native C compiler: cc (gcc 7.3.0 "cc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0")
|Dependency threads found: YES (cached)
|Dependency zlib found: YES (cached)
|Dependency openssl found: YES (cached)
|Library bz2 skipped: feature netpgp:bzip2 disabled
|Has header "zlib.h" with dependency zlib: YES
|Has header "CommonCrypto/CommonDigest.h" : NO
|Has header "direct.h" : NO
|Has header "errno.h" : YES
|Has header "fcntl.h" : YES
|Has header "inttypes.h" : YES
|Has header "limits.h" : YES
|Has header "stdint.h" : YES
|Has header "sys/cdefs.h" : YES
|Has header "sys/mman.h" : YES
|Has header "sys/param.h" : YES
|Has header "termios.h" : YES
|Has header "unistd.h" : YES
|Has header "openssl/aes.h" with dependency openssl: YES
|Has header "openssl/bn.h" with dependency openssl: YES
|Has header "openssl/camellia.h" with dependency openssl: YES
|Has header "openssl/cast.h" with dependency openssl: YES
|Has header "openssl/des.h" with dependency openssl: YES
|Has header "openssl/dsa.h" with dependency openssl: YES
|Has header "openssl/err.h" with dependency openssl: YES
|Has header "openssl/idea.h" with dependency openssl: YES
|Has header "openssl/md5.h" with dependency openssl: YES
|Has header "openssl/rand.h" with dependency openssl: YES
|Has header "openssl/rsa.h" with dependency openssl: YES
|Has header "openssl/sha.h" with dependency openssl: YES
|Configuring config-netpgp-meson.h using configuration
|Compiler for C supports arguments -Wno-misleading-indentation -Wmisleading-indentation: YES
|Build targets in project: 1
|
|Subproject netpgp finished.
Build targets in project: 3
Found ninja-1.8.2 at /usr/bin/ninja
$

[flub]

The netpgp bit is a red herring I think, not having CommonDigest.h and direct.h is expected on modern linux platforms. netpgp supports building on a variety of platforms of varying age and it tries to detect which of these variations it should build with. If this were the problem then indeed the build should fail.

This is most likely an issue with openssl, cyrus-sasl or etpan. Going by version numbers my first guess would be libetpan. So try building with meson -Dforce_etpan_fallback and see if that improves things. Alternatively install a recent version of libetpan from source and make sure it is detected (by putting it's libetpan-config command earlier on PATH) instead of the currently detected version.

[Ampli-fier]

Thanks for pointing me to right direction.

The relevant part/log from my attempt was not provided by me,I interpreted the missing header for netpg as crucial, but it is normal in Linux (as I understand). I tried to use force-etpan-fallback as you suggested, but ran into another problem/error. Instead I found another solution:

In fact, this was the reason the compilation failed on my host (from the failed compilation log):

Program libetpan-config found: YES (/usr/bin/libetpan-config)
Message: Dependency libetpan found: 1.8.0. Checking for compatibility.
Message: Check done. Installed libetpan version is compatible.

The package version I had available was:

libetpan-dev is already the newest version (1.8.0-1).

After purging the package from my system, it all went fine (etpan provided by Core is used - which must differ from the official package):

|Executing subproject libetpan 
|
|Project name: etpan
|Project version: 1.8
|Native C compiler: cc (gcc 7.3.0 "cc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0")
|Dependency threads found: YES (cached)
|Dependency zlib from subproject libs/zlib found: YES (cached)
|Dependency openssl from subproject libs/openssl found: YES (cached)
|Dependency sqlite3 from subproject libs/sqlite found: YES (cached)
|Dependency libsasl2 from subproject libs/cyrussasl found: YES (cached)
|Compiler for C supports arguments -Wno-format-truncation -Wformat-truncation: YES
|Compiler for C supports arguments -Wno-misleading-indentation -Wmisleading-indentation: YES
|Compiler for C supports arguments -Wno-sometimes-uninitialized -Wsometimes-uninitialized: NO
|Compiler for C supports arguments -Wno-unused-function -Wunused-function: YES
|Build targets in project: 5
|
|Subproject libetpan finished.

I can provide a PR for the README with the important details later.

I wonder if there would be another solution than purging, e.g. setting a flag for meson or using another parameter for the meson command.

I always had to use these parameter for meson to work:

meson --wrap-mode=forcefallback --default-library=static

which is not quite clear when following the README.

To follow the installing c-level dependencies seems not to be mandatory at the moment, but can be the reason for compilation errors.

@flub @r10s Can I ask for your opinion/expert knowledge? :)

[Ampli-fier]

Must be related to this PR: #311 Is there any libetpan version that can work beside the bundled one?

[flub]

I'm a little lost following your latest comments. Could you provide a full log of invoking meson and ninja -v? As for the libetpan version, the build system does check the version (1.8) and we've had success using the ubuntu 18.04 one before so I find it a bit unexpected that you failed to use it.

[Ampli-fier]

Here we go, just ask if any other log/info is helpful.

Shell log from fresh attempt. meson-log.txt from builddir/meson-logs.

[flub]

This is great, thanks! From here -and a compiling/linking perspective- this all looks fine, I learned that debian/ubuntu ship with libetpan linked against gnutls rather than openssl though. Maybe that is relevant.

Your shell log shows you failed to connect to the imap server, if I understand the previous comments of this issue correctly you did manage to build a version of libdeltachat.so which did connect to your imap server. Would you mind re-doing this as well and showing the same logs?

[flub]

Could you also show the output of gnutls-cli --list?

I've

[flub]

Here's the list of cipher suites I think your imap server supports: imap.strato.de-stuites.txt

[Ampli-fier]

Here is the gnutls-cli --list printout from my Ubuntu 18.04 Installation.

[Ampli-fier]

This Gist shows how Core was built in a way that IMAP server could be contacted successfully.

I re-used the folder from the previous execution.

In earlier attempts I thought I could achieve the same by using

meson --reconfigure --wrap-mode=forcefallback

to have all bundled versions used. This is shown on top of the log. Just, it does not work for libetpan ...

What works is this meson call:

meson --reconfigure --default-library=static -Dforce-etpan-fallback=true

We can recommend this on the README ... would be good to know under which conditions.

[flub]

Currently my best bet is that your system's libgnutls does not support the cipher suites your imap server has, but that your system's libssl does. Could you paste the output of: gnutls-cli --list and also openssl ciphers? Thanks!

[Ampli-fier]

Here is the openssl ciphers printout. Here is the gnutls-cli --list printout.

There is a notable difference in the format of the ciphers names from the gnu-command:

For example, this cipher:

TLS_DHE_RSA_AES_128_GCM_SHA256

would match, if it would be named

DHE-RSA-AES128-GCM-SHA256

I took your imap.strato.de-stuites.txt as basis for an evaluation (full shell log is here).

With openssl there are 18 matches:

AES128-GCM-SHA256 AES128-SHA AES128-SHA256 AES256-GCM-SHA384 AES256-SHA AES256-SHA256 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-SHA DHE-RSA-AES128-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-SHA DHE-RSA-AES256-SHA256 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA ECDHE-RSA-AES256-SHA384

With gnutls there is no match. Just when modifying the name pattern there are these potential matches:

DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384

[Ampli-fier]

In this block of meson.build the run_command(libetpan_config, ['--libs']) could be evaluated.

For my installation this returns:

dc@dc-VirtualBox:~$ /usr/bin/libetpan-config --libs
-L/usr/lib/x86_64-linux-gnu -letpan -Wl,-Bsymbolic-functions -specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now -lz -lgnutls -ldb-5.3 -lz -llockfile -pthread -lsasl2
dc@dc-VirtualBox:~$ 

If gnutls is used/present, the local libetpan version is not compatible ... most ... likely

[flub]

Great analysis, thanks! Guess we found out how your system has been behaving (summarising here for ppl catching up so they don't have to read the whole thread): the crypto library used (gnutls) did not support the ciphers required by your imap server. When building against openssl the cipher was supported.

I think there's two possible issues here:

1. Can libdeltachat give a better error in this case? /cc @r10s for that one (if so maybe deserves a separate issue)
2. Can we do something at build time about this?

I'm not sure about your suggested solution for 2 though, you're effectively completely discarding using gnutls which, to me at least, seems like not something we should be deciding. I'd be more slightly more happy if we had a list of required and/or recommended cipher suites which we'd test against (and maybe provide switches to ignore these checks if you want). But I'm not sure how to come up with sensible lists for these though. /cc @dignifiedquire who may have some thoughts on ciphers or this approach.

[flub]

@Ampli-fier could you also try echo -n | gnutls-cli -p 993 imap.strato.de and post the output btw?

[Ampli-fier]

This works ... the output is:

dc@dc-VirtualBox:~/DeltaChat/Core/issue/deltachat-core$ echo -n | gnutls-cli -p 993 imap.strato.de
Processed 133 CA certificate(s).
Resolving 'imap.strato.de:993'...
Connecting to '81.169.145.103:993'...
- Certificate type: X.509
- Got a certificate list of 2 certificates.
- Certificate[0] info:
 - subject `CN=imap.strato.de,L=Berlin,ST=Berlin,OU=Rechenzentrum,O=Strato AG,C=DE', issuer `CN=TeleSec ServerPass Class 2 CA,street=Untere Industriestr. 20,L=Netphen,postalCode=57250,ST=Nordrhein Westfalen,OU=T-Systems Trust Center,O=T-Systems International GmbH,C=DE', serial 0x0bf893ccd89d00feeac8e58864ba25ad, RSA key 2048 bits, signed using RSA-SHA256, activated `2019-04-02 09:06:16 UTC', expires `2020-04-07 23:59:59 UTC', pin-sha256="ONz8DLMhPovk1Tys90+Ok8oY1fZKRLG0BW1L7awQbjo="
    Public Key ID:
        sha1:e7993d5fe7527939bdda17a3269ac2e69b0b0788
        sha256:38dcfc0cb3213e8be4d53cacf74f8e93ca18d5f64a44b1b4056d4bedac106e3a
    Public Key PIN:
        pin-sha256:ONz8DLMhPovk1Tys90+Ok8oY1fZKRLG0BW1L7awQbjo=
    Public key's random art:
        +--[ RSA 2048]----+
        |                 |
        |                 |
        |                 |
        |  . .            |
        | E . .  S .     +|
        |      .  o +   *+|
        |     ...  + o ..O|
        |      o+. .. =o+o|
        |      o=+o. o.ooo|
        +-----------------+

- Certificate[1] info:
 - subject `CN=TeleSec ServerPass Class 2 CA,street=Untere Industriestr. 20,L=Netphen,postalCode=57250,ST=Nordrhein Westfalen,OU=T-Systems Trust Center,O=T-Systems International GmbH,C=DE', issuer `CN=T-TeleSec GlobalRoot Class 2,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE', serial 0x7e39c7ad1dd9f043, RSA key 2048 bits, signed using RSA-SHA256, activated `2014-02-11 14:39:10 UTC', expires `2024-02-11 23:59:59 UTC', pin-sha256="c3jf+L8VIAFQnJJDM6Mfb4MtI1JnhVS8JwZHMwJj28M="
- Status: The certificate is trusted. 
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-256-GCM)
- Session ID: (empty)
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA512
- Cipher: AES-256-GCM
- MAC: AEAD
- Compression: NULL
- Options: safe renegotiation,
- Handshake was completed

- Simple Client Mode:

* OK [CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=LOGIN AUTH=CRAM-MD5 AUTH=DIGEST-MD5 CHILDREN ENABLE I18NLEVEL=2 ID IDLE MOVE MULTIAPPEND NAMESPACE QUOTA SORT STATUS=SIZE UIDPLUS UNSELECT WITHIN XLIST] IMAP server ready (P1 TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384)
dc@dc-VirtualBox:~/DeltaChat/Core/issue/deltachat-core$ 

[flub]

Oh, well I guess our earlier conclusion won't do then and we'll have to figure out some more debugging. :(

[lupine]

@flub @Ampli-fier I think this issue was caused by the bug fixed in https://github.com/dinhviethoa/libetpan/pull/330 - it should be safe to just close it now.