Core dump with locally built librtmp

jairoxyz commented 5 years ago

Dear all, I know this is not really an issue of inputstream.rtmp 2.0.4 but I have tried to find an explanation for days now and don't know where else to go. This issue occurred after OpenSSL 1.1 patches were introduced in version 2.0.4. It does not happen in version 2.0.3 with OpenSSL 1.0.2.

Problem: on Ubuntu 18.0.4 - I have build librtmp sources in ~/inputstream.rtmp/build/build/librtmp/src/librtmp after cmake applied SSL 1.1 patches. Build is successful and rtmpdump binary works fine against /usr/lib/x86_64-linux-gnu/librtmp.so.1. for rtmp and rtmpe.

But when I try to run rtmpdump on a rtmpe host against the locally built ~/inputstream.rtmp/build/build/librtmp/src/librtmp/librtmp/librtmp.so.1 it crashes with core dump:

DEBUG: Protocol : RTMPE
DEBUG: Hostname : 93.189.62.10
DEBUG: Port     : 1935
DEBUG: Playpath : raw:1917163
DEBUG: tcUrl    : rtmpe://93.189.62.10:1935/xlive?vi=1917163&
DEBUG: app      : xlive?vi=1917163&
DEBUG: live     : no
DEBUG: timeout  : 30 sec
DEBUG: Setting buffer time to: 36000000ms
Connecting ...
DEBUG: RTMP_Connect1, ... connected, handshaking
DEBUG: HandShake: Client type: 06
DEBUG: HandShake: DH pubkey position: 472
Segmentation fault (core dumped)

This only happens with rtmpe hosts and I have tracked the issue further down to the dh.h header file function:

DHGenerateKey(MDH *dh)
{
  size_t res = 0;
  if (!dh)
    return 0;

  while (!res)
    {
      MP_t q1 = NULL;

      if (!MDH_generate_key(dh))

So the error happens when calling the OpenSSL dh_generate_key function.

This only happens with rtmpe streams. Normal rtmp streams work fine on the locally built librtmp. Is there anyone who could explain or solve this?

Thanks for all the great work.

PS: this seems to be the same issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232901

L2501 commented 5 years ago

build rtmpdump with gnutls instead of openssl

jairoxyz commented 5 years ago

Thanks @L2501 - I guess that is straight forward on Ubuntu, but I'm not so sure about Windows or Android. Also, I want the resulting librtmp.so to be compatible with inputstream.rtmp and I believe to understand that Kodi insists in OpenSSL and so I wonder if that GNUTLS librtmp would work. I'll wait and see if FreeBSD or someone else come up with a bug fix. This should work on all platforms then.

Actually, it just came to my mind to try inputstream.rtmp 2.0.4 built for Windows following https://github.com/xbmc/xbmc/blob/master/docs/README.Windows.md and it actually crashes Kodi 18 Rc2 when trying to connect to rtmpe hosts! This obviously happens when librtmp with SSL 1.1 patches is statically linked into the inputstream dll.

Rechi commented 5 years ago

@jairoxyz please retry with https://github.com/xbmc/inputstream.rtmp/compare/Rechi:fix/openssl1.1.1

jairoxyz commented 5 years ago

Thanks @Rechi

No more core dumps with patched librtmp.so.1. but streams that play with GNUTLS don't play with OPENSSL 1.1.

The issue is with handshaking:

DEBUG: RTMP_Connect1, ... connected, handshaking
DEBUG: HandShake: Client type: 06
DEBUG: HandShake: DH pubkey position: 33
WARNING: DH public key does not fulfill y^q mod p = 1
DEBUG: HandShake: Client digest offset: 846
DEBUG: HandShake: Initial client digest:
DEBUG: 77 bd f2 bc b4 06 45 6b d6 57 13 30 89 28 42 ee
DEBUG: 0d dc 0f a2 11 91 f4 6f 95 86 33 5b 34 51 94 03
DEBUG: HandShake: Type Answer   : 06
DEBUG: HandShake: Server Uptime : 795146699
DEBUG: HandShake: FMS Version   : 5.0.14.1
DEBUG: HandShake: Server DH public key offset: 528
WARNING: DH public key does not fulfill y^q mod p = 1
DEBUG: HandShake: Secret key:

jairoxyz commented 5 years ago

I think I fixed it. Using your original patch for dh.h because I read that the DH_generate_parameters_ex function might not return the expected, I checked and saw that the DH parameters were not set back to the DH object. This is the init function with my changes and it works with rtmpdump:

static MDH *
DHInit(int nKeyBits)
{
  size_t res;
  MDH *dh = MDH_new();

  if (!dh)
    goto failed;

#if !defined(USE_OPENSSL) || !defined(OPENSSL_VERSION_NUMBER) || OPENSSL_VERSION_NUMBER < 0x10100000L
  MP_new(dh->g);

  if (!dh->g)
    goto failed;
#else
  BIGNUM *g = NULL;
  MP_new(g);
  if (!g)
    goto failed;

#endif

#if !defined(USE_OPENSSL) || !defined(OPENSSL_VERSION_NUMBER) || OPENSSL_VERSION_NUMBER < 0x10100000L
  MP_gethex(dh->p, P1024, res); /* prime P1024, see dhgroups.h */
#else
  BIGNUM* p = NULL;
  DH_get0_pqg(dh, (BIGNUM const**)&p, NULL, NULL);
  MP_gethex(p, P1024, res); /* prime P1024, see dhgroups.h */

#endif
  if (!res)
    {
      goto failed;
    }

#if !defined(USE_OPENSSL) || !defined(OPENSSL_VERSION_NUMBER) || OPENSSL_VERSION_NUMBER < 0x10100000L
  MP_set_w(dh->g, 2);   /* base 2 */
#else
  MP_set_w(g, 2);   /* base 2 */
  DH_set0_pqg(dh, p, NULL, g);

#endif

#if !defined(USE_OPENSSL) || !defined(OPENSSL_VERSION_NUMBER) || OPENSSL_VERSION_NUMBER < 0x10100000L
  dh->length = nKeyBits;
#else
  DH_set_length(dh, nKeyBits);
#endif
  return dh;

failed:
  if (dh)
    MDH_free(dh);

  return 0;
}

PS: There is a new issue on Ubuntu when using --swfVfy for hash autocalculation. This also causes core dump and Kodi to crash.

PPS: found that bug too. Its in line 236 of the patch file. Missing HMAC_CTX_new()

+#define HMAC_setup(ctx, key, len) ctx = HMAC_CTX_new(); HMAC_CTX_reset(ctx); HMAC_Init_ex(ctx, key, len, EVP_sha256(), 0)

Jx-

jairoxyz commented 5 years ago

I will close this issue now and make a PR with my fixes. Feel free to use them or reject. No worries. Thanks for you help 👍

xbmc / inputstream.rtmp

Core dump with locally built librtmp #45