Integer signedness and overflow issues in OS X regex engine (TRE)

[GoogleCodeExporter]

The OS X regex engine function tre_tnfa_run_parallel contains the following 
code:

  int tbytes;
...
  if (!match_tags)
    num_tags = 0;
  else
    num_tags = tnfa->num_tags;

...
  {
    int rbytes, pbytes, total_bytes;
    char *tmp_buf;
    /* Compute the length of the block we need. */
    tbytes = sizeof(*tmp_tags) * num_tags;
    rbytes = sizeof(*reach_next) * (tnfa->num_states + 1);
    pbytes = sizeof(*reach_pos) * tnfa->num_states;
    total_bytes =
      (sizeof(long) - 1) * 4 /* for alignment paddings */
      + (rbytes + tbytes * tnfa->num_states) * 2 + tbytes + pbytes;

    DPRINT(("tre_tnfa_run_parallel, allocate %d bytes\n", total_bytes));
    /* Allocate the memory. */
#ifdef TRE_USE_ALLOCA
    buf = alloca(total_bytes);
#else /* !TRE_USE_ALLOCA */
    buf = xmalloc((unsigned)total_bytes);  <-- malloc is called, not alloca
#endif /* !TRE_USE_ALLOCA */
    if (buf == NULL)
      return REG_ESPACE;
    memset(buf, 0, (size_t)total_bytes);

num_states and num_tags are computed based on the requirements of the regex and 
it's quite easy to make them each >64k with a relatively small regex. Note that 
total_bytes is an int and part of its calculation is the product of num_states 
and num_tags.

The types here are all over the place and there's conversion between int, 
unsigned's and size_t.

The attached PoC causes total_bytes to become negative leading to total_bytes 
being sign-extended in the memset call.

Severity medium because I haven't looked for exposed attack surface yet, but 
this doesn't require any non-standard flags (only REG_EXTENDED which is almost 
always used.)

Original issue reported on code.google.com by ianb...@google.com on 4 Jun 2015 at 5:16

Attachments:

• tre_signedness_bad_bzero.c

[GoogleCodeExporter]

Original comment by ianb...@google.com on 10 Jun 2015 at 11:55

• Added labels: Id-623973357, Reported-2015-Jun-10

[GoogleCodeExporter]

OS X advisory: https://support.apple.com/en-us/HT205031
iOS advisory: https://support.apple.com/en-us/HT205030

Original comment by ianb...@google.com on 14 Aug 2015 at 1:44

• Changed state: Fixed
• Added labels: CVE-2015-3798, Fixed-2015-Aug-13, Product-iOS

[GoogleCodeExporter]

Original comment by ianb...@google.com on 21 Sep 2015 at 10:35

• Removed labels: Restrict-View-Commit