Lepton denial-of-service with specially crafted JPEG files

[april-dbx]

This issue was originally reported by Tenable.

As with the image above, it seems to be possible to create a crafted JPEG file which will prompt lepton to produce a LEP file which, when converted back to JPEG to verify that the compression was lossless, will throw lepton into an infinite loop, hanging the process until it's killed. The sensitive section of code seems to be the following bit in jpgcoder.cc:

/* -----------------------------------------------
    run of EOB encoding routine
    ----------------------------------------------- */
int encode_eobrun( abitwriter* huffw, huffCodes* actbl, unsigned int* eobrun )
{{{}}
    unsigned short n;
    unsigned int  s;
    int hc;
    if ( (*eobrun) > 0 ) {
        while ( (*eobrun) > actbl->max_eobrun ) {
            huffw->write( actbl->cval[ 0xE0 ], actbl->clen[ 0xE0 ] ); 
            huffw->write( E_ENVLI( 14, 32767 ), 14 );
            (*eobrun) -= actbl->max_eobrun;
{{        }}}
        s = uint16bit_length((*eobrun));
        dev_assert(s && "actbl->max_eobrun needs to be > 0");
        if (s) s--; 
        n = E_ENVLI( s, (*eobrun) );
        hc = ( s << 4 ); 
        huffw->write( actbl->cval[ hc ], actbl->clen[ hc ] ); 
        huffw->write( n, s ); 
        (*eobrun) = 0; 
    }    
    return 0;
}