negative values in "segment bytes" gives "Decoder failure: Out of memory"

sanderjo commented 5 years ago

As reported here https://forums.sabnzbd.org/viewtopic.php?f=3&t=24140&p=119081#p119081

Out of Memory with a NZB with all "segment bytes" parts set to negative value, for example 'segment bytes="-728914"'

Crafted NZB here https://raw.githubusercontent.com/sanderjo/NZBs/master/test-negative-segment-bytes-reftestnzb.nzb

2019-06-21 07:22:37,232::DEBUG::[nzbstuff:293] Finishing import on reftestnzb 100MB auto 2151de41 2019-06-15 [01/19] - "mytestpost.part01.rar" yEnc (1/15) 10485760
2019-06-21 07:22:37,233::INFO::[decoder:141] Decoder-Queue: 0, Cache: 0, 0, 1003487232
2019-06-21 07:22:37,235::DEBUG::[__init__:924] [sabnzbd.nzbstuff.finish_import] Loading data for SABnzbd_nzf_XX_hp5 from /home/sander/Downloads/incomplete/test-negative-segment-bytes-reftestnzb/__ADMIN__/SABnzbd_nzf_XX_hp5
2019-06-21 07:22:37,235::INFO::[decoder:142] Traceback: 
Traceback (most recent call last):
  File "/usr/share/sabnzbdplus/sabnzbd/decoder.py", line 124, in run
    data = self.decode(article, lines, raw_data)
  File "/usr/share/sabnzbdplus/sabnzbd/decoder.py", line 222, in decode
    decoded_data, output_filename, crc, crc_expected, crc_correct = sabyenc.decode_usenet_chunks(raw_data, article.bytes)
MemoryError
2019-06-21 07:22:37,239::INFO::[downloader:279] Pausing

Strange, because sabyenc seems to check for negative values before doing the mallo() https://github.com/sabnzbd/sabyenc/blob/sabyenc-python2/src/sabyenc.c#L563-L573

Doing malloc() with a negative number is very bad.

sanderjo commented 5 years ago

Ugly code:

    def decode(self, article, data, raw_data):
        # Do we have SABYenc? Let it do all the work
        if sabnzbd.decoder.SABYENC_ENABLED:
            mytotal = 0
            for myi in raw_data: 
                mytotal += len(myi)
            logging.debug("SJ: length of raw_data (calculated): %d", mytotal )
            logging.debug("SJ: article.bytes: %s", article.bytes)
            #logging.debug("SJ: raw_data: %s", raw_data )
            decoded_data, output_filename, crc, crc_expected, crc_correct = sabyenc.decode_usenet_chunks(raw_data, abs(article.bytes))

gives

019-06-21 08:07:00,846::INFO::[nzbstuff:1693] Checking all filenames for test-negative-segment-bytes-reftestnzb
2019-06-21 08:07:00,847::INFO::[nzbstuff:1696] Re-sorting test-negative-segment-bytes-reftestnzb after getting filename information
2019-06-21 08:07:03,647::DEBUG::[decoder:225] SJ: length of raw_data (calculated): 728203
2019-06-21 08:07:03,648::DEBUG::[decoder:226] SJ: article.bytes: -728845
2019-06-21 08:07:03,991::DEBUG::[decoder:225] SJ: length of raw_data (calculated): 728204
2019-06-21 08:07:03,992::DEBUG::[decoder:226] SJ: article.bytes: -728846
2019-06-21 08:07:04,681::DEBUG::[decoder:225] SJ: length of raw_data (calculated): 728207
2019-06-21 08:07:04,683::DEBUG::[decoder:226] SJ: article.bytes: -728906
2019-06-21 08:07:05,356::DEBUG::[decoder:225] SJ: length of raw_data (calculated): 728207
2019-06-21 08:07:05,357::DEBUG::[decoder:226] SJ: article.bytes: -728850
2019-06-21 08:07:05,630::DEBUG::[bpsmeter:268] bps: 1551934.70799
2019-06-21 08:07:05,733::DEBUG::[decoder:225] SJ: length of raw_data (calculated): 728207
2019-06-21 08:07:05,734::DEBUG::[decoder:226] SJ: article.bytes: -728850

... and the download works

sanderjo commented 5 years ago

Solved by https://github.com/sabnzbd/sabyenc/pull/15

Closing

sabnzbd / sabctools

negative values in "segment bytes" gives "Decoder failure: Out of memory" #13