Import scan error for charmap

[hertzl]

We import scans via pytenable from standalone/airgapped nessus scanners. We have few imports that continue have errors such as the following:

20-11-09 16:40:26 root INFO Failure in api call scans.import_scan() in iteration 1 with error message: 'charmap' codec can't decode byte 0x81 in position 6539199: character maps to 20-11-09 16:40:26 root INFO retrying... 20-11-09 16:40:28 tenable.errors.InvalidInputError ERROR POST https://cloud.tenable.com/scans/import?include_aggregate=1 >> [ID removed]:400 {"error":"Invalid XML file format (only NessusClientData_v2 is supported)"}

We have seen Cyrillic characters in the plugin output at the offset...

The same nessus file imports manually through the cloud.tenable.com UI without issue.

[SteveMcGrath]

can you please provide some debug logging and some example code?

[hertzl]

Sample call and more logging:

while api_call_count <= limit and success == 0:
    try:
        # perform the api call here
        xxx_import = tio.scans.import_scan(fobj=fobj)
        success = 1
        # return the data from the call
        return xxx_import
    except Exception as e:

        # print the error message to the console and write it to a log file
        log('Failure in api call scans.import_scan() in iteration ' + str(api_call_count) + " with error message: " + str(e))
        if api_call_count < limit:
            api_call_count = api_call_count + 1
            log('retrying...')
        else:
            success = -1
            log('Maximum number of retries reached exiting...')
return {}

DEBUG:root:----------------------------------------START---------------------------------------- Kicking off automation - log can be found at .\scan_logs\20201211_xxx.log DEBUG:root:Kicking off automation - log can be found at .\scan_logs\20201211_xxx.log 1 file(s) found DEBUG:root:1 file(s) found Handling xxxXXX_Group2_larhk4.nessus DEBUG:root:Handling xxxXXX_Group2_larhk4.nessus Site code: XXX DEBUG:root:Site code: XXX xxxXXX_Group2_larhk4.nessus has been moved to ..xxx_staging\xxxXXX_Group2_larhk4.nessus DEBUG:root:xxxXXX_Group2_larhk4.nessus has been moved to ..xxx_staging\xxxXXX_Group2_larhk4.nessus Importing ..xxx_staging\xxxXXX_Group2_larhk4.nessus DEBUG:root:Importing ..xxx_staging\xxxXXX_Group2_larhk4.nessus DEBUG:tenable.io.TenableIO:{"method": "POST", "url": "https://cloud.tenable.com/file/upload", "params": {}, "body": {}}

e:\automation\kxxx.py(51)tioScansImportScan() -> log('Failure in api call scans.import_scan() in iteration ' + str(api_call_count) + " with error message: " + str(e))

Traceback (most recent call last): File "xxx.py", line 43, in tioScansImportScan xxx_import = tio.scans.import_scan(fobj=fobj) File "c:\Program Files (x86)\Python\lib\site-packages\tenable\io\scans.py", line 743, in import_scan payload['file'] = self._api.files.upload(fobj) File "c:\Program Files (x86)\Python\lib\site-packages\tenable\io\files.py", line 48, in upload return self._api.post('file/upload', kw).json()['fileuploaded'] File "c:\Program Files (x86)\Python\lib\site-packages\tenable\base\v1.py", line 578, in post return self._request('POST', path, kwargs) File "c:\Program Files (x86)\Python\lib\site-packages\tenable\base\v1.py", line 488, in _request resp = self._session.request(method, File "C:\Users\XXX\AppData\Roaming\Python\Python38\site-packages\requests\sessions.py", line 519, in request prep = self.prepare_request(req) File "C:\Users\XXX\AppData\Roaming\Python\Python38\site-packages\requests\sessions.py", line 452, in prepare_request p.prepare( File "C:\Users\XXX\AppData\Roaming\Python\Python38\site-packages\requests\models.py", line 316, in prepare self.prepare_body(data, files, json) File "C:\Users\XXX\AppData\Roaming\Python\Python38\site-packages\requests\models.py", line 504, in prepare_body (body, content_type) = self._encode_files(files, data) File "C:\Users\XXX\AppData\Roaming\Python\Python38\site-packages\requests\models.py", line 159, in _encode_files fdata = fp.read() File "c:\Program Files (x86)\Python\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6453615: character maps to

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "xxx.py", line 174, in xxx_import = tioScansImportScan(tio, nessus, MAX_RETRIES) File "xxx.py", line 51, in tioScansImportScan log('Failure in api call scans.import_scan() in iteration ' + str(api_call_count) + " with error message: " + str(e)) File "xxx.py", line 51, in tioScansImportScan log('Failure in api call scans.import_scan() in iteration ' + str(api_call_count) + " with error message: " + str(e)) File "c:\Program Files (x86)\Python\lib\bdb.py", line 88, in trace_dispatch return self.dispatch_line(frame) File "c:\Program Files (x86)\Python\lib\bdb.py", line 113, in dispatch_line if self.quitting: raise BdbQuit bdb.BdbQuit

[SteveMcGrath]

how are you opening the file? in binary mode or string mode?

[hertzl]

String mode.  I now see binary used in the pytenable examples...is this the fix?   On Tuesday, January 5, 2021, 10:01:06 AM MST, Steven McGrath notifications@github.com wrote:

how are you opening the file? in binary mode or string mode?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[SteveMcGrath]

Correct, use binary mode and you likely wont have the issue. Please report back and close the ticket if the issue is resolved.

[hertzl]

Will do... On Tuesday, January 5, 2021, 01:18:33 PM MST, Steven McGrath notifications@github.com wrote:

Correct, use binary mode and you likely wont have the issue. Please report back and close the ticket if the issue is resolved.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[hertzl]

Binary mode resolved the problem...