Error while processing bytes data in python 3

[bj8798]

Faced this issue while using splunklib with pyhton 3 lib version: 1.6.0 platform: Linux(centos) framework version: Python 3.7.2 Splunk version: 8.0.0 (build: 7f1fa0caa96a)

The error is coming when some of the data is coming in bytes format, following is the trace of the error.

2019-07-04 17:28:11,907, Level=ERROR, Pid=28569, File=search_command.py, Line=981, TypeError at "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/internals.py", line 590 : can't concat str to bytes** Traceback: File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/search_command.py", line 603, in _process_protocol_v1 self._execute(ifile, None) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/streaming_command.py", line 54, in _execute SearchCommand._execute(self, ifile, self.stream) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/search_command.py", line 848, in _execute self._record_writer.write_records(process(self._records(ifile))) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/internals.py", line 522, in write_records write_record(record) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/internals.py", line 590, in _write_record sv += value + '\n'**

In below code snippet, it is trying to concat bytes with str, so that it is throwing an error. Issues I can identify is,

• type bytes is not handled
• line value = repr(value).encode('utf-8', errors='backslashreplace') is converting incoming data in bytes format

• concat performed in line sv += value+ '\n', which will try to concat string and bytes.

                 if value_t is not bytes:
                      if value_t is bool:
                          value = str(value.real)
                      elif value_t is six.text_type:
                          value = value
                      elif value_t is int or value_t is int or value_t is float or value_t is complex:
                          value = str(value)
                      elif issubclass(value_t, (dict, list, tuple)):
                          value = str(''.join(RecordWriter._iterencode_json(value, 0)))
                      else:
                          value = repr(value).encode('utf-8', errors='backslashreplace')
  
                  sv += value+ '\n'
                  mv += value.replace('$', '$$') + '$;$'

I changed above code like this to make it work, though I am not sure about decoding format. could you please look into it?

                 if value_t is not bytes:

                        if value_t is bool:
                            value = str(value.real)
                        elif value_t is six.text_type:
                            value = value
                        elif value_t is int or value_t is int or value_t is float or value_t is complex:
                            value = str(value)
                        elif issubclass(value_t, (dict, list, tuple)):
                            value = str(''.join(RecordWriter._iterencode_json(value, 0)))
                        else:
                            if six.PY2:
                                value = repr(value).encode('utf-8', errors='backslashreplace')
                            else:
                                value = repr(value)

                    if six.PY3 and value_t is bytes:
                        value = value.decode('utf-16')

                    sv += value + '\n'
                    mv += value.replace('$', '$$') + '$;$'

[shakeelmohamed]

@bj8798 thanks for reporting the bug, can you try upgrading to the latest SDK? If that doesn't work, please open a PR if you've got a working fix

[tdhellmann]

This issue was reported on a version of the Python SDK that did not include support for Python 3 and hasn't had activity since - if this issue can be reproed on 1.6.6+, I'll reopen