Python 3.x byte vs. string problem

kdelee commented 6 years ago

Specify type:

Bug

Bug severity (if applicable):

Medium

Description:

When running rho in a python 3.6 virtual environment, an error can be produced that does not block the scan, but is disconcerting.

Bug Report

Version of rho:

[caea38dadfd8bac42b41c8e05023dc8a0e0f6866 ] (master)

Expected behavior:

No tracebacks from python errors are produced by normal workflow.

Actual behavior:

At connection phase, the following error is produced:

Attempting connection discovery with auth "sonar-root".
Exception in thread STDOUT/ERR thread for pid 5349:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/elijah/envs/camayoc/lib64/python3.6/site-packages/sh.py", line 1540, in wrap
    fn(*args, **kwargs)
  File "/home/elijah/envs/camayoc/lib64/python3.6/site-packages/sh.py", line 2484, in output_thread
    done = f.read()
  File "/home/elijah/envs/camayoc/lib64/python3.6/site-packages/sh.py", line 2943, in read
    self.write_chunk(chunk)
  File "/home/elijah/envs/camayoc/lib64/python3.6/site-packages/sh.py", line 2918, in write_chunk
    self.should_quit = self.process_chunk(chunk)
  File "/home/elijah/envs/camayoc/lib64/python3.6/site-packages/sh.py", line 2839, in process
    return handler(chunk)
  File "/home/elijah/envs/camayoc/lib64/python3.6/site-packages/sh.py", line 1624, in fn
    return handler(chunk, *args)
  File "/home/elijah/sfw/quipu/rho/rho/utilities.py", line 125, in process_discovery_scan
    logfile.write('******* %s *******' % (str(datetime.now())))
TypeError: a bytes-like object is required, not 'str'

(scan continues and is able to complete after this)

Steps to reproduce:

Install rho in a python3.6 virtual environment (including system-site-packages re: #512) Scan test-matrix machines (did not happen on only localhost/local hosted VM scan)

I'd talk to Elyezer about this, he has battled this type of python 3 issue before.

Environment information:

Type	Operating System	OS Version	Python Version	Virtualization
Scanning System	Fedora	26	3.6	Bare Metal
Target Systems	RHEL	5.9, 6.9, 7.3	2.X	VCenter

kdelee commented 6 years ago

FYI this does NOT happen in python 2.X, this is something that arises in python 3.X because of changes in the relationship between byte strings and strings between 2.X and 3.X

I'm not an expert on this subject, but here is some info:

https://eli.thegreenplace.net/2012/01/30/the-bytesstr-dichotomy-in-python-3

elyezer commented 6 years ago

Usually I follow the unicode sandwitch (for more info [1]) where inside the application code all data is string and when reading or outputting convert to bytest. That said the generated string can be then encoded, that should provide good result on Python 2 and 3, something like:

logfile.write(('******* %s *******' % (str(datetime.now()))).encode('utf-8'))

[1] https://nedbatchelder.com/text/unipain.html

quipucords / rho