openpreserve / jpylyzer

JP2 (JPEG 2000 Part 1) validator and properties extractor. Jpylyzer was specifically created to check that a JP2 file really conforms to the format's specifications. Additionally jpylyzer is able to extract technical characteristics.
http://jpylyzer.openpreservation.org/
Other
69 stars 28 forks source link

pylint and pychecker show many warnings, one error, when run against jpylyzer.py #91

Closed boxerab closed 4 years ago

boxerab commented 8 years ago

Looks like static analysis would be helpful.

bitsgalore commented 6 years ago

Finally got round to a look at this. The Pylint fixes by @stweil are now merged into the master branch.

Running Pylint resulted in lots of warnings; by the look of it too many to fix manually (and most don't look serious).

However running Pylint with the -E switch yielded the following errors:

************* Module jpylyzer.jpylyzer
E:623,53: Instance of 'file' has no 'buffer' member (no-member)
************* Module jpylyzer.six
E: 92,17: Instance of '_LazyDescr' has no '_resolve' member (no-member) 
E:723, 4: function already defined line 721 (function-redefined)
E:779, 4: function already defined line 721 (function-redefined)
E:852, 4: Undefined variable '__spec__' (undefined-variable)

Ignoring the six errors for now, the 'file' has no 'buffer' originates from the following lines (the offending line is only used for Python 3):

# Set encoding of the terminal to UTF-8
if config.PYTHON_VERSION.startswith(config.PYTHON_2):
    out = codecs.getwriter(config.UTF8_ENCODING)(sys.stdout)
elif config.PYTHON_VERSION.startswith(config.PYTHON_3):
    out = codecs.getwriter(config.UTF8_ENCODING)(sys.stdout.buffer)

Looking at the Python docs, this looks completely legal to me in Python 3 (but not Python 2):

https://docs.python.org/3/library/sys.html

Pylint3 does not report this error, but instead it reports various other errors:

************* Module jpylyzer.boxvalidator
E: 22, 0: Unable to import 'shared' (import-error)
E: 23, 0: Unable to import 'shared' (import-error)
E:227,11: Module 'config' has no 'outputVerboseFlag' member (no-member)
E:2059,15: Module 'config' has no 'extractNullTerminatedXMLFlag' member (no-member)
E:2109,19: Module 'config' has no 'extractNullTerminatedXMLFlag' member (no-member)
************* Module jpylyzer.etpatch
E: 17, 0: Unable to import 'byteconv' (import-error)
E: 18, 0: Unable to import 'byteconv' (import-error)
E:110,15: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:110,48: Module 'config' has no 'PYTHON_2' member (no-member)
E:112,37: Undefined variable 'long' (undefined-variable)
************* Module jpylyzer.jpylyzer
E: 49, 0: Unable to import 'boxvalidator' (import-error)
E: 51, 0: Unable to import 'byteconv' (import-error)
E: 52, 0: Unable to import 'shared' (import-error)
E:260,15: Module 'config' has no 'PLATFORM' member (no-member)
E:381,17: Module 'config' has no 'ERR_CODE_NO_IMAGES' member (no-member)
E:390,17: Module 'config' has no 'ERR_CODE_NO_IMAGES' member (no-member)
E:406, 7: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:406,40: Module 'config' has no 'PYTHON_3' member (no-member)
E:417, 7: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:417,40: Module 'config' has no 'PYTHON_2' member (no-member)
E:486,11: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:486,44: Module 'config' has no 'PYTHON_2' member (no-member)
E:488,19: Undefined variable 'unicode' (undefined-variable)
E:578, 7: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:578,40: Module 'config' has no 'PYTHON_2' member (no-member)
E:580, 7: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:580,40: Module 'config' has no 'PYTHON_3' member (no-member)
E:600,11: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:600,44: Module 'config' has no 'PYTHON_2' member (no-member)
E:605,11: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:605,44: Module 'config' has no 'PYTHON_3' member (no-member)
E:620, 7: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:620,40: Module 'config' has no 'PYTHON_2' member (no-member)
E:621,31: Module 'config' has no 'UTF8_ENCODING' member (no-member)
E:622, 9: Module 'config' has no 'PYTHON_VERSION' member (no-member)
E:622,42: Module 'config' has no 'PYTHON_3' member (no-member)
E:623,31: Module 'config' has no 'UTF8_ENCODING' member (no-member)
************* Module jpylyzer.six
E: 49,19: Undefined variable 'basestring' (undefined-variable)
E: 50,26: Undefined variable 'long' (undefined-variable)
E: 51,25: Module 'types' has no 'ClassType' member (no-member)
E: 52,16: Undefined variable 'unicode' (undefined-variable)
E: 92,17: Instance of '_LazyDescr' has no '_resolve' member (no-member)
E:647,15: Undefined variable 'unicode' (undefined-variable)
E:656,34: Module 'itertools' has no 'imap' member (no-member)
E:723, 4: function already defined line 721 (function-redefined)
E:730,36: Undefined variable 'basestring' (undefined-variable)
E:733,31: Undefined variable 'file' (undefined-variable)
E:734,37: Undefined variable 'unicode' (undefined-variable)
E:744,31: Undefined variable 'unicode' (undefined-variable)
E:750,31: Undefined variable 'unicode' (undefined-variable)
E:758,35: Undefined variable 'unicode' (undefined-variable)
E:762,22: Undefined variable 'unicode' (undefined-variable)
E:763,20: Undefined variable 'unicode' (undefined-variable)
E:779, 4: function already defined line 721 (function-redefined)

Some observations:

bitsgalore commented 6 years ago

Related to this, I also ran the code through PEP8 (hadn't done that in a while) which also yielded several (mostly stylistic) issues. These are now all fixed (https://github.com/openpreserve/jpylyzer/commit/650d95c1d7d7652a5a17304fed13e97a4fadaa55)

bitsgalore commented 6 years ago

Also related: instead of PyChecker (which hasn't been updated since 2011 7 also doesn't work with Py 3 i think) a better alternative might be PyFlakes. Running the Py 2.7 version:

pyflakes ~/jpylyzer/jpylyzer/*.py >pyflakes.txt 

Result:

/home/johan/jpylyzer/jpylyzer/boxvalidator.py:328: local variable 'bytesTotal' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:959: local variable 'bytesTotal' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:1308: local variable 'tilePartIndex' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:1935: local variable 'foundSODMarker' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:2138: local variable 'bytesTotal' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:2225: local variable 'tmp' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:2254: local variable 'bytesTotal' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:41: 'struct' imported but unused
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:47: 'fnmatch' imported but unused
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:51: 'bytesToText' imported but unused
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:485: local variable 'thisDirectory' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/six.py:852: undefined name '__spec__'

Most of these errors are fixed in https://github.com/openpreserve/jpylyzer/commit/37b68c9b3656eb318dcc6642ad3caf9d66bdb26d.

And with the Python 3 version:

/home/johan/jpylyzer/jpylyzer/boxvalidator.py:328: local variable 'bytesTotal' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:959: local variable 'bytesTotal' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:1308: local variable 'tilePartIndex' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:1935: local variable 'foundSODMarker' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:2138: local variable 'bytesTotal' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:2225: local variable 'tmp' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/boxvalidator.py:2254: local variable 'bytesTotal' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/etpatch.py:112: undefined name 'long'
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:41: 'struct' imported but unused
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:47: 'fnmatch' imported but unused
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:51: 'byteconv.bytesToText' imported but unused
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:485: local variable 'thisDirectory' is assigned to but never used
/home/johan/jpylyzer/jpylyzer/jpylyzer.py:501: undefined name 'unicode'
/home/johan/jpylyzer/jpylyzer/six.py:49: undefined name 'basestring'
/home/johan/jpylyzer/jpylyzer/six.py:50: undefined name 'long'
/home/johan/jpylyzer/jpylyzer/six.py:52: undefined name 'unicode'
/home/johan/jpylyzer/jpylyzer/six.py:647: undefined name 'unicode'
/home/johan/jpylyzer/jpylyzer/six.py:730: undefined name 'basestring'
/home/johan/jpylyzer/jpylyzer/six.py:733: undefined name 'file'
/home/johan/jpylyzer/jpylyzer/six.py:734: undefined name 'unicode'
/home/johan/jpylyzer/jpylyzer/six.py:744: undefined name 'unicode'
/home/johan/jpylyzer/jpylyzer/six.py:750: undefined name 'unicode'
/home/johan/jpylyzer/jpylyzer/six.py:758: undefined name 'unicode'
/home/johan/jpylyzer/jpylyzer/six.py:762: undefined name 'unicode'
/home/johan/jpylyzer/jpylyzer/six.py:763: undefined name 'unicode'

Turns out the additional errors reported by the Py3 version all refer to Python 2-specific parts of the code, so for compatibility reasons they shouldn't be fixed.

boxerab commented 6 years ago

Thanks so much for looking into this. Jpylyzer is awesome!

bitsgalore commented 6 years ago

@boxerab Thank you! For info I just published a new release here with some additional Pylint fixes. Meanwhile I'll leave this issue open because this needs more work this in the future.