search

iDigBio / idb-backend

iDigBio server and backend code for data ingestion, media processing, record indexing, and data API.
GNU General Public License v3.0
7 stars 0 forks source link

pylint Python 3 porting report for `idigbio_ingestion` #160

Open danstoner opened 3 years ago

danstoner commented 3 years ago 
~/git/IDIGBIO/idb-backend]$ pylint --py3k idigbio_ingestion
************* Module idigbio_ingestion.db_check
idigbio_ingestion/db_check.py:304:56: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
idigbio_ingestion/db_check.py:308:52: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
idigbio_ingestion/db_check.py:395:13: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
idigbio_ingestion/db_check.py:396:13: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.db_rsids
idigbio_ingestion/db_rsids.py:1:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.verify_ceph_objects
idigbio_ingestion/verify_ceph_objects.py:49:31: W1645: Exception.message removed in Python 3 (exception-message-attribute)
************* Module idigbio_ingestion.update_publisher_recordset
idigbio_ingestion/update_publisher_recordset.py:2:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.lib.eml
idigbio_ingestion/lib/eml.py:2:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
idigbio_ingestion/lib/eml.py:141:15: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.lib.util
idigbio_ingestion/lib/util.py:1:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.lib.delimited
idigbio_ingestion/lib/delimited.py:1:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
idigbio_ingestion/lib/delimited.py:97:20: W1654: dict.items referenced when not iterating (dict-items-not-iterating)
idigbio_ingestion/lib/delimited.py:122:4: W1653: next method defined (next-method-defined)
************* Module idigbio_ingestion.lib.fileproxy
idigbio_ingestion/lib/fileproxy.py:60:4: W1653: next method defined (next-method-defined)
idigbio_ingestion/lib/fileproxy.py:61:12: W1622: Called a next() method on an object (next-method-called)
idigbio_ingestion/lib/fileproxy.py:108:4: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.lib.dwca
idigbio_ingestion/lib/dwca.py:1:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.lib.xmlDictTools
idigbio_ingestion/lib/xmlDictTools.py:5:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module idigbio_ingestion.mediaing.fetcher
idigbio_ingestion/mediaing/fetcher.py:49:12: W1656: dict.values referenced when not iterating (dict-values-not-iterating)
idigbio_ingestion/mediaing/fetcher.py:116:16: W1636: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.mediaing.derivatives
idigbio_ingestion/mediaing/derivatives.py:87:39: W1639: filter built-in referenced when not iterating (filter-builtin-not-iterating)
idigbio_ingestion/mediaing/derivatives.py:180:16: W1636: map built-in referenced when not iterating (map-builtin-not-iterating)

-----------------------------------
Your code has been rated at 9.90/10
danstoner commented 3 years ago

From that big list, the following are the ones that would need to befixed for Python 3 conversion project:

W1655 dict-keys-not-iterating Message 'dict.keys referenced when not iterating'

Description Used when dict.keys is referenced in a non-iterating context (returns an iterator in Python 3)

W1653 next-method-defined Message 'next method defined'

Description Used when a next method is defined that would be an iterator in Python 2 but is treated as a normal function in Python 3.

W1622 next-method-called Message 'Called a next() method on an object'

Description Used when an object’s next() method is called (Python 3 uses the next() built-in function)

W1639 filter-builtin-not-iterating Message 'filter built-in referenced when not iterating'

Description Used when the filter built-in is referenced in a non-iterating context (returns an iterator in Python 3)

W1636 map-builtin-not-iterating Message 'map built-in referenced when not iterating'

Description Used when the map built-in is referenced in a non-iterating context (returns an iterator in Python 3)

Issues in verify_ceph_objects.py can wait since that is not a program we are going to run on a regular basis, or possibly, ever again.

danstoner commented 3 years ago

Get rid of the no-absolute-import lines (basically a false positive since we don't need that in py3):

pylint --py3k --disable=no-absolute-import   idigbio_ingestion
************* Module idigbio_ingestion.verify_ceph_objects
idigbio_ingestion/verify_ceph_objects.py:49:31: W1645: Exception.message removed in Python 3 (exception-message-attribute)
************* Module idigbio_ingestion.db_check
idigbio_ingestion/db_check.py:308:52: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.lib.fileproxy
idigbio_ingestion/lib/fileproxy.py:60:4: W1653: next method defined (next-method-defined)
idigbio_ingestion/lib/fileproxy.py:61:12: W1622: Called a next() method on an object (next-method-called)
************* Module idigbio_ingestion.lib.delimited
idigbio_ingestion/lib/delimited.py:97:20: W1654: dict.items referenced when not iterating (dict-items-not-iterating)
idigbio_ingestion/lib/delimited.py:122:4: W1653: next method defined (next-method-defined)
************* Module idigbio_ingestion.lib.eml
idigbio_ingestion/lib/eml.py:141:15: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.mediaing.fetcher
idigbio_ingestion/mediaing/fetcher.py:48:12: W1656: dict.values referenced when not iterating (dict-values-not-iterating)
idigbio_ingestion/mediaing/fetcher.py:115:16: W1636: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.mediaing.derivatives
idigbio_ingestion/mediaing/derivatives.py:87:39: W1639: filter built-in referenced when not iterating (filter-builtin-not-iterating)
idigbio_ingestion/mediaing/derivatives.py:180:16: W1636: map built-in referenced when not iterating (map-builtin-not-iterating)

------------------------------------------------------------------
danstoner commented 2 years ago

The --py3k option only works in python2 pylint.

Updated 3/23/2022:

pylint --py3k --disable=no-absolute-import   idigbio_ingestion
No config file found, using default configuration
************* Module idigbio_ingestion.db_rsids
E: 36, 8: print statement used (print-statement)
E: 41,12: print statement used (print-statement)
E: 51, 8: print statement used (print-statement)
************* Module idigbio_ingestion.verify_ceph_objects
W: 49,31: Exception.message removed in Python 3 (exception-message-attribute)
************* Module idigbio_ingestion.mediaing.fetcher
W: 48,12: dict.values referenced when not iterating (dict-values-not-iterating)
W:115,16: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.mediaing.derivatives
W: 87,39: filter built-in referenced when not iterating (filter-builtin-not-iterating)
W:180,16: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.lib.xmlDictTools
E:147, 4: print statement used (print-statement)
E:270, 4: print statement used (print-statement)
************* Module idigbio_ingestion.lib.delimited
W: 97,20: dict.items referenced when not iterating (dict-items-not-iterating)
W:122, 4: next method defined (next-method-defined)
************* Module idigbio_ingestion.lib.eml
W:141,15: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.lib.dwca
E:101,28: print statement used (print-statement)
************* Module idigbio_ingestion.lib.fileproxy
W: 60, 4: next method defined (next-method-defined)
W: 61,12: Called a next() method on an object (next-method-called)
E:120,12: print statement used (print-statement)

-----------------------------------
Your code has been rated at 9.76/10
danstoner commented 2 years ago

The --py3k option generates false positives since it must run under Python 2.

Disable checks that are fixed and ready for Python3 at this point in the convert_python_3 branch.

This is the "final" list of issues needed to make idigbio_ingestion compatible with Python 3:

$ pylint --py3k --disable=no-absolute-import,print-statement,exception-message-attribute   idigbio_ingestion
No config file found, using default configuration
************* Module idigbio_ingestion.mediaing.fetcher
W: 48,12: dict.values referenced when not iterating (dict-values-not-iterating)
W:115,16: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.mediaing.derivatives
W: 87,39: filter built-in referenced when not iterating (filter-builtin-not-iterating)
W:180,16: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module idigbio_ingestion.lib.delimited
W: 97,20: dict.items referenced when not iterating (dict-items-not-iterating)
W:122, 4: next method defined (next-method-defined)
************* Module idigbio_ingestion.lib.eml
W:141,15: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module idigbio_ingestion.lib.fileproxy
W: 60, 4: next method defined (next-method-defined)
W: 61,12: Called a next() method on an object (next-method-called)

------------------------------------------------------------------
Your code has been rated at 9.95/10 (previous run: 9.95/10, +0.01)
danstoner commented 2 years ago

Tried to run pylint under python3 and search the output for just those issues found above. Strangely, these have extra error in parens, not sure how that relates and the codes like E0011 don't seem to match the expected lint error. I can no longer find the website that I recall having good information on these.

And python3 linter does not actually find the same issues The two items for mediaing.fetcher are in the pylint output but not filter-builtin-not-iterating, dict-items-not-iterating, and the others.

$ pylint --version
pylint 2.12.2
astroid 2.9.3
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0]

$ pylint --output=pylint3.txt idigbio_ingestion

These are the ones that match the --py3k list:

idigbio_ingestion/mediaing/fetcher.py:48:0: E0011: Unrecognized file option 'dict-values-not-iterating' (unrecognized-inline-option)

idigbio_ingestion/mediaing/fetcher.py:115:0: E0011: Unrecognized file option 'map-builtin-not-iterating' (unrecognized-inline-option)
danstoner commented 1 year ago

Errors only:

(venv3) (convert_python_3) dan@dan-Latitude-7370:~/git/IDIGBIO/idb-backend$ pylint --errors-only --output=E.pylint3.20230209.txt idigbio_ingestion
(venv3) (convert_python_3) dan@dan-Latitude-7370:~/git/IDIGBIO/idb-backend$ cat E.pylint3.20230209.txt 
************* Module idigbio_ingestion.db_rsids
idigbio_ingestion/db_rsids.py:43:12: E1101: Class 'traceback' has no 'print_exc' member (no-member)
idigbio_ingestion/db_rsids.py:43:12: E0602: Undefined variable 'traceback' (undefined-variable)
************* Module idigbio_ingestion.lib.fileproxy
idigbio_ingestion/lib/fileproxy.py:91:4: E0301: __iter__ returns non-iterator (non-iterator-returned)
************* Module idigbio_ingestion.lib.xmlDictTools
idigbio_ingestion/lib/xmlDictTools.py:132:4: E0401: Unable to import 'StringIO' (import-error)
************* Module idigbio_ingestion.mediaing.fetcher
idigbio_ingestion/mediaing/fetcher.py:48:0: E0011: Unrecognized file option 'dict-values-not-iterating' (unrecognized-inline-option)
idigbio_ingestion/mediaing/fetcher.py:115:0: E0011: Unrecognized file option 'map-builtin-not-iterating' (unrecognized-inline-option)
************* Module idigbio_ingestion.mediaing.cli
idigbio_ingestion/mediaing/cli.py:18:8: E1101: Module 'click' has no 'abort' member; maybe 'Abort'? (no-member)