Closed cclauss closed 6 years ago
@apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage.
For general "how-to" questions, our user forum (and Chinese version) is a good place to get help.
Sorry could you clarify the issue and give a bit of context?
These are variables that are not defined in the current context which can raise NameError at runtime. Often they are typos or missing imports or code refactoring errors or things like basestring or xrange() being defined in Python 2 but being removed in Python 3.
Just try reading thru the code as if you were the Python interpreter and ask yourself "Where is this variable defined?".
Oh I understand now, that's some static code analysis right? Thanks for running that, seems like some files need fixing.
Are there any specific advantages that flake8 has over pylint? pylint has equivalent messages such as 'undefined-variable' (equivalent to F821). So just wondering if it would be easier to add more options to pylintrc and include in the existing build process.
flake8 will find undefined names that pylint will not because flake8 makes an abstract syntax tree for each Python file and plyint does not. This also explains why flake8 gives different results in Python 2 and Python 3 (the have different ASTs while pylint does not.
Take running code like isinstance(unicode('Hello'), basestring) through pylint on Python 2 and Python 3 and see if there are differences in the output. Now try same with flake8.
I take it back... You are right.
PyLint is now able to find undefined names.
$ python3 -m pylint *.py
************* Module pylint_test
pylint_test.py:1:0: C0111: Missing module docstring (missing-docstring)
pylint_test.py:1:11: E0602: Undefined variable 'unicode' (undefined-variable)
pylint_test.py:1:29: E0602: Undefined variable 'basestring' (undefined-variable)
flake8 testing of https://github.com/apache/incubator-mxnet on Python 3.7.0
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
/home/travis/virtualenv/python3.7.0/lib/python3.7/site-packages/pycodestyle.py:113: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
./example/reinforcement-learning/a3c/a3c.py:206:16: F821 undefined name 'robo_data'
dataiter = robo_data.RobosimsDataIter('scenes', args.batch_size, args.input_length, web_viz=True)
^
./example/ssd/dataset/pycocotools/coco.py:266:41: F821 undefined name 'm'
img = np.ones( (m.shape[0], m.shape[1], 3) )
^
./example/ssd/dataset/pycocotools/coco.py:266:53: F821 undefined name 'm'
img = np.ones( (m.shape[0], m.shape[1], 3) )
^
./example/ssd/dataset/pycocotools/coco.py:273:52: F821 undefined name 'm'
ax.imshow(np.dstack( (img, m*0.5) ))
^
./example/ssd/dataset/pycocotools/coco.py:435:16: F821 undefined name 'm'
return m
^
./example/ssd/symbol/common.py:209:23: F821 undefined name 'start_offset'
min_sizes = [start_offset] + tmp.tolist()
^
./example/ssd/symbol/common.py:210:46: F821 undefined name 'start_offset'
max_sizes = tmp.tolist() + [tmp[-1]+start_offset]
^
./example/fcn-xs/image_segmentaion.py:100:60: F821 undefined name 'ctx'
fcnxs_args["data"] = mx.nd.array(get_data(args.input), ctx)
^
./example/fcn-xs/image_segmentaion.py:103:60: F821 undefined name 'ctx'
fcnxs_args["softmax_label"] = mx.nd.empty(label_shape, ctx)
^
./example/fcn-xs/image_segmentaion.py:104:26: F821 undefined name 'ctx'
exector = fcnxs.bind(ctx, fcnxs_args, args_grad=None, grad_req="null", aux_states=fcnxs_args)
^
./example/neural-style/end_to_end/model_vgg19.py:94:17: F821 undefined name 'out'
arg_names = out.list_arguments()
^
./example/neural-style/end_to_end/model_vgg19.py:105:16: F821 undefined name 'out'
executor = out.bind(ctx=ctx, args=arg_dict, args_grad=[], grad_req="null")
^
./example/sparse/factorization_machine/metric.py:111:22: F821 undefined name 'label_zero_num'
total_area = label_zero_num * label_one_num
^
./example/sparse/factorization_machine/metric.py:111:39: F821 undefined name 'label_one_num'
total_area = label_zero_num * label_one_num
^
./example/profiler/profiler_executor.py:89:15: F821 undefined name 'search_plan'
sym = search_plan(sym, data=data_shapes)
^
./example/profiler/profiler_executor.py:89:37: F821 undefined name 'data_shapes'
sym = search_plan(sym, data=data_shapes)
^
./python/mxnet/initializer.py:700:20: F821 undefined name '_INITIALIZER_REGISTRY'
init = _INITIALIZER_REGISTRY[klass.lower()](**kwargs)
^
./python/mxnet/optimizer.py:718:34: F821 undefined name 'array'
weight_master_copy = array(weight, ctx=weight.context, dtype=numpy.float32)
^
./python/mxnet/optimizer.py:769:16: F821 undefined name 'multiply'
norm = multiply(v, v).asnumpy().sum()
^
./tests/python/unittest/test_engine_import.py:33:13: F821 undefined name 'reload'
reload(mxnet)
^
./tests/nightly/model_backwards_compatibility_check/common.py:214:12: F821 undefined name 'cmp'
return cmp(normalize(version1), normalize(version2))
^
./docs/mxdoc.py:75:16: F821 undefined name 'root_path'
pdf_path = root_path + '/docs/api/r/mxnet-r-reference-manual.pdf'
^
22 F821 undefined name 'root_path'
22
@cclauss I'm working on fixing these errors. Will add you to the PRs as I submit them.
Pylint reports undefined variables when there is a wildcard import. The list from Pylint is at https://github.com/apache/incubator-mxnet/issues/11904
flake8 (on Python 3.7.0) moved from finding 22 undefined names down to 19.
flake8 testing of https://github.com/apache/incubator-mxnet on Python 3.7.0
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
./example/reinforcement-learning/a3c/a3c.py:206:16: F821 undefined name 'robo_data'
dataiter = robo_data.RobosimsDataIter('scenes', args.batch_size, args.input_length, web_viz=True)
^
./example/ssd/dataset/pycocotools/coco.py:266:41: F821 undefined name 'm'
img = np.ones( (m.shape[0], m.shape[1], 3) )
^
./example/ssd/dataset/pycocotools/coco.py:266:53: F821 undefined name 'm'
img = np.ones( (m.shape[0], m.shape[1], 3) )
^
./example/ssd/dataset/pycocotools/coco.py:273:52: F821 undefined name 'm'
ax.imshow(np.dstack( (img, m*0.5) ))
^
./example/ssd/dataset/pycocotools/coco.py:435:16: F821 undefined name 'm'
return m
^
./example/ssd/symbol/common.py:209:23: F821 undefined name 'start_offset'
min_sizes = [start_offset] + tmp.tolist()
^
./example/ssd/symbol/common.py:210:46: F821 undefined name 'start_offset'
max_sizes = tmp.tolist() + [tmp[-1]+start_offset]
^
./example/fcn-xs/image_segmentaion.py:100:60: F821 undefined name 'ctx'
fcnxs_args["data"] = mx.nd.array(get_data(args.input), ctx)
^
./example/fcn-xs/image_segmentaion.py:103:60: F821 undefined name 'ctx'
fcnxs_args["softmax_label"] = mx.nd.empty(label_shape, ctx)
^
./example/fcn-xs/image_segmentaion.py:104:26: F821 undefined name 'ctx'
exector = fcnxs.bind(ctx, fcnxs_args, args_grad=None, grad_req="null", aux_states=fcnxs_args)
^
./example/neural-style/end_to_end/model_vgg19.py:94:17: F821 undefined name 'out'
arg_names = out.list_arguments()
^
./example/neural-style/end_to_end/model_vgg19.py:105:16: F821 undefined name 'out'
executor = out.bind(ctx=ctx, args=arg_dict, args_grad=[], grad_req="null")
^
./example/sparse/factorization_machine/metric.py:111:22: F821 undefined name 'label_zero_num'
total_area = label_zero_num * label_one_num
^
./example/sparse/factorization_machine/metric.py:111:39: F821 undefined name 'label_one_num'
total_area = label_zero_num * label_one_num
^
./example/profiler/profiler_executor.py:89:15: F821 undefined name 'search_plan'
sym = search_plan(sym, data=data_shapes)
^
./example/profiler/profiler_executor.py:89:37: F821 undefined name 'data_shapes'
sym = search_plan(sym, data=data_shapes)
^
./tests/python/unittest/test_engine_import.py:33:13: F821 undefined name 'reload'
reload(mxnet)
^
./tests/nightly/model_backwards_compatibility_check/common.py:216:12: F821 undefined name 'cmp'
return cmp(normalize(version1), normalize(version2))
^
./docs/mxdoc.py:75:16: F821 undefined name 'root_path'
pdf_path = root_path + '/docs/api/r/mxnet-r-reference-manual.pdf'
^
19 F821 undefined name 'root_path'
19
With the merge of #11982, flake8 (on Python 3) is now down to 10 undefined names:
flake8 testing of https://github.com/apache/incubator-mxnet on Python 3.7.0
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
./example/reinforcement-learning/a3c/a3c.py:206:16: F821 undefined name 'robo_data'
dataiter = robo_data.RobosimsDataIter('scenes', args.batch_size, args.input_length, web_viz=True)
^
./example/neural-style/end_to_end/model_vgg19.py:94:17: F821 undefined name 'out'
arg_names = out.list_arguments()
^
./example/neural-style/end_to_end/model_vgg19.py:105:16: F821 undefined name 'out'
executor = out.bind(ctx=ctx, args=arg_dict, args_grad=[], grad_req="null")
^
./example/sparse/factorization_machine/metric.py:111:22: F821 undefined name 'label_zero_num'
total_area = label_zero_num * label_one_num
^
./example/sparse/factorization_machine/metric.py:111:39: F821 undefined name 'label_one_num'
total_area = label_zero_num * label_one_num
^
./example/profiler/profiler_executor.py:89:15: F821 undefined name 'search_plan'
sym = search_plan(sym, data=data_shapes)
^
./example/profiler/profiler_executor.py:89:37: F821 undefined name 'data_shapes'
sym = search_plan(sym, data=data_shapes)
^
./tests/python/unittest/test_engine_import.py:33:13: F821 undefined name 'reload'
reload(mxnet)
^
./tests/nightly/model_backwards_compatibility_check/common.py:216:12: F821 undefined name 'cmp'
return cmp(normalize(version1), normalize(version2))
^
./docs/mxdoc.py:75:16: F821 undefined name 'root_path'
pdf_path = root_path + '/docs/api/r/mxnet-r-reference-manual.pdf'
^
10 F821 undefined name 'root_path'
10
./python/mxnet/base.py:735:12: F821 undefined name 'C' return C.c_int(init_val) ^ ./python/mxnet/base.py:739:14: F821 undefined name 'C' x_addr = C.addressof(x) ^ ./python/mxnet/base.py:740:13: F821 undefined name 'C' int_p = C.POINTER(C.c_int) ^ ./python/mxnet/base.py:740:23: F821 undefined name 'C' int_p = C.POINTER(C.c_int) ^ ./python/mxnet/base.py:741:18: F821 undefined name 'C' x_int_addr = C.cast(x_addr, int_p) ^
@mkolod @KellenSunderland These have been introduced by the TensorRT commit. Could you have a look at these?
@vandanavk Taking a look.
@vandanavk Thanks for the catch! Should be addressed in https://github.com/apache/incubator-mxnet/pull/12147
Thanks @KellenSunderland
@cclauss I don't see the errors
./tests/python/unittest/test_engine_import.py:33:13: F821 undefined name 'reload'
reload(mxnet)
^
./tests/nightly/model_backwards_compatibility_check/common.py:216:12: F821 undefined name 'cmp'
return cmp(normalize(version1), normalize(version2))
I've tried flake8 and pylint, Python2 and Python3.
Although I do see
************* Module mxnet.ndarray
python/mxnet/ndarray/__init__.py(34): [E0602 ] Undefined variable 'ndarray'
************* Module mxnet.symbol
python/mxnet/symbol/__init__.py(31): [E0602 ] Undefined variable 'symbol'
After these are fixed, I'll add "undefined-variable" to pylintrc in the CI build. This will check future commits to python/mxnet.
See PEP8 for reasons why star imports are for the birds...
On the python/mxnet/ndarray/__init__.py
issue, you could try adding a line from .ndarray import __all__ as ndarray___all__
and then below replace ndarray.__all__
with ndarray___all__
.
flake8 testing of https://github.com/apache/incubator-mxnet on Python 3.7.0
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
./amalgamation/python/mxnet_predict.py:68:40: F821 undefined name 'libinfo_py'
libinfo = {'__file__': libinfo_py}
^
./amalgamation/python/mxnet_predict.py:69:35: F821 undefined name 'libinfo_py'
exec(compile(open(libinfo_py, "rb").read(), libinfo_py, 'exec'), libinfo, libinfo)
^
./amalgamation/python/mxnet_predict.py:69:61: F821 undefined name 'libinfo_py'
exec(compile(open(libinfo_py, "rb").read(), libinfo_py, 'exec'), libinfo, libinfo)
^
./tests/python/unittest/test_engine_import.py:33:13: F821 undefined name 'reload'
reload(mxnet)
^
./tests/nightly/model_backwards_compatibility_check/common.py:216:12: F821 undefined name 'cmp'
return cmp(normalize(version1), normalize(version2))
^
5 F821 undefined name 'libinfo_py'
6
@hqucms Could you have a look at these https://github.com/apache/incubator-mxnet/commit/f3070fbe04b94e9715ef950cb33ce3402be5c077#r30092025?
@vandanavk Thank you for spotting this! I made a fix in https://github.com/apache/incubator-mxnet/pull/12211.
@cclauss looks like all the PRs for fixing these errors are merged. Do you think this issue can be closed now?
https://github.com/apache/incubator-mxnet/pull/12295 is this the last open PR related to this issue @cclauss ?
Yes.
Undefined names can raise NameError at runtime.
flake8 testing of https://github.com/apache/incubator-mxnet on Python 2.7.14
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics