jquast
commented
9 years ago @hschilling use the prospector argument --die-on-tool-error and it may help discover the root issue
Closed hschilling closed 9 years ago
jquast
commented
9 years ago @hschilling use the prospector argument --die-on-tool-error and it may help discover the root issue
hschilling
commented
9 years ago Hi Jeff,
Thanks for the suggestion. Maybe our project size is stretching the limits ? Thanks for the help
Traceback (most recent call last):
File "/usr/local/bin/prospector", line 9, in
On Dec 4, 2014, at 6:42 PM, Jeff Quast notifications@github.com<mailto:notifications@github.com> wrote:
@hschillinghttps://github.com/hschilling use the prospector argument --die-on-tool-error and it may help discover the root issue
— Reply to this email directly or view it on GitHubhttps://github.com/landscapeio/prospector/issues/73#issuecomment-65724482.
Herb Schilling Scientific Applications and Visualization Team http://ocio.grc.nasa.gov/gvis Office of the Chief Information Officer NASA Glenn Research Center hschilling@nasa.govmailto:hschilling@nasa.gov
jquast
commented
9 years ago Suggest this issue be closed, prospector can't help you much here.
However I filed a bug on your behalf with astroid: https://bitbucket.org/logilab/astroid/issue/65/oserror-errno-24-too-many-open-files you may watch this issue for any resolution or response.
In the meantime, I suggest finding a way to increase the number of open files for your user account.
hschilling
commented
9 years ago Hi,
I see. Thanks for filing that bug for me.
I did try ( I am using Mac OS X 10.8.5 )
sudo launchctl limit maxfiles 1000000 1000000
and got the same error.
I monitored the process using Activity Monitor's Open Files pane and I saw that it just kept growing as it opened up source files but it sure didn't look like 1000000 files in there !
On Dec 6, 2014, at 6:34 PM, Jeff Quast notifications@github.com<mailto:notifications@github.com> wrote:
Suggest this issue be closed, prospector can't help you much here.
However I filed a bug on your behalf with astroid: https://bitbucket.org/logilab/astroid/issue/65/oserror-errno-24-too-many-open-files you may watch this issue for any resolution or response.
In the meantime, I suggest finding a way to increase the number of open files for your user account.
— Reply to this email directly or view it on GitHubhttps://github.com/landscapeio/prospector/issues/73#issuecomment-65919466.
Herb Schilling Scientific Applications and Visualization Team http://ocio.grc.nasa.gov/gvis Office of the Chief Information Officer NASA Glenn Research Center hschilling@nasa.govmailto:hschilling@nasa.gov
jquast
commented
9 years ago That doesn't work for some reason. Just type:
$ ulimit -n 16384
on your shell. 1000000 is also too high, 16384 is the actual upperbound limit?
jquast
commented
9 years ago This was fixed upstream, Pylint 1.4.1 should resolve the issue.
prospector is currently bound to pylint,<1.4 but I understand this should be resolved once prospector drops python2.6 support and removes this upperbound version. https://bitbucket.org/logilab/astroid/issue/65/oserror-errno-24-too-many-open-files
jquast
commented
9 years ago If you are interested, I have a cheap workaround for installing and running prospector using tox, removing the upperbound, and upgrading pylint and astroid:
tox.ini excerpt:
[testenv:prospector]
usedevelop = True
deps = -r{toxinidir}/requirements-analysis.txt
whitelist = pip gsed prospector
# cheap workaround until prospector's next release
# https://github.com/landscapeio/prospector/issues/75
commands = pip install --upgrade astroid pylint
gsed -i 's|pylint>=1.3,<1.4|pylint>=1.3|g' {envsitepackagesdir}/prospector-0.7.4-py2.7.egg-info/requires.txt
prospector \
--die-on-tool-error \
--test-warnings \
--doc-warnings \
{toxinidir}requirements-analysis.txt
prospector[with_dodgy,with_frosted,with_mccabe,with_pep257,with_pep8,with_pylint,with_pyroma,with_vulture]Hi Jeff,
Thanks for this and also for letting me know about the fix to Pylint !
On Dec 15, 2014, at 11:53 AM, Jeff Quast notifications@github.com<mailto:notifications@github.com> wrote:
If you are interested, I have a cheap workaround for installing and running prospector using tox, removing the upperbound, and upgrading pylint and astroid:
tox.ini excerpt:
[testenv:prospector] usedevelop = True deps = -r{toxinidir}/requirements-analysis.txt whitelist = pip gsed prospector
commands = pip install --upgrade astroid pylint gsed -i 's|pylint>=1.3,<1.4|pylint>=1.3|g' {envsitepackagesdir}/prospector-0.7.4-py2.7.egg-info/requires.txt prospector \ --die-on-tool-error \ --test-warnings \ --doc-warnings \ {toxinidir}
requirements-analysis.txt
prospector[with_dodgy,with_frosted,with_mccabe,with_pep257,with_pep8,with_pylint,with_pyroma,with_vulture]
— Reply to this email directly or view it on GitHubhttps://github.com/landscapeio/prospector/issues/73#issuecomment-67025396.
Herb Schilling Scientific Applications and Visualization Team http://ocio.grc.nasa.gov/gvis Office of the Chief Information Officer NASA Glenn Research Center hschilling@nasa.govmailto:hschilling@nasa.gov
carlio
commented
9 years ago Finally got around to making a 0.8 release, sorry about the delay. #10 and #74 turned out to be a bit more fiddly than expected.
Let me know if this fixes these issues - it should do as pylint>=1.4 is now the minimum requirement.
@jquast by the way you can use pip install prospector[with_everything] to avoid having to specify each dependency!
carlio
commented
9 years ago @hschilling if you upgrade to prospector 0.8, this should hopefully fix your issue.
hschilling
commented
9 years ago Thanks ! Will give it a try over the weekend
hschilling
commented
9 years ago I tried it. I did install everything using:
pip install prospector[with_everything]
When I just do
prospector
in a directory with a super simple Python script, it works.
If I do the same in our OpenMDAO project directory, I get this.
Traceback (most recent call last):
File "/Users/herbertschilling/Documents/OpenMDAO/dev/latest/devenv/bin/prospector", line 9, in
The value that gets returned from autodetect_libraries(self.workdir) is:
set([u'django'])Let me know if I should try something else.
Sorry!
jquast
commented
9 years ago I'm submitting a PR for this shortly, this broke for me too
jquast
commented
9 years ago submitted new issue #82 for tracking
jquast
commented
9 years ago Fixed in PR #85
carlio
commented
9 years ago Merged #85 and released 0.8.2 with this bugfix, thanks @jquast !
hschilling
commented
9 years ago Thanks! I tried it again and things are working much better. The only problem left is I see this in the output at the top of the output file. I ran this with --with-tool pyroma --with-tool vulture. There is nothing in the output that tells me more about the exception. I can seem to run pylint and pyroma without errors
. Line: None pylint: failure / Tool pylint failed to run (exception was raised) pyroma: failure / Tool pyroma failed to run (exception was raised)
jquast
commented
9 years ago Don't forget, use argument --die-on-tool-error to see the exception that was raised.
carlio
commented
9 years ago @hschilling I tried this myself on the OpenMDAO-Framework repository and pylint worked fine, so if you could use --die-on-tool-error to show the stack trace that would be great.
Pyroma fails because one of the setup.py files in the project imports NumPY which wasn't available for me, so I created #93 to address that. It may not be the same as your issue though if you have numpy installed.
hschilling
commented
9 years ago Sorry, I DID forget about --die-on-tool-error :-)
Pylinit dies with "RuntimeError: maximum recursion depth exceeded" Here is most of the stack trace. I cut out the part where it has a LOT of these lines repeated over and over again.
File "/Users/herbertschilling/Documents/OpenMDAO/dev/latest/devenv/lib/python2.7/site-packages/astroid/bases.py", line 341, in wrapped for res in _func(node, context, _kwargs): File "/Users/herbertschilling/Documents/OpenMDAO/dev/latest/devenv/lib/python2.7/site-packages/astroid/bases.py", line 138, in _infer_stmts for infered in stmt.infer(context, _kw): File "/Users/herbertschilling/Documents/OpenMDAO/dev/latest/devenv/lib/python2.7/site-packages/astroid/bases.py", line 110, in cache_generator for result in generator:
Traceback (most recent call last):
File "/Users/herbertschilling/Documents/OpenMDAO/dev/latest/devenv/bin/prospector", line 9, in
hschilling
commented
9 years ago BTW, the usage: output for prospector looks like it could be tweaked. Missing --die-on-tool-error and the alternate ways of using --with-tool,etc...
usage: prospector [-h] [-0] [-A] [-u USES] [-B] [--no-common-plugin] [-D] [-T] [-8] [-F] [--max-line-length MAX_LINE_LENGTH] [-M] [-S] [-o {emacs,grouped,json,pylint,text,yaml}] [--absolute-paths] [-t TOOLS] [-w WITH_TOOLS] [-W WITHOUT_TOOLS] [-P PROFILES] [--profile-path PROFILE_PATH] [-s {veryhigh,high,medium,low,verylow}] [-E] [--pylint-config-file PYLINT_CONFIG_FILE] [-p PATH] [-I IGNORE_PATTERNS] [-i IGNORE_PATHS] [-X] [--loquacious-pylint] [--version] [PATH [PATH ...]]
hschilling
commented
9 years ago I do have numpy installed. This is why pyroma dies for me. Looks like you just need to use
if '_setuptools' in data:
instead of
if data['_setuptools']:
In fact, if I make that code change, everything runs for me if I --without-tool pylint
Traceback (most recent call last):
File "/Users/herbertschilling/Documents/OpenMDAO/dev/latest/devenv/bin/prospector", line 9, in
and data is equal to
(Pdb) data {'ext_modules': [<numpy.distutils.extension.Extension instance at 0x106f48290>], 'description': 'AXOD - axial turbine off-design analysis.', 'license': 'public domain', 'distclass': <class numpy.distutils.numpy_distribution.NumpyDistribution at 0x106ab08d8>, 'install_requires': ['numpy'], 'zip_safe': False, '_setuptools': True, 'headers': [], 'version': '0.1.0', 'cmdclass': {'bdist_rpm': <class numpy.distutils.command.bdist_rpm.bdist_rpm at 0x106e0ef58>, 'sdist': <class numpy.distutils.command.sdist.sdist at 0x106e0e328>, 'build_src': <class numpy.distutils.command.build_src.build_src at 0x106e0e8d8>, 'config_fc': <class numpy.distutils.command.config_compiler.config_fc at 0x106ac0db8>, 'build_clib': <class numpy.distutils.command.build_clib.build_clib at 0x106e0e258>, 'build_scripts': <class numpy.distutils.command.build_scripts.build_scripts at 0x106e0e2c0>, 'build_ext': <class numpy.distutils.command.build_ext.build_ext at 0x106df7050>, 'develop': <class numpy.distutils.command.develop.develop at 0x106e23f58>, 'build_py': <class numpy.distutils.command.build_py.build_py at 0x106add390>, 'scons': <class numpy.distutils.command.scons.scons at 0x106e23c18>, 'install_clib': <class numpy.distutils.command.install_clib.install_clib at 0x106e0ec80>, 'egg_info': <class numpy.distutils.command.egg_info.egg_info at 0x106f46050>, 'install_headers': <class numpy.distutils.command.install_headers.install_headers at 0x106e0ec18>, 'install_data': <class numpy.distutils.command.install_data.install_data at 0x106e0ea10>, 'easy_install': <class setuptools.command.easy_install.easy_install at 0x106aa9c18>, 'bdist_egg': <class setuptools.command.bdist_egg.bdist_egg at 0x106aa9738>, 'install': <class numpy.distutils.command.install.install at 0x106e0ece8>, 'config_cc': <class numpy.distutils.command.config_compiler.config_cc at 0x106ac0e20>, 'config': <class numpy.distutils.command.config.config at 0x106ac0d50>, 'build': <class numpy.distutils.command.build.build at 0x106ac0f58>}, 'data_files': [('', ['AXOD_AAO.pdf', 'axod.pdf', 'README.txt']), ('src/', ['src/cpw.f', 'src/tempdata.f', 'src/Makefile', 'src/pratio.f', 'src/axodm.f', 'src/phim.f', 'src/besfit.f', 'src/input.f', 'src/ploter.f', 'src/flow2.f', 'src/sta1a.f', 'src/sta2.f', 'src/sta2a.f', 'src/gama.f', 'src/icommon.f', 'src/fit1.f', 'src/sta01.f', 'src/r.f', 'src/loss2.f', 'src/cpf.f', 'src/read_input.txt', 'src/inst2.f', 'src/etacf.f', 'src/ovrall.f', 'src/init.f', 'src/instg.f', 'src/cpa.f', 'src/loss1.f', 'src/axod.f', 'src/getdata.f', 'src/sta1.f', 'src/flow1.f', 'src/etapr.f', 'src/diagt.f', 'src/testaxod.py', 'src/loop.f', 'src/ReadMe.txt', 'src/fit2.f']), ('test/', ['test/one_stage.inp', 'test/axod_compn.py', 'test/test_axod_comp.py', 'test/axod.out2', 'test/eee_hpt.inp', 'test/datain.py', 'test/testaxodn.py', 'test/newtest1.py', 'test/testaxod.py', 'test/axod.out1', 'test/one_stageO.inp', 'test/read_input.txt', 'test/eee_hpt.inpO', 'test/one_stage.inpO', 'test/axod_comp.py'])], 'name': 'axod'}
carlio
commented
9 years ago @hschilling I've seen the recursion depth thing before but I think in an older version of astroid. Can you do a 'pip freeze' and show the versions of astroid, logilab-common and pylint?
For pyroma, it's the same stacktrace as the one I got, but the problem is in pyroma itself, so I'll need to make a PR on that.
hschilling
commented
9 years ago @carlio Thanks. Here is the output of pip freeze:
Cython==0.20.1 Fabric==0.9.3 Jinja2==2.4 PyYAML==3.11 Pyevolve==0.6 Pygments==1.3.1 SetupDocs==1.0.5 Sphinx==1.2.2 argparse==1.3.0 astroid==1.3.2 backports.ssl-match-hostname==3.4.0.2 boto==2.0rc1 bson==0.3.3 cobyla==1.0.1 conmin==1.0.1 decorator==3.2.0 docutils==0.12 dodgy==0.1.6 enum34==1.0.4 frosted==1.4.1 ipaddress==1.0.7 ipython==2.0.0 logilab-common==0.63.2 matplotlib==1.3.1 mccabe==0.3 mock==1.0.1 networkx==1.8.1 newsumt==1.1.0 nose==1.3.3 numpy==1.7.1 objgraph==1.8.0 openmdao.devtools==0.10.3.2 openmdao.examples.bar3simulation==0.10.3.2 openmdao.examples.expected-improvement==0.10.3.2 openmdao.examples.mdao==0.10.3.2 openmdao.examples.metamodel-tutorial==0.10.3.2 openmdao.examples.nozzle-geometry-doe==0.10.3.2 openmdao.examples.simple==0.10.3.2 openmdao.lib==0.10.3.2 openmdao.main==0.10.3.2 openmdao.test==0.10.3.2 openmdao.units==0.10.3.2 openmdao.util==0.10.3.2 ordereddict==1.1 paramiko==1.7.7.1 pep257==0.3.2 pep8==1.5.7 pep8-naming==0.2.2 pies==2.6.3 pies2overrides==2.6.3 prospector==0.8.2 pycrypto==2.3 pyflakes==0.8.1 pylint==1.4.0 pylint-celery==0.3 pylint-common==0.2.1 pylint-django==0.5.5 pylint-plugin-utils==0.2.2 pyparsing==1.5.7 pyroma==1.7 python-dateutil==2.2 pytz==2011k pyzmq==14.1.1 readline==6.2.4.1 requests==2.2.1 requirements-detector==0.3 scipy==0.12.0 see==1.0.1 setoptconf==0.2.0 six==1.9.0 slsqp==1.0.1 stevedore==0.10 tornado==3.2 traits==4.3.0 virtualenv==1.9.1 virtualenv-clone==0.2.4 virtualenvwrapper==4.1.1 vulture==0.6 wsgiref==0.1.2 xdot==0.5 xtrace==0.4 zope.interface==3.6.1
carlio
commented
9 years ago @hschilling thanks, unfortunately those are exactly the versions I was using so that's not the issue.
I tried it again on my Mac this time, and I get a similar problem - I don't get a stacktrace but prospector is taking forever and when I Ctrl-C I see a very deep stacktrace at the same point you did. This doesn't happen on my Ubuntu machine. Very odd.
Out of interest, how long does it take to get the stacktrace?
hschilling
commented
9 years ago 47 seconds
carlio
commented
9 years ago It just finished for me, and no stack trace or problem. I'm 100% baffled why it's not working for you.
carlio
commented
9 years ago Wait, ignore that! Of course, if I don't have all the dependencies installed, then the pylint inference will be less intense. I'll try installing everything and see what happens.
carlio
commented
9 years ago I installed as much as I could, but some things were not available - for example, Traits 4.3.0 is no longer available for download.
I think in general though that the problem is simply 'too much code'. Pylint builds up a map of all code in the project but also follows inference chains for libraries used, so there will be so much going on.
This is basically a limitation of pylint/astroid rather than a bug in prospector. I'm not sure exactly what could be done either, without severe restructuring to avoid recursing when inferring types.
The only solution I have would be to run prospector several times on different parts of the code.
hschilling
commented
9 years ago Makes sense to me. BTW, we are trying to get rid of Traits as a dependency. Good to know you can't even download it any more!
jquast
commented
9 years ago (i have a love/hate relationship with traits too)
the reason it doesn't find it is because enthought does a very poor job of publishing it, this is what pip sees: https://pypi.python.org/simple/traits/ -- as you see, 4.3.0 just redirects to a website where no download is found, you can probably safely upgrade to latest traits.
carlio
commented
9 years ago I'll close this as there's not much I can do to fix it and #93 covers what can be fixed.
hschilling
commented
9 years ago That's fine. Thanks for the help !
On Jan 10, 2015, at 11:16 PM, Carl Crowder notifications@github.com<mailto:notifications@github.com> wrote:
I'll close this as there's not much I can do to fix it and #93https://github.com/landscapeio/prospector/issues/93 covers what can be fixed.
— Reply to this email directly or view it on GitHubhttps://github.com/landscapeio/prospector/issues/73#issuecomment-69483078.
Herb Schilling Scientific Applications and Visualization Team http://ocio.grc.nasa.gov/gvis Office of the Chief Information Officer NASA Glenn Research Center hschilling@nasa.govmailto:hschilling@nasa.gov
For a very large code base ( https://github.com/OpenMDAO/OpenMDAO-Framework ) , I get this in the output
.: L-:- None: pylint - failure Tool pylint failed to run (exception was raised)
.: L-:- None: vulture - failure Tool vulture failed to run (exception was raised)
There are no errors given by either tool as a result.
I tried using pylint and vulture with some smaller codes and it worked fine.
Great tool though!!!