search

jameslamb / doppel-cli

Test framework for comparing the consistency of library APIs
https://doppel-cli.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
8 stars 12 forks source link

doppel-describe is broken for "networkx" #78

Closed jameslamb closed 5 years ago

jameslamb commented 5 years ago

analyze.py is broken in some way that causes it to fail on the networkx library.

doppel-describe -p networkx --language python --data-dir $(pwd)/test_data

yields:

Testing package networkx [python]
/Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py
Describing package with command:
 /Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py --pkg networkx --output_dir /Users/jlamb/repos/doppel-cli/test_data --kwargs-string ~~KWARGS~~ --constructor-string ~~CONSTRUCTOR~~
AmbiguousSolution is an Exception. Skipping.
'DiGraph' is a class in this package, adding it
Traceback (most recent call last):
  File "/Users/jlamb/anaconda3/lib/python3.6/inspect.py", line 1119, in getfullargspec
    sigcls=Signature)
  File "/Users/jlamb/anaconda3/lib/python3.6/inspect.py", line 2342, in _signature_from_callable
    'no signature found for builtin type {!r}'.format(obj))
ValueError: no signature found for builtin type <class 'dict'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py", line 132, in <module>
    method_args = _get_arg_names(func, KWARGS_STRING)
  File "/Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py", line 68, in _get_arg_names
    f_dict = inspect.getfullargspec(f)._asdict()
  File "/Users/jlamb/anaconda3/lib/python3.6/inspect.py", line 1125, in getfullargspec
    raise TypeError('unsupported callable') from ex
TypeError: unsupported callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py", line 135, in <module>
    func.__wrapped__,
AttributeError: type object 'dict' has no attribute '__wrapped__'
Session Info

aiohttp==3.3.2 alabaster==0.7.10 anaconda-client==1.6.14 anaconda-navigator==1.8.2 anaconda-project==0.8.2 AnyQt==0.0.8 appnope==0.1.0 appscript==1.0.1 argcomplete==1.8.2 argh==0.26.2 asn1crypto==0.24.0 astroid==1.6.3 astropy==3.0.1 async-timeout==3.0.0 asyncio==3.4.3 atomicwrites==1.3.0 attrs==17.4.0 auto-ml==2.9.10 autofocus==0.1.0 avro==1.8.2 avro-python3==1.8.2 awscli==1.15.7 Babel==2.5.3 backcall==0.1.0 backoff==1.6.0 backports.shutil-get-terminal-size==1.0.0 basemap==1.1.0 beartype==0.0.3 beautifulsoup4==4.6.0 biopython==1.72 bitarray==0.8.1 blaze==0.11.3 bleach==2.1.3 blinker==1.4 bokeh==0.12.15 boto==2.48.0 boto3==1.9.59 botocore==1.12.59 Bottleneck==1.2.1 browsercookie==0.7.5 bz2file==0.98 cattrs==0.9.0 certifi==2018.11.29 cffi==1.11.5 chardet==3.0.4 chest==0.2.3 click==6.7 cloudpickle==0.5.2 clyent==1.2.2 colorama==0.3.7 commonmark==0.8.1 conda==4.6.0 conda-build==3.8.0 conda-verify==2.0.0 configobj==5.0.6 contextlib2==0.5.5 coverage==4.4.1 cryptography==2.2.1 curlify==2.1.0 cycler==0.10.0 cymem==1.31.2 Cython==0.28.1 cytoolz==0.8.2 dask==0.17.2 datashape==0.5.4 datasketch==1.2.10 deap==1.2.2 decorator==4.2.1 dewey2==0.0.1 dill==0.2.7.1 distributed==1.21.6 docker==3.5.0 docker-pycreds==0.3.0 docutils==0.14 docx2txt==0.7 doppel==0.0.1 doppel-cli==0.1.7 dpath==1.4.2 en-core-web-sm==2.0.0 entrypoints==0.2.3 escher-write-client==1.13.0 et-xmlfile==1.0.1 fastavro==0.19.9 fastcache==1.0.2 filelock==3.0.4 Flask==0.12.2 Flask-Cors==3.0.3 funcy==1.11 future==0.16.0 gensim==3.4.0 gevent==1.3a1 gitdb2==2.0.4 GitPython==2.1.11 glob2==0.5 gmpy2==2.0.8 great-expectations==0.4.1 greenlet==0.4.13 grequests==0.3.0 h5py==2.7.1 heapdict==1.0.0 html5lib==1.0.1 idna==2.6 idna-ssl==1.1.0 imageio==2.3.0 imagesize==1.0.0 impyla==0.14.1 ipykernel==4.8.2 ipython==6.3.1 ipython-genutils==0.2.0 ipywidgets==7.2.0 isort==4.3.4 itsdangerous==0.24 jdcal==1.3 jedi==0.11.1 Jinja2==2.10 joblib==0.11 jsonpickle==0.9.6 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.2.3 jupyter-console==5.2.0 jupyter-core==4.4.0 kafka==1.3.5 Keras==2.2.4 Keras-Applications==1.0.6 Keras-Preprocessing==1.0.5 keyring==12.2.1 keyrings.alt==3.0 langdetect==1.0.7 lazy-object-proxy==1.3.1 lightgbm==2.0.12 livereload==2.5.2 llvmlite==0.21.0 locket==0.2.0 lxml==4.2.1 lz4==2.1.0 MarkupSafe==1.0 matplotlib==2.0.2 mccabe==0.6.1 mistune==0.8.3 mock==2.0.0 more-itertools==4.1.0 msgpack-numpy==0.4.1 msgpack-python==0.5.6 multidict==4.3.1 multipledispatch==0.5.0 multiprocess==0.70.6.1 murmurhash==0.28.0 nb-anacondacloud==1.4.0 nb-conda==2.2.1 nb-conda-kernels==2.2.0 nbconvert==5.3.1 nbformat==4.4.0 nbpresent==3.0.2 nbsphinx==0.3.5 networkx==2.1 nltk==3.2.4 nose==1.3.7 notebook==5.4.1 numba==0.36.2 numexpr==2.6.4 numpy==1.14.2 numpydoc==0.8.0 oauthlib==2.1.0 odfpy==1.4.0 odo==0.5.1 olefile==0.45.1 openpyxl==2.5.1 packaging==17.1 pandas==0.23.4 pandas-profiling==1.4.1 pandocfilters==1.4.1 parso==0.1.1 partd==0.3.8 path.py==10.3.1 pathlib==1.0.1 pathlib2==2.3.0 pathos==0.2.2.1 patsy==0.5.0 pbr==4.1.1 pdfminer3k==1.3.1 pep8==1.7.1 pexpect==4.4.0 pickleshare==0.7.4 Pillow==4.2.1 pkginfo==1.4.2 plac==0.9.6 ply==3.11 pockets==0.6.2 pox==0.2.4 ppft==1.6.4.8 preshed==1.0.0 psutil==5.4.3 psycopg2==2.7.5 ptyprocess==0.5.2 py==1.5.3 pyasn1==0.4.2 pycodestyle==2.3.1 pycosat==0.6.3 pycparser==2.18 pycrypto==2.6.1 pycurl==7.43.0 pyflakes==1.6.0 Pygments==2.2.0 PyJWT==1.6.4 pyLDAvis==2.1.2 pylint==1.8.4 pyOpenSSL==17.5.0 pyparsing==2.2.0 pyproj==1.9.5.1 PyQt5==5.10.1 pyqtgraph==0.10.0 pyshp==1.2.12 PySocks==1.6.8 pytest==4.0.2 pytest-arraydiff==0.2 pytest-astropy==0.2.1 pytest-cov==2.6.1 pytest-doctestplus==0.1.2 pytest-openfiles==0.2.0 pytest-remotedata==0.2.0 python-dateutil==2.7.2 python-geohash==0.8.5 python-louvain==0.11 pytz==2018.3 PyWavelets==0.5.2 PyYAML==3.12 pyzmq==17.0.0 QtAwesome==0.4.4 qtconsole==4.3.1 QtPy==1.4.0 recommonmark==0.4.0 redis==2.10.6 regex==2017.4.5 requests==2.18.4 requests-oauthlib==1.0.0 responses==0.10.5 rgf-python==3.3.0 rope==0.10.7 rsa==3.4.2 ruamel-yaml==0.15.35 s3transfer==0.1.13 scikit-image==0.13.1 scikit-learn==0.19.0 scipy==0.19.1 seaborn==0.9.0 Send2Trash==1.5.0 serverfiles==0.2.1 setuptools-git==1.2 Shapely==1.6.4.post1 simhash==1.9.0 simplegeneric==0.8.1 simplejson==3.13.2 singledispatch==3.4.0.3 sip==4.19.8 six==1.11.0 sklearn-deap2==0.2.2 smart-open==1.7.1 smmap2==2.0.4 snowballstemmer==1.2.1 sockjs-tornado==1.0.3 sortedcontainers==1.5.9 spacy==2.0.11 Sphinx==1.8.2 sphinx-autodoc-typehints==1.5.0 sphinx-click==2.0.1 sphinx-rtd-theme==0.3.0 sphinxcontrib-napoleon==0.6.1 sphinxcontrib-programoutput==0.11 sphinxcontrib-websupport==1.0.1 spyder==3.2.8 SQLAlchemy==1.2.6 statsmodels==0.8.0 stop-words==2018.7.23 tables==3.4.2 tabulate==0.8.2 tblib==1.3.2 tensorflow-hub==0.2.0 termcolor==1.1.0 terminado==0.8.1 testpath==0.3.1 Theano==1.0.3 thinc==6.10.2 thriftpy==0.3.9 toolz==0.9.0 tornado==5.0.2 tqdm==4.21.0 traitlets==4.3.2 tweepy==3.6.0 typing==3.6.4 ufal.udpipe==1.2.0.1 ufal.udpipe-temp==1.2.0.4 ujson==1.35 unicodecsv==0.14.1 uplink==0.5.4 uritemplate==3.0.0 urllib3==1.22 uuid==1.30 validate-email==1.3 wcwidth==0.1.7 webencodings==0.5 websocket-client==0.53.0 Werkzeug==0.14.1 widgetsnbextension==3.2.0 wikipedia==1.4.0 wrapt==1.10.11 xgboost==0.81 xlearn==0.31a1 xlrd==1.1.0 XlsxWriter==1.0.2 xlwings==0.11.7 xlwt==1.3.0 yarl==1.2.6 zict==0.1.3

jameslamb commented 5 years ago

Can report the same error on scipy!

doppel-describe -p scipy --language python --data-dir $(pwd)/test_data
Logs with error

Testing package scipy [python] /Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py Describing package with command: /Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py --pkg scipy --output_dir /Users/jlamb/repos/doppel-cli/test_data --kwargs-string ~~KWARGS~~ --constructor-string ~~CONSTRUCTOR~~ Could not figure out what ALLOW_THREADS is AxisError is an Exception. Skipping. Could not figure out what BUFSIZE is Could not figure out what CLIP is ComplexWarning is an Exception. Skipping. Could not figure out what ERR_CALL is Could not figure out what ERR_DEFAULT is Could not figure out what ERR_IGNORE is Could not figure out what ERR_LOG is Could not figure out what ERR_PRINT is Could not figure out what ERR_RAISE is Could not figure out what ERR_WARN is Could not figure out what FLOATING_POINT_SUPPORT is Could not figure out what FPE_DIVIDEBYZERO is Could not figure out what FPE_INVALID is Could not figure out what FPE_OVERFLOW is Could not figure out what FPE_UNDERFLOW is Could not figure out what False_ is Could not figure out what Inf is Could not figure out what Infinity is 'LowLevelCallable' is a class in this package, adding it Traceback (most recent call last): File "/Users/jlamb/anaconda3/lib/python3.6/inspect.py", line 1119, in getfullargspec sigcls=Signature) File "/Users/jlamb/anaconda3/lib/python3.6/inspect.py", line 2266, in _signature_from_callable skip_bound_arg=skip_bound_arg) File "/Users/jlamb/anaconda3/lib/python3.6/inspect.py", line 2090, in _signature_from_builtin raise ValueError("no signature found for builtin {!r}".format(func)) ValueError: no signature found for builtin The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py", line 132, in method_args = _get_arg_names(func, KWARGS_STRING) File "/Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py", line 68, in _get_arg_names f_dict = inspect.getfullargspec(f)._asdict() File "/Users/jlamb/anaconda3/lib/python3.6/inspect.py", line 1125, in getfullargspec raise TypeError('unsupported callable') from ex TypeError: unsupported callable During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/jlamb/Library/Caches/Python-Eggs/doppel_cli-0.1.7-py3.6.egg-tmp/doppel/bin/analyze.py", line 135, in func.__wrapped__, AttributeError: 'method_descriptor' object has no attribute '__wrapped__'

jameslamb commented 5 years ago

Ok yeah this is bad, was able to replicate this error on:

Runs fine on

jameslamb commented 5 years ago

Still investigating. I've narrowed it do to networkx.DiGraph.adjlist_inner_dict_factory (as on example)

This is confusing to me:

In [35]: callable(networkx.DiGraph.adjlist_inner_dict_factory)
Out[35]: True

In [36]: str(networkx.DiGraph.adjlist_inner_dict_factory)
Out[36]: "<class 'dict'>"
jameslamb commented 5 years ago

Think I figured it out! networkx.DiGraph.adjlist_inner_dict_factory isn't a function...it's a class factory! By default, it's dict

"adjlist_inner_dict_factory : function, (default: dict) Factory function to be used to create the adjacency list dict which holds edge data keyed by neighbor. It should require no arguments and return a dict-like object"

The key file: https://github.com/networkx/networkx/blob/master/networkx/classes/graph.py

So the bug in doppel is that in the code for "is this thing in the class a method", doppel is checking callable(this_thing). However, classes are also callable! some_dict = dict() generates an instance of class dict.

SO, to solve this I think the thing that makes sense is:

  1. Change the check in describe.py to just check for functions the same way doppel does at the top level (isinstance(obj, types.FunctionType))
  2. Open a "question" issue about representing class factories which are themselves callable classes (not functions) in the API JSON produced by doppel