search

jpn-- / larch

Larch: a Python tool for choice modeling
http://larch.newman.me
GNU General Public License v3.0
41 stars 14 forks source link

Reporter class crashes when adding data. #14

Closed m-richards closed 2 years ago

m-richards commented 2 years ago

The larch reporter class does not appear to be working. I am new to larch so am not sure if this is an environment specific problem or a general bug.

The minimal reproducible example is to attempt run example 201 as outlined in the start of example 202:

import larch, numpy, pandas, os
from larch import P, X
exampville_mode_choice_file = larch.example(201, output_file='/tmp/exampville_mode_choice.html')

Running through example 201 line by line following https://larch.newman.me/example/201_exville_mode_choice.html gives an issue near the very end at:

report = larch.Reporter(title=m.title)
report << '# Parameter Summary' << m.parameter_summary()

I have been able to reproduce this issue on both Windows and windows subsystem for linux (ubuntu 20).

I created an environment on both platforms as follows:

conda create -n larch2 python=3.7 larch
conda install geopandas

where geopandas is installed additionally to run the exampville examples.

Conda environment package versions: ``` # This file may be used to create an environment using: # $ conda create --name --file # platform: linux-64 _libgcc_mutex=0.1=conda_forge _openmp_mutex=4.5=1_gnu abseil-cpp=20210324.2=h9c3ff4c_0 addict=2.4.0=py37h89c1867_0 appdirs=1.4.4=pyh9f0ad1d_0 arrow-cpp=5.0.0=py37h43e6a4c_0_cpu atk-1.0=2.36.0=h3371d22_4 attrs=21.2.0=pyhd8ed1ab_0 aws-c-cal=0.5.11=h95a6274_0 aws-c-common=0.6.2=h7f98852_0 aws-c-event-stream=0.2.7=h3541f99_13 aws-c-io=0.10.5=hfb6a706_0 aws-checksums=0.1.11=ha31a3da_7 aws-sdk-cpp=1.8.186=hb4091e7_3 backcall=0.2.0=pyh9f0ad1d_0 backports=1.0=py_2 backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0 beautifulsoup4=4.9.3=pyhb0f4dca_0 blosc=1.21.0=h9c3ff4c_0 boost-cpp=1.74.0=h312852a_4 bzip2=1.0.8=h7f98852_4 c-ares=1.17.1=h7f98852_1 ca-certificates=2021.5.30=ha878542_0 cairo=1.16.0=h6cf1ce9_1008 certifi=2021.5.30=py37h89c1867_0 cfitsio=3.470=hb418390_7 click=7.1.2=pyh9f0ad1d_0 click-plugins=1.1.1=py_0 cligj=0.7.2=pyhd8ed1ab_0 cloudpickle=1.6.0=py_0 colorama=0.4.4=pyh9f0ad1d_0 curl=7.78.0=hea6ffbf_0 cycler=0.10.0=py_2 decorator=4.4.2=py_0 docutils=0.17.1=py37h89c1867_0 et_xmlfile=1.0.1=py_1001 expat=2.4.1=h9c3ff4c_0 fiona=1.8.20=py37hc1d69b0_1 font-ttf-dejavu-sans-mono=2.37=hab24e00_0 font-ttf-inconsolata=3.000=h77eed37_0 font-ttf-source-code-pro=2.038=h77eed37_0 font-ttf-ubuntu=0.83=hab24e00_0 fontconfig=2.13.1=hba837de_1005 fonts-conda-ecosystem=1=0 fonts-conda-forge=1=0 freetype=2.10.4=h0708190_1 freexl=1.0.6=h7f98852_0 fribidi=1.0.10=h36c2ea0_0 gdal=3.3.1=py37hb0e9ad2_1 gdk-pixbuf=2.42.6=h04a7f16_0 geopandas=0.9.0=pyhd8ed1ab_1 geopandas-base=0.9.0=pyhd8ed1ab_1 geos=3.9.1=h9c3ff4c_2 geotiff=1.6.0=h4f31c25_6 gettext=0.19.8.1=h0b5b191_1005 gflags=2.2.2=he1b5a44_1004 giflib=5.2.1=h36c2ea0_2 glog=0.5.0=h48cff8f_0 graphite2=1.3.13=h58526e2_1001 graphviz=2.48.0=h85b4f2f_0 grpc-cpp=1.38.1=h36ce80c_0 gtk2=2.24.33=h539f30e_1 gts=0.7.6=h64030ff_2 harfbuzz=2.8.2=h83ec7ef_0 hdf4=4.2.15=h10796ff_3 hdf5=1.10.6=nompi_h6a2412b_1114 icu=68.1=h58526e2_0 importlib-metadata=4.6.1=py37h89c1867_0 importlib_metadata=4.6.1=hd8ed1ab_0 iniconfig=1.1.1=pyh9f0ad1d_0 ipython=7.25.0=py37h085eea5_1 ipython_genutils=0.2.0=py_1 jbig=2.1=h7f98852_2003 jdcal=1.4.1=py_0 jedi=0.18.0=py37h89c1867_2 jinja2=3.0.1=pyhd8ed1ab_0 joblib=1.0.1=pyhd8ed1ab_0 jpeg=9d=h36c2ea0_0 json-c=0.15=h98cffda_0 kealib=1.4.14=hcc255d8_2 kiwisolver=1.3.1=py37h2527ec5_1 krb5=1.19.2=hcc1bbae_0 larch=5.5.5=py37h6f94858_0 lcms2=2.12=hddcbb42_0 ld_impl_linux-64=2.36.1=hea4e1c9_2 lerc=2.2.1=h9c3ff4c_0 libblas=3.9.0=10_openblas libbrotlicommon=1.0.9=h7f98852_5 libbrotlidec=1.0.9=h7f98852_5 libbrotlienc=1.0.9=h7f98852_5 libcblas=3.9.0=10_openblas libcurl=7.78.0=h2574ce0_0 libdap4=3.20.6=hd7c4107_2 libdeflate=1.7=h7f98852_5 libedit=3.1.20191231=he28a2e2_2 libev=4.33=h516909a_1 libevent=2.1.10=hcdb4288_3 libffi=3.3=h58526e2_2 libgcc-ng=11.1.0=hc902ee8_5 libgd=2.3.2=h78a0170_0 libgdal=3.3.1=h8f005ca_1 libgfortran-ng=11.1.0=h69a702a_5 libgfortran5=11.1.0=h6c583b3_5 libglib=2.68.3=h3e27bee_0 libgomp=11.1.0=hc902ee8_5 libiconv=1.16=h516909a_0 libkml=1.3.0=h238a007_1014 liblapack=3.9.0=10_openblas libnetcdf=4.8.0=nompi_hcd642e3_103 libnghttp2=1.43.0=h812cca2_0 libopenblas=0.3.17=pthreads_h8fe5266_1 libpng=1.6.37=h21135ba_2 libpq=13.3=hd57d9b9_0 libprotobuf=3.16.0=h780b84a_0 librsvg=2.50.7=hc3c00ef_0 librttopo=1.1.0=h1185371_6 libspatialindex=1.9.3=h9c3ff4c_4 libspatialite=5.0.1=h8694cbe_5 libssh2=1.9.0=ha56f1ee_6 libstdcxx-ng=11.1.0=h56837e0_5 libthrift=0.14.2=he6d91bd_1 libtiff=4.3.0=hf544144_1 libtool=2.4.6=h58526e2_1007 libutf8proc=2.6.1=h7f98852_0 libuuid=2.32.1=h7f98852_1000 libwebp=1.2.0=h3452ae3_0 libwebp-base=1.2.0=h7f98852_2 libxcb=1.13=h7f98852_1003 libxml2=2.9.12=h72842e0_0 libxslt=1.1.33=h15afd5d_2 libzip=1.8.0=h4de3113_0 lxml=4.6.3=py37h77fd288_0 lz4-c=1.9.3=h9c3ff4c_1 lzo=2.10=h516909a_1000 mapclassify=2.4.3=pyhd8ed1ab_0 markupsafe=2.0.1=py37h5e8e339_0 matplotlib-base=3.4.2=py37hdd32ed1_0 matplotlib-inline=0.1.2=pyhd8ed1ab_2 mock=4.0.3=py37h89c1867_1 more-itertools=8.8.0=pyhd8ed1ab_0 munch=2.5.0=py_0 ncurses=6.2=h58526e2_4 networkx=2.6.2=pyhd8ed1ab_0 numexpr=2.7.3=py37hdc94413_0 numpy=1.21.1=py37h038b26d_0 olefile=0.46=pyh9f0ad1d_1 openjpeg=2.4.0=hb52868f_1 openpyxl=3.0.7=pyhd8ed1ab_0 openssl=1.1.1k=h7f98852_0 orc=1.6.9=h58a87f1_0 packaging=21.0=pyhd8ed1ab_0 pandas=1.3.1=py37h219a48f_0 pango=1.48.7=hb8ff022_0 parquet-cpp=1.5.1=2 parso=0.8.2=pyhd8ed1ab_0 patsy=0.5.1=py_0 pcre=8.45=h9c3ff4c_0 pexpect=4.8.0=pyh9f0ad1d_2 pickleshare=0.7.5=py_1003 pillow=8.3.1=py37h0f21c89_0 pip=21.2.1=pyhd8ed1ab_0 pixman=0.40.0=h36c2ea0_0 pluggy=0.13.1=py37h89c1867_4 poppler=21.03.0=h93df280_0 poppler-data=0.4.10=0 postgresql=13.3=h2510834_0 proj=8.0.1=h277dcde_0 prompt-toolkit=3.0.19=pyha770c72_0 pthread-stubs=0.4=h36c2ea0_1001 ptyprocess=0.7.0=pyhd3deb0d_0 py=1.10.0=pyhd3deb0d_0 pyarrow=5.0.0=py37h58331f5_0_cpu pydot=1.4.2=py37h89c1867_0 pygments=2.9.0=pyhd8ed1ab_0 pyparsing=2.4.7=pyh9f0ad1d_0 pyproj=3.1.0=py37h2f13a41_3 pytables=3.6.1=py37h0c4f3e0_3 pytest=6.2.4=py37h89c1867_0 python=3.7.10=hffdb5ce_100_cpython python-dateutil=2.8.2=pyhd8ed1ab_0 python_abi=3.7=2_cp37m pytz=2021.1=pyhd8ed1ab_0 pyyaml=5.4.1=py37h5e8e339_0 re2=2021.06.01=h9c3ff4c_0 readline=8.1=h46c0cb4_0 rtree=0.9.7=py37h0b55af0_2 s2n=1.0.10=h9b69904_0 scikit-learn=0.24.2=py37h18a542f_0 scipy=1.7.0=py37h29e03ee_1 seaborn=0.11.1=hd8ed1ab_1 seaborn-base=0.11.1=pyhd8ed1ab_1 setuptools=49.6.0=py37h89c1867_3 shapely=1.7.1=py37h2d1e849_5 six=1.16.0=pyh6c4a22f_0 snappy=1.1.8=he1b5a44_3 soupsieve=2.0.1=py_1 sqlite=3.36.0=h9cd32fc_0 statsmodels=0.12.2=py37h902c9e0_0 threadpoolctl=2.2.0=pyh8a188c0_0 tiledb=2.3.2=he87e0bf_0 tk=8.6.10=h21135ba_1 toml=0.10.2=pyhd8ed1ab_0 tornado=6.1=py37h5e8e339_1 tqdm=4.62.0=pyhd8ed1ab_0 traitlets=5.0.5=py_0 typing_extensions=3.10.0.0=pyha770c72_0 tzcode=2021a=h7f98852_2 tzdata=2021a=he74cb21_1 wcwidth=0.2.5=pyh9f0ad1d_2 wheel=0.36.2=pyhd3deb0d_0 xerces-c=3.2.3=h9d8b166_2 xlsxwriter=1.4.5=pyhd8ed1ab_0 xmle=0.1.21=pyh44b312d_0 xorg-kbproto=1.0.7=h7f98852_1002 xorg-libice=1.0.10=h7f98852_0 xorg-libsm=1.2.3=hd9c2040_1000 xorg-libx11=1.7.2=h7f98852_0 xorg-libxau=1.0.9=h7f98852_0 xorg-libxdmcp=1.1.3=h7f98852_0 xorg-libxext=1.3.4=h7f98852_1 xorg-libxrender=0.9.10=h7f98852_1003 xorg-renderproto=0.11.1=h7f98852_1002 xorg-xextproto=7.3.0=h7f98852_1002 xorg-xproto=7.0.31=h7f98852_1007 xz=5.2.5=h516909a_1 yaml=0.2.5=h516909a_0 zipp=3.5.0=pyhd8ed1ab_0 zlib=1.2.11=h516909a_1010 zstd=1.5.0=ha95c52a_0 ```

There's an infinite recursion in the xml writing. This is the stack trace I get on WSL(on windows I didn't even get a stack trace, just "Process finished with exit code -1073741571 (0xC00000FD)"):

Traceback (most recent call last):
  File "/home/m-richards/conda/envs/larch2/lib/python3.7/site-packages/xmle/elem.py", line 179, in from_string
    return xml.etree.ElementTree.fromstring(xml_as_string, parser=XMLParser(target=TreeBuilder(element_factory=cls)))
  File "/home/m-richards/conda/envs/larch2/lib/python3.7/xml/etree/ElementTree.py", line 1315, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: undefined entity: line 17, column 32

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/m-richards/conda/envs/larch2/lib/python3.7/site-packages/xmle/elem.py", line 186, in from_string
    return cls.from_string("<div>"+xml_as_string.replace("<style scoped>", "<style scoped='1'>")+"</div>", _fix1=True)
  File "/home/m-richards/conda/envs/larch2/lib/python3.7/site-packages/xmle/elem.py", line 186, in from_string
    return cls.from_string("<div>"+xml_as_string.replace("<style scoped>", "<style scoped='1'>")+"</div>", _fix1=True)
  File "/home/m-richards/conda/envs/larch2/lib/python3.7/site-packages/xmle/elem.py", line 186, in from_string
    return cls.from_string("<div>"+xml_as_string.replace("<style scoped>", "<style scoped='1'>")+"</div>", _fix1=True)
  [Previous line repeated 994 more times]
  File "/home/m-richards/conda/envs/larch2/lib/python3.7/site-packages/xmle/elem.py", line 179, in from_string
    return xml.etree.ElementTree.fromstring(xml_as_string, parser=XMLParser(target=TreeBuilder(element_factory=cls)))
  File "/home/m-richards/conda/envs/larch2/lib/python3.7/xml/etree/ElementTree.py", line 1315, in XML
    parser.feed(text)
  File "/home/m-richards/conda/envs/larch2/lib/python3.7/site-packages/xmle/elem.py", line 28, in __init__
    if isinstance(tag, Element):
RecursionError: maximum recursion depth exceeded while calling a Python object
jpn-- commented 2 years ago

It appears that a recent pandas update made a change in how dataframes were rendered using the styler, which broke the reporter. I've fixed the xmle library that larch uses for reporting, this should work now if you use conda to update xmle to 0.1.22. (conda install -n larch2 xmle=0.1.22 -c conda-forge).

m-richards commented 2 years ago

It's now working so that's great. Thanks for the quick fix!