[BUG] Transit network silently drops data due to partridge pruning

[lmz]

Describe the bug

The transit library, partridge, apparently silently prunes out rows in the dataset; from the partridge Readme.rsg

At the core of Partridge is a dependency graph rooted at trips.txt. Disconnected data is pruned away according to this graph when reading the contents of a feed.

This isn't obvious to the user of network_wrangler and resulted in some stops and a lot of fare rules and fare attributes being dropped. @i-am-sijia implemented a workaround to the fare files here: https://github.com/BayAreaMetro/network_wrangler/commit/068e0cf43ed342c1b9c3b4ef1debf20b43cc339a

But I have concerns with this:

• If other rows are being dropped silently, that could be surprising to the user and an indication there's a problem with the network. A warning would be preferable.
• Are the fare file column types being read consistently as partridge?

Status

• [x] Defined
• [x] Planned
• [x] Implemented
• [x] Tested

To Reproduce

Reading MTC's version 12 standard transit files pre base project chards (without the fare patch) drop stops, fare_rules and fare_attributes. See image below, in which I added logging to a local partridge.

Failing tests

• [ ] No applicable test failed, need to create.

Triggering line of code

Thoughts on resolution

If the dropping is legit, then warn on it. Not sure why the fare lines are being dropped... Could be a partridge bug?

Full stack trace

Environment

Operating system: Windows Context (conda, jupyter, etc): conda jupyter Environment (e.g. output from conda list):

(tm2_network_dev) PS C:\Users\lzorn\Documents\GitHub\travel-model-two-networks> conda list
# packages in environment at C:\Users\lzorn\.conda\envs\tm2_network_dev:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.12                   pypi_0    pypi
argon2-cffi               20.1.0           py37he774522_1
atomicwrites              1.4.0                    pypi_0    pypi
attrs                     20.2.0                     py_0
babel                     2.9.1                    pypi_0    pypi
backcall                  0.2.0                      py_0
black                     21.12b0                  pypi_0    pypi
blas                      1.0                         mkl
bleach                    3.2.1                      py_0
branca                    0.4.2                    pypi_0    pypi
brotlipy                  0.7.0           py37he774522_1000
bzip2                     1.0.8                he774522_0
ca-certificates           2020.7.22                     0
cchardet                  2.1.7                    pypi_0    pypi
certifi                   2020.6.20                py37_0
cffi                      1.14.2           py37h7a1dbc1_0
cfitsio                   3.470                he774522_5
chardet                   3.0.4                 py37_1003
click                     7.1.2                      py_0
click-plugins             1.1.1                      py_0
cligj                     0.5.0                    py37_0
colorama                  0.4.3                      py_0
commonmark                0.9.1                    pypi_0    pypi
cryptography              3.1              py37h7a1dbc1_0
curl                      7.67.0               h2a8f88b_0
cycler                    0.10.0                   pypi_0    pypi
dbfread                   2.0.7                    pypi_0    pypi
decorator                 4.4.2                      py_0
defusedxml                0.6.0                      py_0
descartes                 1.1.0                    pypi_0    pypi
docutils                  0.18.1                   pypi_0    pypi
entrypoints               0.3                      py37_0
expat                     2.2.9                h33f27b4_2
fiona                     1.8.13.post1     py37hd760492_0
folium                    0.12.1.post1             pypi_0    pypi
freexl                    1.0.5                hfa6e2cd_0
gdal                      3.0.2            py37hdf43c64_0
geographiclib             1.52                     pypi_0    pypi
geojson                   2.5.0                    pypi_0    pypi
geopandas                 0.8.1                      py_0
geos                      3.8.0                h33f27b4_0
geotiff                   1.5.1                h5770a2b_1
hdf4                      4.2.13               h712560f_2
hdf5                      1.10.4               h7ebc959_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha925a31_3
idna                      2.10                       py_0
imagesize                 1.3.0                    pypi_0    pypi
importlib-metadata        1.7.0                    py37_0
importlib_metadata        1.7.0                         0
iniconfig                 1.1.1                    pypi_0    pypi
intel-openmp              2020.2                      254
ipykernel                 5.3.4            py37h5ca1d4c_0
ipython                   7.18.1           py37h5ca1d4c_0
ipython_genutils          0.2.0                    py37_0
ipywidgets                7.6.5                    pypi_0    pypi
isoweek                   1.3.3                    pypi_0    pypi
jedi                      0.17.2                   py37_0
jinja2                    2.11.2                     py_0
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9b                   hb83a4c4_2
json5                     0.9.5                      py_0
jsonschema                3.2.0                    py37_1
jupyter                   1.0.0                    pypi_0    pypi
jupyter-console           6.4.0                    pypi_0    pypi
jupyter_client            6.1.6                      py_0
jupyter_core              4.6.3                    py37_0
jupyterlab                2.2.6                      py_0
jupyterlab-widgets        1.0.2                    pypi_0    pypi
jupyterlab_server         1.2.0                      py_0
kealib                    1.4.7                h07cbb95_6
kiwisolver                1.2.0                    pypi_0    pypi
krb5                      1.16.4               hc04afaa_0
lark-parser               0.12.0                   pypi_0    pypi
libboost                  1.67.0               hd9e427e_4
libcurl                   7.67.0               h2a8f88b_0
libgdal                   3.0.2                h1155b67_0
libiconv                  1.15                 h1df5818_7
libkml                    1.3.0                he5f2a48_4
libnetcdf                 4.6.1                h411e497_2
libpng                    1.6.37               h2a8f88b_0
libpq                     11.2                 h3235a2c_0
libsodium                 1.0.18               h62dcd97_0
libspatialindex           1.9.3                h33f27b4_0
libspatialite             4.3.0a               h7ffb84d_0
libssh2                   1.9.0                h7a1dbc1_1
libtiff                   4.1.0                h56a325e_0
libxml2                   2.9.10               h464c3ec_1
lz4-c                     1.8.1.2              h2fa13f4_0
m2w64-expat               2.1.1                         2
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gettext             0.19.7                        2
m2w64-gmp                 6.1.0                         2
m2w64-libiconv            1.14                          6
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
m2w64-xz                  5.2.2                         2
markupsafe                1.1.1            py37hfa6e2cd_1
matplotlib                3.3.2                    pypi_0    pypi
mistune                   0.8.4           py37hfa6e2cd_1001
mkl                       2020.2                      256
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.1.0            py37h45dec08_0
mkl_random                1.1.1            py37h47e9c7a_0
msys2-conda-epoch         20160418                      1
munch                     2.5.0                      py_0
mypy-extensions           0.4.3                    pypi_0    pypi
nbconvert                 5.6.1                    py37_1
nbformat                  5.0.7                      py_0
networkx                  2.5                        py_0
notebook                  6.1.1                    py37_0
numpy                     1.21.5                   pypi_0    pypi
openjpeg                  2.3.0                h5ec785f_1
openssl                   1.1.1g               he774522_1
osmnx                     0.16.0                   pypi_0    pypi
packaging                 20.4                       py_0
pandas                    1.1.1            py37ha925a31_0
pandoc                    2.10.1                        0
pandocfilters             1.4.2                    py37_1
parso                     0.7.0                      py_0
partridge                 1.1.1                     dev_0    <develop>
pathspec                  0.9.0                    pypi_0    pypi
pcre                      8.44                 ha925a31_0
peartree                  0.6.4                    pypi_0    pypi
pickleshare               0.7.5                 py37_1001
pillow                    7.2.0                    pypi_0    pypi
pip                       20.2.2                   py37_0
platformdirs              2.4.1                    pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
postgresql                11.2                 h3235a2c_0
proj                      6.2.1                h9f7ef89_0
prometheus_client         0.8.0                      py_0
prompt-toolkit            3.0.7                      py_0
py                        1.11.0                   pypi_0    pypi
pyarrow                   6.0.1                    pypi_0    pypi
pycparser                 2.20                       py_2
pygments                  2.7.0                      py_0
pyopenssl                 19.1.0                     py_1
pyparsing                 2.4.7                      py_0
pyproj                    2.6.1.post1      py37hcfa1391_1
pyrsistent                0.17.3           py37he774522_0
pysocks                   1.7.1                    py37_1
pytest                    6.2.5                    pypi_0    pypi
python                    3.7.9                h60c2a47_0
python-dateutil           2.8.1                      py_0
python-graphviz           0.19.1                   pypi_0    pypi
pytz                      2020.1                     py_0
pywin32                   227              py37he774522_1
pywinpty                  0.5.7                    py37_0
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     19.0.1           py37ha925a31_1
qtconsole                 5.2.2                    pypi_0    pypi
qtpy                      2.0.0                    pypi_0    pypi
recommonmark              0.7.1                    pypi_0    pypi
requests                  2.24.0                     py_0
rtree                     0.9.4            py37h21ff451_1
scikit-learn              1.0.2                    pypi_0    pypi
scipy                     1.7.0                    pypi_0    pypi
send2trash                1.5.0                    py37_0
setuptools                49.6.0                   py37_0
shapely                   1.7.1            py37h210f175_0
simpledbf                 0.2.6                    pypi_0    pypi
six                       1.15.0                     py_0
sklearn                   0.0                      pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
sphinx                    4.3.2                    pypi_0    pypi
sphinx-autodoc-typehints  1.15.2                   pypi_0    pypi
sphinx-rtd-theme          1.0.0                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
sqlite                    3.33.0               h2a8f88b_0
tbb                       2018.0.5             he980bc4_0
terminado                 0.8.3                    py37_0
testpath                  0.4.4                      py_0
threadpoolctl             3.0.0                    pypi_0    pypi
tiledb                    1.6.3                h7b000aa_0
tk                        8.6.10               he774522_0
toml                      0.10.2                   pypi_0    pypi
tomli                     1.2.3                    pypi_0    pypi
tornado                   6.0.4            py37he774522_1
traitlets                 4.3.3                    py37_0
typed-ast                 1.5.1                    pypi_0    pypi
typing-extensions         4.0.1                    pypi_0    pypi
unidecode                 1.3.2                    pypi_0    pypi
urllib3                   1.25.10                    py_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_3
wcwidth                   0.2.5                      py_0
webencodings              0.5.1                    py37_1
wheel                     0.35.1                     py_0
widgetsnbextension        3.5.2                    pypi_0    pypi
win_inet_pton             1.1.0                    py37_0
wincertstore              0.2                      py37_0
winpty                    0.4.3                         4
wrangler-lasso            0.0.1                     dev_0    <develop>
xerces-c                  3.2.3                ha925a31_0
xz                        5.2.5                h62dcd97_0
zeromq                    4.3.2                ha925a31_3
zipp                      3.1.0                      py_0
zlib                      1.2.11               h62dcd97_4
zstd                      1.3.7                h508b16e_0

[e-lo]

Resolved by removing Partridge as a dependency