masked_area_interpolate errors

[adamConnerSax]

Hi! Tobler is awesome! Thank you for all the work!

I am aggregating census count data from the census-tract level to the congressional district level. Using "area_interpolate" this works fine. Thanks!!

Now I'm trying to improve this using the NLCD data and "masked_area_interpolate". When I do that I get lots of errors of the form: "CBR: result (after common-bits addition) is INVALID: Self-intersection at or near point..." or "CBR: result (after common-bits addition) is INVALID: Too few points in geometry component at or near point..."

each followed by many many numbers. I've tried projecting both source and target geometries to EPSG:3857 and EPSG:4326 and it happens either way.

It continues to run, despite the errors. So I have 2 questions:

1. Is there something I should be doing with the data before calling "masked_area_interpolate" that might fix this?
2. Will the interpolation still be useful? That is, is masked_area_interpolate going to be an improvement on area_interpolate regardless of all these errors, or should I just stick with the results of area_interpolate until/unless I figure out the errors?

Thanks!

[sjsrey]

Thanks for reporting this.

Can you provide a bit more info regarding your os and version of geopandas?

[adamConnerSax]

Running on mac OSX (10.15.7m Catalina) Python 3.8.5 geopandas 0.8.2 Tobler 0.7

[knaaptime]

tough to diagnose, but my first guess would be you have some topology errors in your source/target data (both of those errors come from shapely).

try running

source = source.buffer(0)
target = target.buffer(0)

before using area_interpolate_masked (though if this were the issue, im not sure why area_interpolate would work, but masked wouldnt...)

Another thing to try would be to see if you can follow this notebook using your data (though you can omit the OSM step if you want) which performs essentially the same task as masked_area_interpolate but with a different approach

[martinfleis]

You'll need

source.geometry = source.buffer(0)
target.geometry = target.buffer(0)

Buffer returns GeoSeries.

[knaaptime]

sorry, yes thank you @martinfleis

[adamConnerSax]

Tried

source.geometry = source.buffer(0)
target.geometry = target.buffer(0)

and I still get the same/similar errors. Will try the notebook thing when I get some time...

[jtamerius]

I'm getting this same error doing a similar calculation. Specifically, I am trying to estimate population from census tracts using developed land rasterized by the geocube api. I also tried the NLCD data and received same error.

[martinfleis]

@jtamerius Can you share the full traceback and a specification of your environment?

[knaaptime]

can you share the full traceback so I can try and see where the root error is coming from? or possibly the source/target data you're using? That's a low-level error coming from geos and usually has to do with invalid geometries afaik, though i have no idea why it would occur only in masked_area_interpolate.

As a workaround, you may be able to use the method described in this notebook instead

[knaaptime]

@martinfleis lol beat me to the punch while i was pasting in the url and i didnt even notice

[jtamerius]

As a workaround, you may be able to use the method described in this notebook instead

@knaaptime : this looks like a great method for estimating population. Unfortunately, this link is not working for me: https://s3-us-west-2.amazonaws.com/mrlc/NLCD_2016_Impervious_L48_20190405.zip

p.s. I'll get you all the traceback soon.

[knaaptime]

hm, they may have updated the links but you should be able to pick an appropriate year from here and use that

https://www.mrlc.gov/data?f%5B0%5D=category%3AUrban%20Imperviousness&f%5B1%5D=category%3Aland%20cover&f%5B2%5D=region%3Aconus

[jtamerius]

So, there is not much of a traceback:

CBR: result (after common-bits addition) is INVALID: Too few points in geometry component at or near point 144907.12541067583 -286284.33574648993 (144907.12541067582788 -286284.3357464899309) <A> GEOMETRYCOLLECTION (LINESTRING (157566.5237387096858583 -293342.2898403313010931, 157817.6488116196996998 -293103.4683011914021336), LINESTRING (155667.2435246870154515 -291631.9672550756367855, 155665.4470387034816667 -291624.1666764495312236), LINESTRING (155742.3051454601809382 -291823.6784229896729812, 155748.8153089357656427 -291825.4875020556501113),...

I truncated list of coordinates because it goes on forever. I added teh target and source geometries to zip file attached. Here are packages I have installed:

Name Version Build Channel affine 2.3.0 py_0 amqp 5.0.6 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge atomicwrites 1.4.0 py_0 attrs 21.2.0 pyhd3eb1b0_0 backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge bcrypt 3.2.0 py39hb82d6ee_1 conda-forge beautifulsoup4 4.9.3 pyha847dfd_0 billiard 3.6.4.0 py39hb82d6ee_0 conda-forge blas 1.0 mkl blosc 1.21.0 h0e60522_0 conda-forge bokeh 2.3.3 py39hcbf5309_0 conda-forge branca 0.4.2 pyhd8ed1ab_0 conda-forge brotli-python 1.0.9 py39h415ef7b_4 conda-forge brotlipy 0.7.0 py39h2bbff1b_1003 bs4 4.9.3 hd3eb1b0_0 bzip2 1.0.8 he774522_0 ca-certificates 2021.5.30 h5b45459_0 conda-forge cachetools 4.2.2 pyhd8ed1ab_0 conda-forge cairo 1.16.0 hb19e0ff_1008 conda-forge celery 5.1.2 pyhd8ed1ab_0 conda-forge censusdata 1.13 pypi_0 pypi certifi 2021.5.30 py39hcbf5309_0 conda-forge cffi 1.14.6 py39h2bbff1b_0 cfitsio 3.470 he774522_6 cftime 1.5.0 py39h5d4886f_0 conda-forge chardet 4.0.0 py39haa95532_1003 click 7.1.2 pyhd3eb1b0_0 click-didyoumean 0.0.3 pyh8c360ce_0 conda-forge click-plugins 1.1.1 py_0 click-repl 0.2.0 pyhd8ed1ab_0 conda-forge cligj 0.7.2 py39haa95532_0 cloudpickle 1.6.0 py_0 conda-forge colorama 0.4.4 pyhd3eb1b0_0 cryptography 3.4.7 py39h71e12ea_0 curl 7.77.0 h789b8ee_0 conda-forge cycler 0.10.0 py39haa95532_0 cytoolz 0.11.0 py39hb82d6ee_3 conda-forge dash 1.20.0 pyhd8ed1ab_0 conda-forge dash-core-components 1.16.0 pyhd8ed1ab_0 conda-forge dash-html-components 1.1.3 pyhd8ed1ab_0 conda-forge dash-renderer 1.9.1 pyhd3eb1b0_0 dash-table 4.11.3 pyhd8ed1ab_0 conda-forge dask 2021.7.0 pyhd8ed1ab_0 conda-forge dask-core 2021.7.0 pyhd8ed1ab_0 conda-forge datacube 1.8.3 py_1 conda-forge decorator 5.0.9 pyhd3eb1b0_0 distributed 2021.7.0 py39hcbf5309_0 conda-forge expat 2.4.1 h6c2663c_2 fiona 1.8.19 py39h9f1b043_0 conda-forge flask 1.1.2 pyhd3eb1b0_0 flask-compress 1.10.1 pyhd3eb1b0_0 folium 0.12.0 pyhd8ed1ab_1 conda-forge fontconfig 2.13.1 h1989441_1005 conda-forge freetype 2.10.4 hd328e21_0 freexl 1.0.6 h2bbff1b_0 fsspec 2021.7.0 pyhd8ed1ab_0 conda-forge future 0.18.2 py39haa95532_1 gdal 3.2.2 py39h6795fcd_7 conda-forge geocube 0.0.17 pyhd8ed1ab_0 conda-forge geopandas 0.9.0 pyhd8ed1ab_0 conda-forge geos 3.9.1 h6c2663c_0 geotiff 1.6.0 ha8a8a2d_6 conda-forge gettext 0.19.8.1 h1a89ca6_1005 conda-forge greenlet 1.1.0 py39h415ef7b_0 conda-forge h3-py 3.7.3 py39h415ef7b_1 conda-forge hdf4 4.2.15 h0e5069d_3 conda-forge hdf5 1.10.6 nompi_h5268f04_1114 conda-forge heapdict 1.0.1 py_0 conda-forge icc_rt 2019.0.0 h0cc432a_1 icu 68.1 h6c2663c_0 idna 2.10 pyhd3eb1b0_0 importlib-metadata 4.6.1 py39hcbf5309_0 conda-forge iniconfig 1.1.1 pyhd3eb1b0_0 intel-openmp 2021.3.0 haa95532_3372 itsdangerous 2.0.1 pyhd3eb1b0_0 jbig 2.1 h8d14728_2003 conda-forge jinja2 3.0.1 pyhd3eb1b0_0 joblib 1.0.1 pyhd3eb1b0_0 jpeg 9d h8ffe710_0 conda-forge jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge kealib 1.4.14 h96bfa42_2 conda-forge kiwisolver 1.3.1 py39hd77b12b_0 kombu 5.1.0 py39hcbf5309_0 conda-forge krb5 1.19.1 h5b6d351_0 lark-parser 0.11.3 pyhd8ed1ab_0 conda-forge lerc 2.2.1 hd77b12b_0 libblas 3.9.0 9_mkl conda-forge libboost 1.67.0 hd9e427e_4 libcblas 3.9.0 9_mkl conda-forge libclang 11.1.0 default_h5c34c98_1 conda-forge libcurl 7.77.0 h789b8ee_0 conda-forge libdeflate 1.7 h2bbff1b_5 libffi 3.3 h0e60522_2 conda-forge libgdal 3.2.2 ha6cada3_7 conda-forge libglib 2.68.1 h1e62bf3_0 conda-forge libiconv 1.16 he774522_0 conda-forge libkml 1.3.0 he5f2a48_4 liblapack 3.9.0 9_mkl conda-forge libnetcdf 4.8.0 nompi_hee75d68_101 conda-forge libpng 1.6.37 h2a8f88b_0 libpq 13.3 hfcc5ef8_0 conda-forge libpysal 4.5.1 pyhd3eb1b0_0 librttopo 1.1.0 hb340de5_6 conda-forge libspatialindex 1.9.3 h6c2663c_0 libspatialite 5.0.1 h762a7f4_5 conda-forge libssh2 1.9.0 h7a1dbc1_1 libtiff 4.3.0 h0c97f57_1 conda-forge libwebp-base 1.2.0 h2bbff1b_0 libxml2 2.9.12 h0ad7f3c_0 libxslt 1.1.34 he774522_0 libzip 1.7.3 hfed4ece_0 conda-forge locket 0.2.0 py_2 conda-forge lxml 4.6.3 py39h9b66d53_0 lz4-c 1.9.3 h2bbff1b_0 markupsafe 2.0.1 py39h2bbff1b_0 matplotlib 3.4.2 py39hcbf5309_0 conda-forge matplotlib-base 3.4.2 py39h581301d_0 conda-forge mkl 2021.2.0 hb70f87d_389 conda-forge mkl-service 2.4.0 py39h2bbff1b_0 mock 4.0.3 pyhd3eb1b0_0 more-itertools 8.8.0 pyhd3eb1b0_0 msgpack-python 1.0.2 py39h2e07f2f_1 conda-forge munch 2.5.0 py_0 netcdf4 1.5.7 nompi_py39hc20e565_100 conda-forge networkx 2.5 py_0 conda-forge numexpr 2.7.3 py39hb80d3ca_1 numpy 1.20.2 py39h6635163_0 conda-forge olefile 0.46 py_0 openjpeg 2.4.0 h48faf41_0 conda-forge openssl 1.1.1k h8ffe710_0 conda-forge osmnx 1.1.1 pyhd8ed1ab_0 conda-forge packaging 21.0 pyhd3eb1b0_0 pandas 1.2.4 py39h2e25243_0 conda-forge paramiko 2.7.2 pyh9f0ad1d_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge patsy 0.5.1 py39haa95532_0 pcre 8.45 hd77b12b_0 pillow 8.3.1 py39h4fa10fc_0 pip 21.1.3 py39haa95532_0 pixman 0.40.0 he774522_0 plotly 4.14.3 pyh44b312d_0 conda-forge pluggy 0.13.1 py39haa95532_0 poppler 21.03.0 h9ff6ed8_0 conda-forge poppler-data 0.4.10 0 conda-forge postgresql 13.3 h1c22c4f_0 conda-forge progressbar2 3.37.1 py39haa95532_0 proj 8.0.1 h1cfcee9_0 conda-forge prompt-toolkit 3.0.19 pyha770c72_0 conda-forge prompt_toolkit 3.0.19 hd8ed1ab_0 conda-forge psutil 5.8.0 py39hb82d6ee_1 conda-forge psycopg2 2.9.1 py39h0878f49_0 conda-forge py 1.10.0 pyhd3eb1b0_0 pycparser 2.20 py_2 pygam 0.8.0 py_0 conda-forge pygeos 0.10 py39haadaec5_0 conda-forge pynacl 1.4.0 py39hb3671d1_2 conda-forge pyopenssl 20.0.1 pyhd3eb1b0_1 pyparsing 2.4.7 pyhd3eb1b0_0 pyproj 3.1.0 py39ha996c60_3 conda-forge pyqt 5.12.3 py39hcbf5309_7 conda-forge pyqt-impl 5.12.3 py39h415ef7b_7 conda-forge pyqt5-sip 4.19.18 py39h415ef7b_7 conda-forge pyqtchart 5.12 py39h415ef7b_7 conda-forge pyqtwebengine 5.12.1 py39h415ef7b_7 conda-forge pyrsistent 0.17.3 py39hb82d6ee_2 conda-forge pysocks 1.7.1 py39haa95532_0 pytables 3.6.1 py39h42e6cd8_3 conda-forge pytest 6.2.4 py39haa95532_2 pytest-runner 5.3.1 pyhd3eb1b0_0 python 3.9.5 h6244533_3 python-dateutil 2.8.2 pyhd3eb1b0_0 python-utils 2.5.6 py39haa95532_0 python_abi 3.9 1_cp39 conda-forge pytz 2021.1 pyhd3eb1b0_0 pyyaml 5.4.1 py39h2bbff1b_1 qt 5.12.9 h5909a2a_4 conda-forge rasterio 1.2.6 py39hb4f844e_1 conda-forge rasterstats 0.14.0 py_0 redis-py 3.5.3 pyh9f0ad1d_0 conda-forge requests 2.25.1 pyhd3eb1b0_0 retrying 1.3.3 py_2 rioxarray 0.5.0 pyhd8ed1ab_0 conda-forge rtree 0.9.7 py39h2eaa2aa_1 scikit-learn 0.24.2 py39he931e04_0 conda-forge scipy 1.6.2 py39h66253e8_1 setuptools 52.0.0 py39haa95532_0 shapely 1.7.1 py39h90c6b7e_4 conda-forge simplejson 3.17.3 py39h2bbff1b_2 six 1.16.0 pyhd3eb1b0_0 snuggs 1.4.7 py_0 sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.2.1 pyhd3eb1b0_0 sqlalchemy 1.4.22 py39hb82d6ee_0 conda-forge sqlite 3.36.0 h2bbff1b_0 sshtunnel 0.3.1 pyhd8ed1ab_0 conda-forge statsmodels 0.12.2 py39h2bbff1b_0 tbb 2021.2.0 h59b6b97_0 tblib 1.7.0 pyhd8ed1ab_0 conda-forge threadpoolctl 2.2.0 pyhb85f177_0 tiledb 2.3.2 h78dabda_0 conda-forge tk 8.6.10 he774522_0 tobler 0.8.2 pyhd8ed1ab_0 conda-forge toml 0.10.2 pyhd3eb1b0_0 toolz 0.11.1 py_0 conda-forge tornado 6.1 py39h2bbff1b_0 tqdm 4.61.2 pyhd3eb1b0_1 typing_extensions 3.10.0.0 pyha770c72_0 conda-forge tzdata 2021a h52ac0ba_0 urllib3 1.26.4 pypi_0 pypi vc 14.2 h21ff451_1 vine 5.0.0 pyhd8ed1ab_1 conda-forge vs2015_runtime 14.27.29016 h5e58377_2 wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge werkzeug 1.0.1 pyhd3eb1b0_0 wheel 0.36.2 pyhd3eb1b0_0 win_inet_pton 1.1.0 py39haa95532_0 wincertstore 0.2 py39h2bbff1b_0 xarray 0.18.2 pyhd8ed1ab_0 conda-forge xerces-c 3.2.3 ha925a31_0 xz 5.2.5 h62dcd97_0 yaml 0.2.5 he774522_0 zict 2.0.0 py_0 conda-forge zipp 3.5.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h62dcd97_4 zstd 1.5.0 h6255e5f_0 conda-forge

source_target_vars.zip