elasticmachine
commented
2 years ago Pinging @elastic/kibana-gis (Team:Geo)
Open Heenawter opened 2 years ago
elasticmachine
commented
2 years ago Pinging @elastic/kibana-gis (Team:Geo)
nreese
commented
2 years ago @Heenawter Would it be possible to get a view the network tab during a failure? Can you show the response for the last /internal/file_upload/import request?
nreese
commented
2 years ago Shapefile with issues can be downloaded at https://www12.statcan.gc.ca/census-recensement/alternative_alternatif.cfm?l=eng&dispext=zip&teng=ler_000b16a_e.zip&k=%20%20%20%2030118&loc=http://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/ler_000b16a_e.zip.
I have isolated the shape causing problems and its 50619861 bytes which is not the largest in the file. The shape is MultiPolygon with 15770 polygons. This seems to be causing lots of problems. Other large MultiPolygons in the file top out around 1500 polygons.
The indexing timeouts are coming from Elasticsearch. See https://github.com/elastic/elasticsearch/issues/69890 for details about long indexing times for complex shapes.
Maybe we could do something client side where we avoid sending "complex" polygons and multipolygons to elasticsearch. Then in the results page, show a list of large features not indexed and allow user to index each individually so that way the bulk of the file upload can complete but large features do not torpedo the entire process.
@jsanz @nickpeihl thoughts?
nickpeihl
commented
2 years ago One thing I've noticed with this dataset is that the shapefile loader returns coordinates with an absurdly high precision of 14 decimal places (1.1 nanometers). There is absolutely no reason to ever have this level of coordinate precision (see relevant xkcd).
Generally, I view any coordinate precision higher than six decimal places as dubious. We might consider truncating the number of decimal places before sending to Elasticsearch. That may help with some of the bottleneck.
jsanz
commented
2 years ago Generally, I view any coordinate precision higher than six decimal places as dubious. We might consider truncating the number of decimal places before sending to Elasticsearch. That may help with some of the bottleneck.
In fact, RFC7946 for GeoJSON suggests six decimals.
For the example shapefile provided with just 76 multipolygons that make up to 27 thousand parts, this mapshaper command generates a valid shapefile that can be indexed without isuses:
$ mapshaper -i ler_000b16a_e.shp -proj crs=EPSG:4326 \
-snap precision=0.000001 -clean \
-simplify percentage=99% -explode \
-o ler_000b16a_e_4326.shp
[clean] Removed 42 / 151 slivers using 0.01+ sqkm variable threshold
[clean] Detected DBF text encoding: win1252
[clean] Retained 76 of 76 features
[explode] Exploded 76 features into 27,060 features
[o] Wrote ler_000b16a_e_4326.shp
[o] Wrote ler_000b16a_e_4326.shx
[o] Wrote ler_000b16a_e_4326.dbf
[o] Wrote ler_000b16a_e_4326.prjThe command:
$ ogrinfo -summary ler_000b16a_e_4326.shp ler_000b16a_e_4326
INFO: Open of `ler_000b16a_e_4326.shp'
using driver `ESRI Shapefile' successful.
Layer name: ler_000b16a_e_4326
Metadata:
DBF_DATE_LAST_UPDATE=2022-06-20
Geometry: Polygon
Feature Count: 27060
Extent: (-141.018073, 41.681435) - (-52.619409, 83.135503)
Layer SRS WKT:
GEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["latitude",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["longitude",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
ERUID: String (4.0)
ERNAME: String (84.0)
PRUID: String (2.0)
PRNAME: String (51.0)
jsanz
commented
2 years ago @nreese @nickpeihl do you think it'd be useful to add a new section to our troubleshoot guide with the instructions on how to clean/simplify/snap shapefiles and geojson files using mapshaper similar to my previous comment?
nickpeihl
commented
2 years ago do you think it'd be useful to add a new section to our troubleshoot guide with the instructions on how to clean/simplify/snap shapefiles and geojson files using mapshaper similar to my previous comment?
Anything we can do to help via our documentation is good.
I have also started working on a PR to reduce the coordinate precision which should help immensely.
rashmivkulkarni
commented
2 years ago Encountered the same blank screen while uploading the data ( screenshot above) . Zoomed with Hannah and Nick to debug more- reopening this issue for more clarity . This might be something related to browser.
nickpeihl
commented
2 years ago WebGL context was lost. maps.chunk.3.js:18:627388 TypeError: this.props.importResults.error.error is undefined
We should look closely at this error from the original report as losing the WebGL context would cause the map to disappear.
The blank screen issue does not appear to happen consistently. But it is possible that clicking away to another tab while the upload is running may trigger the blank screen. More investigation is needed.
rashmivkulkarni
commented
2 years ago Loaded the above data , went to discover and it showed the following error message
Can't store an async search response larger than [10485760] bytes. This limit can be set by changing the [search.max_async_search_response_size] settingTo get past it, I did this in dev tools
PUT _cluster/settings
{
"persistent" : {
"search.max_async_search_response_size" : "20mb"
}
}Now , when I went back to maps app, it did load the data set successfully, ( although for a sec- it did show "page unresponsive" ) and then continued loading the data.
nreese
commented
2 years ago This issue highlights an edge case of uploading/using extremely large/complex shapes. They are not going to upload well and they are not going to render well. I think the solution is to not try uploading any shape that is larger then 1 or 2 mbs and instead message users to simplify shapes before ingest.
I just don't see how a shape that is 20mb large can be used to provide any meaningful visualization that can't be accomplished with a shape that is broken into smaller features or a simplified shape that has less vertices.
jsanz
commented
2 years ago Unfortunately this kind of shapes are not that uncommon. This is a nice post on the topic, applied to Postgres but the issue is the same, and how dividing complex polygons into smaller pieces can help indexing, rendering and analyzing.
Our EMS World Countries dataset has for example a polygon (Canada) with 409 parts and a total of 53778 vertices for that single document, around 19MB as a single GeoJSON feature. This table shows for that dataset the top countries per vertices count and the number of parts each geometry has.
And this is a generic dataset, for other data types it can be way worse, I've seen datasets with just a few documents and gigabytes of size.
In any case I agree we should notify the user about this situation and help them to adapt their data to Elasticsearch with some troubleshooting documentation #135319.
nickpeihl
commented
2 years ago We could also consider simplifying client-side complex shapes using Visvalingam or Douglas-Peucker algorithms before sending the shapes to Elasticsearch. If we do this, I would suggest making this an option (on by default) in the UI. Something like:
elasticmachine
commented
2 years ago Pinging @elastic/kibana-presentation (Team:Presentation)
Kibana version: 8.3.0 SNAPSHOT
Browser: Firefox
Describe the bug: Sometimes, when uploading a large shape file, the maps interface gets stuck in a perpetual loading state during indexing - however, despite it apparently being stuck, it seems as though the index is properly created anyway? The only step that seems to be missing is creating the data view. Here's a video demonstrating this:
https://user-images.githubusercontent.com/8698078/172693244-793513ff-67f2-4852-bd1b-4988a141c829.mov
I left the above example running for a good 5 minutes, and it was stuck at
98.7% completethat entire time - when I let it run for even longer, I eventually got redirected to a blank page:At this point, this is what my console looked like:
Steps to reproduce:
Download a large shape file - for example, I was able to produce this bug three times via the following shape file of Canadian provinces:
https://www12.statcan.gc.ca/census-recensement/alternative_alternatif.cfm?l=eng&dispext=zip&teng=ler_000b16a_e.zip&k=%20%20%20%2030118&loc=http://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/ler_000b16a_e.zip
Upload the shape file to a new map
Upload may or may not get stuck at
Writing to indexstage - note that this doesn't seem to happen consistently, so I'm not sure what exactly is causing it. But it's happening often enough that it's not a one-off - for example, in my testing over the last few days, I've had it happen to me four of five times with different shape files.Provide logs and/or server output (if relevant):