R arcgis fails to upload complex geometries in reliable way

MadGeologist commented 6 months ago

Hi, so I have an issue where complex geometries fail to draw when they are uploaded to AGOL using the publish_layer() feature. It appears that complex geometries have a significant difficulty loading when published to arcgis online.

Here is an example with some code that demonstrates this glaring issue in R while trying to pull down a public dataset involving CCRI:

`### Login to AGOL token_AGOL <- auth_user( username = "U", password = "PW", host = arc_host(), expiration = 21600 ) set_arc_token(token_AGOL)

Pull Down CCRI

CCRI_URL = "https://services.arcgis.com/XG15cJAlne2vxtgt/arcgis/rest/services/Counties_FEMA_Community_Resilience_Challenges_Index_(CRCI)/FeatureServer/54"

Open Layer

CCRI_Layer <- arc_open(CCRI_URL) CCRI_Layer

Use Arc Select to isolate the data

NY_CCRI_DF <- arc_select(CCRI_Layer,where = "STATEFP = '36'") NY_CCRI_DF

Publish this data to arcgis online

res <- publish_layer(NY_CCRI_DF, "R-arcgisBridge Upload Test") res '

Unfortunately, once the shapefile is uploaded it collapses and fails to draw any of the more complex geometries - i.e. the entirety of long island

Within the Rstudio environment ,there is no issue with plotting the data.... via the command plot(NY_CCRI_DF)

The questions I have are:

Can this issue be reproduced
Are there any additional caveats to be considered when uploading data frames into AGOL
If not, can a fix be issued? These are similar problems that occur with the arcgis api for python. As a positive critique I think there needs to be much more stress-testing with the R/python arcgis API suite in general. It's great to see examples of these features tested out on the square state of Colorado at the User Summit, but if it can't handle the very well-known region of Long Island this shouldn't have been released.

JosiahParry commented 6 months ago

Thank you for the report! I was actually just investigating an issue similar to this last week which (frustratingly) resolved itself on its own.

"It's great to see examples of these features tested out on the square state of Colorado at the User Summit..."

🤣 I'll have you know that I have never tested on Colorado. I prefer the historic and curvy Guerry dataset.

I think this is not necessarily an issue with {arcgis} but might be an issue with the webviewer and the json that is being generated. Every request succeeds.

We can see that the featurecollection json is available and viewable

The json is a published item:

BUT when I read the item back into memory, that location and a few others seems corrupted. I think this is an issue specifically with MultiPolygons

The json of the one item that ~~sucks~~ is breaking things

```json {"hasZ":false,"hasM":false,"rings":[[[-8019103.30813177,5049264.76842296],[-8018894.9380365,5050096.24323182],[-8018759.10884689,5050346.05271808],[-8018432.75932931,5050946.28301834],[-8018280.45429148,5051226.41910717],[-8017906.71889885,5051530.25073903],[-8017840.5194716,5051584.06910913],[-8017816.42129743,5051833.97336458],[-8017728.75476833,5052743.15339829],[-8017610.75601943,5052397.0131557],[-8017608.59289739,5052401.80365809],[-8017307.01832549,5053069.69964814],[-8017284.92378632,5053118.63408699],[-8016158.40970021,5052534.41919369],[-8015879.73733903,5052389.90387139],[-8015300.54183181,5051716.32172398],[-8014374.47475588,5052324.88378964],[-8014221.07643938,5053665.21797449],[-8014014.57881638,5053928.1520597],[-8013275.41711041,5053953.92741213],[-8012846.40139546,5053796.98385283],[-8012783.8301277,5053774.09405428],[-8012153.0936383,5053868.30627311],[-8012127.4853633,5053889.76125755],[-8011541.7177467,5054380.54134213],[-8011443.20904103,5054463.07845878],[-8011055.24910374,5054393.4063699],[-8010550.377649,5054302.73926812],[-8010542.41142843,5054301.30869602],[-8010247.53563195,5054370.65561177],[-8009870.52075765,5054459.32031474],[-8009756.27292503,5054486.18872456],[-8009077.00117752,5054377.45273075],[-8008720.89002191,5054719.8121999],[-8008254.795222,5055399.53008762],[-8008150.65010085,5055392.30367121],[-8007238.4479217,5055329.00787051],[-8006347.5058598,5055507.21415527],[-8006251.82291565,5055526.35290221],[-8006992.65436647,5054917.88547237],[-8007072.0252226,5054388.71164228],[-8007796.49265067,5053816.75634637],[-8007811.72350598,5053814.93285057],[-8008483.99070174,5053734.44852746],[-8008765.30651539,5053700.76948311],[-8009022.23189611,5053183.80819277],[-8009691.37358918,5053039.24169546],[-8009981.81702602,5053055.78610856],[-8010474.06123818,5053083.82588963],[-8011007.83836178,5053187.21481517],[-8011384.76622245,5052721.82511369],[-8012273.87521783,5052223.28139418],[-8012696.52598645,5051940.61905197],[-8013314.7128091,5051527.19989012],[-8014095.50790989,5050899.43667667],[-8014415.32891455,5050243.283204],[-8014494.19352438,5050194.73178813],[-8014966.77950967,5049903.79927422],[-8015277.38731744,5049712.58817984],[-8015338.61638091,5049705.31567003],[-8015345.40568348,5049704.50924126],[-8015666.33775885,5049666.39064432],[-8016136.28481877,5050066.05197715],[-8016136.44009265,5050066.18407868],[-8016909.44289572,5050091.35674556],[-8017816.80832653,5049410.67935521],[-8019105.10924833,5049257.58160524],[-8019103.30813177,5049264.76842296]]],"spatialReference":{"wkid":4326}} ```

I appreciate you taking the time to make a reproducible issues. Its impossible for me to single handedly know and anticipate all of the places that issues may occur.

MadGeologist commented 6 months ago

Thanks for the lightning-fast response and you're absolutely correct, products like these are a huge undertaking - it's impossible to see everything caveat, and I would've 1000% failed if I had to make this tool myself - so thanks for a bunch for the work you do! I'll stop being an angry geologist haha.

After doing some digging I can concur that the regions that failed to load where in part to nested polygons.

^^ is an example of a region where it imploded

^^ is an example of a region where it draws properly

I wonder if there's a way to un-nest the jsons/make them into a singular list (I have no idea what I'm talking about)

JosiahParry commented 6 months ago

I wonder if there's a way to un-nest the jsons/make them into a singular list (I have no idea what I'm talking about)

you're not actually far off! Theoretically we could use sf::st_cast(geometry) to expand them into multiple. But then we lose the fact that each feature is one row!

the esriGeometryPolygon object, as far as I can tell does not support multiple polygons

Source

{
  "hasZ": <true | false>,
  "hasM": <true | false>,
  "rings": [
    [
      [<x11>, <y11>, <z11>, <m11>],
      [<x1N>, <y1N>, <z1N>, <m1N>]
    ],
    [
      [<xk1>, <yk1>, <zk1>, <mk1>],
      [<xkM>, <ykM>, <zkM>, <mkM>]
    ]
  ],
  "spatialReference": {<spatialReference>}
}

I've reached out to some internal resources to see what i can find out!

MadGeologist commented 6 months ago

Hmm interesting, I might try a few things to see if there's a way to keep track of the original row when "de-listing" or "combining" the rings.

Thanks for reaching out to internal support, one interesting thing that I've noticed is that the arcgis api for python (versions 2.0 >) found a workaround for multi-polygons as I've been able to upload those without facing artifact issues.

Here's an example in case this reference is of any help:

### Libraries and installation
##############################

import arcgis
import numpy as np
import os 
import pandas as pd 
from pandas import json_normalize
import json
#import shapefile
from arcgis import GIS
import arcgis
import geopandas as gpd 
from geopandas import GeoSeries, GeoDataFrame
from arcgis.features import FeatureCollection,FeatureLayer,FeatureSet
from arcgis.features import GeoAccessor, GeoSeriesAccessor,FeatureLayerCollection

### Login to AGOL 
###################
# Set up keybase enviornment 
ENV_SCOPE = os.getenv("p_env_secretscope_name")

# Extract the username and password 
un  = dbutils.secrets.get(scope=ENV_SCOPE, key="sec-ago-headless-username")
pw = dbutils.secrets.get(scope=ENV_SCOPE, key="sec-ago-headless-password")
# Login 
gis = GIS("https://fema.maps.arcgis.com",
          username = un,
          password = pw
         )

print("Logged into AGOL")

# Pull down the CCRI Counties:
CCRI_Counties = gis.content.get(itemid ="ff17676850114091b28cb4c502e9fb4e")
CCRI_Counties_Layer = CCRI_Counties.layers[0].query(where = "STATEFP = '36'")

# Processs the data, convert to geojson

CCRI_Counties_string = CCRI_Counties_Layer.to_geojson
CCRI_Counties_gsjon_dict = json.loads(CCRI_Counties_string)
print("Converted Item to JSON")

# Convert the geojson to dataframe
CCRI_Counties_gdf =  gpd.GeoDataFrame.from_features(CCRI_Counties_gsjon_dict['features'])
CCRI_Counties_gdf = gpd.GeoDataFrame(CCRI_Counties_gdf, geometry='geometry')
CCRI_Counties_gdf = CCRI_Counties_gdf.set_crs(epsg=3857, inplace=False,allow_override=True)
CCRI_Counties_gdf = CCRI_Counties_gdf.to_crs(epsg=4326)

# Cast projection 
CCRI_Counties_gdf = CCRI_Counties_gdf.to_crs(epsg=4263)

# Convert into a spatially enabled data frame 
**CCRI_Counties_SDF = GeoAccessor.from_geodataframe(CCRI_Counties_gdf,column_name ="geometry")**

# Cast the projection 
CCRI_Counties_SDF.spatial.set_geometry(col="geometry",sr="4263",inplace=True)

# Send to arcgis online 

**CCRI_Counties_SDF.spatial.to_featurelayer('Gitlab Demonstration - Delete Later')**

Gets them nested geometries in with no problem:

It looks like the answer is inside the 'to_featurelayer()' function it must be doing some translation to work with the glitchy webview json haha or maybe converting the geopandas into a spatially enabled dataframe (is there a R equivalent?) - having this work in R would be awesome for our team as we have an easier time automating tasks in R (long story).

JosiahParry commented 6 months ago

@MadGeologist would you be able to share the json from the SEDF for the feature that is causing us problems?

MadGeologist commented 6 months ago

Sure thing, here's the JSON from the SEDF that was sent up into the AGOL platform. SpatiallEnabledDataFrameExample.json

Let me know if this link works!

MadGeologist commented 5 months ago

I hope all is well, any updates on the fate of this? :)