Looking for larger datasets

[zeffii]

in order for this to progress from my part, i'd need a number of representative datasets (small, medium, large) to include ways that would let users easily control cherrypicking, passing a bounding box, etc.

If each gis type file has a variety of potential structures (keys and values) then it may be more important to offer a way to easily apply a filter, with some kind of script node disigned for digging into these kinds of datatypes.

[Marcus-Richmond]

@zeffii Sure thing, check out this example dataset: https://github.com/Marcus-Richmond/example-datasets/blob/main/NaturalEarthTestData.zip. It's just a gpkg of some layers from the Natural Earth dataset. Included are 3 layers each of cities (points), rivers (lines), and countries (polygons) across the globe. The '110m' layers are the lowest resolution, '50m' is medium resolution, and '10m' is the highest resolution.

I definitely see each gis file (kind of) having a variety of potential structures. The structure is the same in that all these files are really just tabular data, but differ in that the attributes (columns) of the tabular data will differ. For example, the cities and countries files I think have a 'population' attribute, but the rivers obviously won't have this. And, other data sources, like say a country's census organization, could provide their own cities point data, but might name the population attribute 'Pop' or something else, differing from Natural Earth or other sources. So, I think some note to show and pick out attributes could be very useful.

Also, thanks for continuing to work on this, I still really appreciate all the help. The last few days/week were busier than expected, but I think this week will allow me more time to commit to this. Looking forward to it.

[zeffii]

going for simpler introspection..

[zeffii]

https://github.com/Marcus-Richmond/example-datasets/blob/main/NaturalEarthTestData.zip. It's just a gpkg of some layers from the Natural Earth dataset. Included are 3 layers each of cities (points), rivers (lines), and countries (polygons) across the globe. The '110m' layers are the lowest resolution, '50m' is medium resolution, and '10m' is the highest resolution

forgive my ignorance, but how does one find those? layers?

[zeffii]

geopandas has demo data built in. nice.

"""
in gis_path FP
out coordinates s
"""

try:
    import geopandas as gpd

    world_gdf = gpd.read_file(
    gpd.datasets.get_path('naturalearth_lowres')
    )
    geom = world_gdf.to_dict()['geometry']
    for idx, multi_polygon in geom.items():
        if str(multi_polygon.type) == "Polygon":   # don't do this.
            polygon = multi_polygon
            array_2d = np.array(polygon.__geo_interface__['coordinates'][0])
            N = array_2d.shape
            zeros = np.zeros(N[0])
            array_3d = np.c_[ array_2d, zeros]
            coordinates.append(array_3d)

except Exception as err:
    print(err)

this is intentionally skipping "MultiPolygon".

[zeffii]

i imagine there will be some usecase for shapely.ops.triangulate(geom, tolerance=0.0, edges=False)

[Marcus-Richmond]

@zeffii best way to view layers in a geopackage? - use QGIS. Although it's a full fledged GIS software it can be used to quickly view layers. I think this video is a good intro too.

Even cooler to see the built in test data, of course it is the same as the example data I put together haha.

I didn't know shapely had that function either, definitely will be useful. I think that's the most exciting part about this too, getting into these libraries and accessing all the functions to use in Sverchok.

[zeffii]

unfortunately shapely.ops.triangulate does not do what I hoped. tesselation from arbitrary polygon shapes. luckily Blender does provide such an operation (and it returns indices from the original point list)

[Marcus-Richmond]

So, I forgot if I asked this already, but can we pass a geodataframe between nodes? If we could have a socket in a gis import node that was the geodataframe, we could connect that to a shapely.ops.triangulate node (I'm assuming this function takes a geodataframe). In other words, if we could have a geodataframe socket, that could make bringing in functions from other libraries a whole lot easier.

[zeffii]

you can pass anything through a SvStringsSocket, as long you give the socket a suitable name that reflects its content, the user will know which node-sockets it can string together.

The reason we have a variety of sockets in Sverchok is to not just to make it clearer to the user which sockets can be connected to eachother in a meaningful way, but if you look at the socket classes some sockets have additional functions and functionality. I'd like to avoid making sockets for the time being. Most sockets will only ever have one kind of datastream, like Color and Coordinates and Matrices, it makes sense to have dedicated sockets for those, there are no surprises in their homogenous data types.

it might turn out that geodataframe is also worthy of it's own socket, i'll read more about them today.