search

nortikin / sverchok

Sverchok
http://nortikin.github.io/sverchok/
GNU General Public License v3.0
2.26k stars 233 forks source link

(Question) Interested in developing custom nodes/scripts for Sverchok. Where to start? And best practices? #4418

Closed Marcus-Richmond closed 2 years ago

Marcus-Richmond commented 2 years ago

Goal/preface

I'm interested in importing GIS (geographic information systems) data into Blender to create informative and interesting visualizations. Although GIS and 3D Graphics are separate fields, they both rely heavily on data visualization and could benefit from having more integration.

Also, if this is not the appropriate place to pose these questions/discussion I'll move it. I thought this discussion was too broad for stack exchange.

Why?

GIS data often have multitudes of attributes associated with them, but the software that is traditionally used to visualize these data have relatively limited visualization options. For example, QGIS has tons of data driven options for labeling and styling data, but this is essentially limited to 2D map views.

Consider this example

Imagine you have fulfilled your lifelong dream of becoming a geologist and are trying to visualize and display earthquake data across the planet. A program like QGIS would give you lots of options, but most of these options would be limited to a 2D map. Rather, you could try visualizing this data in 3D, by representing each earthquake as a point, and color that point based on the earthquake's magnitude. Well, conveniently enough, this is an actual visualization, viewable on sketchfab here

What does this have to do with Sverchok?

Also imagine if you were trying to add to the above earthquake visualization by doing something like animating the points, or making each point's size dependent on the earthquake's magnitude. I think you can see where I'm going with this - if you could bring this earthquake data into Blender, and designed the visualization parametrically with Sverchok, the options at your disposal would be endless.

But can't you already bring GIS data into Blender?

There are a couple methods, like using the addons: BlenderGIS or Blender-OSM, or using elevation rasters directly in blender. However, these methods are geared towards working with rasters or readily available (OSM) vector data. These methods lack an easy way to work with the attributes of GIS data, so making a visualization like the earthquake visualization described above, would be difficult and not straightforward.

Ok, so how do you bring GIS data into Blender?

With a lot of help from this community, I managed to write a few scripts that can bring GIS data into Blender through the Sverchok interface, and saved them here: https://github.com/Marcus-Richmond/sverchok-gis-scripts.

For instance, building upon our earthquake example, we can take earthquake records with latitude/longitude coordinates, and convert them to polar coordinates to get a 3D representation:

image

image

And we can keep building on this example, adding outlines of countries for context:

image

Let's also start using that attribute data that I've claimed is so important, how about we use the Depth attribute of the earthquakes to control the rho value:

image

image

Cool, but what's your question?

Right now, these scripts are just scripts to plug into the SNLite node, but I'm interested in adding in these functions (import gpkg layer geometry, import attributes, convert CRS, etc.) as new nodes in Sverchok. I read through this page, this page, and this page, which were easy enough to understand, so I wrote one of these scripts (import attributes) as a custom node, saved here.

So, I can go from the basic scripts to creating custom nodes, but now how do I make these custom nodes into an addon for Sverchok? I know the answer to this question is likely very complex, and to get a better understanding of blender addons and sverchok I have looked through:

Again, my question is essentially "how do I make an addon for Sverchok?" And if the answer is that I need to understand more about object-oriented programming, I understand and will keep trying to expand my knowledge on this topic. I'm hoping I can at least know what may be specific to Sverchok addons that I wouldn't learn through generic OOP or Blender addon tutorials/courses.

Last Comments

I also want to thank everyone in the Sverchok team and community. The resources put together by all of you are amazing, and your willingness to respond to user questions is impressive to say the least.

Here is the data/blend file I've been working with: https://github.com/Marcus-Richmond/example-datasets/blob/main/BlenderAndGIS_20220402.zip (the links will undoubtedly be broken in the blend file, but shouldn't be difficult to fix). The scripts I've been using rely upon geopandas, so you will need to install that for any of this to work.

I am more than happy to get suggestions/constructive criticism on any of this code. If you want to help out and write some GIS nodes/scripts that would be awesome too!

Durman commented 2 years ago

It would be nice if the question was covered in the documentation. Probably @portnov can help.

portnov commented 2 years ago

You can use existing extensions as examples: https://github.com/nortikin/sverchok/wiki/Sverchok-Extensions. In particular, in the readme of https://github.com/vicdoval/sverchok-open3d it is stated that that repo can be used as a template for new addons :)

In general, what such addon does is

Everything else (such as settings) is either standard functionality of common Blender addon (you can refer to any tutorials on how to create a blender addon, I believe there should be a number of them), or automation of nodes registration, to do not explicitly call register() for each node.

The most interesting things (apart from nodes themselves) in such addons are happening in __init__.py. Everything else is either imported from __init__.py, or explicitly called from there.

zeffii commented 2 years ago

i'm willing to help you with Sverchok GIS nodes. (not sure how much time i'll have)

Marcus-Richmond commented 2 years ago

Thanks for the responses @Durman, @portnov, and @zeffii.

@portnov - Good to know that I can use https://github.com/vicdoval/sverchok-open3d as a template! (Thanks @vicdoval too!)

@zeffii - Great! I've sent a collaborator invite, no worries if you are busy and can't find the time too.

Next steps

So, this brings up another question - Is this thread the best place to keep asking questions about sverchok addon development? Does anyone have a preference for here or elsewhere?

I am looking into the other Sverchok addons and trying to copy/modify these to start GIS nodes. I was even able to get this version of GIS nodes to load into blender ok. I started with the Open3D template, couldn't get it working at first, but after reviewing and adding code from the other extensions, was finally able to load it into blender without any errors.

But, what I can't figure out now, is that I cannot see the new nodes. For example, there is not an extra node menu with Shift-A for GIS nodes. I also can't find the new nodes when searching. I am planning to keep looking into the other extensions and their code, but if anyone has time/interest to look at this version please feel free to do so. Anyway, I just wanted to put this issue out there in case I should keep adding questions about this addon to this thread.

zeffii commented 2 years ago

it's certainly a grey area where to continue this, it will depend on the kinds of questions - but for the time being most questions will be related directly to getting the open3d template to cooperate. I will be checking your repo daily and answer any and all questions at my first possible convenience - the next 2 weeks are looking good.

Marcus-Richmond commented 2 years ago

@zeffii perfect, that will be a huge help.

Please see the most recent version here: https://github.com/Marcus-Richmond/sverchok-gis-nodes. This version will install without issues, but for some reason the nodes are not actually showing up in the add or search menus.

Durman commented 2 years ago

We could enable discussions for this repository because they are definitely better than issues for questions. Some answers can be copied into the documentation.

victorcalixto commented 2 years ago

Hi @Marcus-Richmond, That is really great =). It was a surprise, because I have been working in a prototype of GIS tools for Sverchok to use in my thesis. I am in from Architecture and Urban Design field. The GIS module will be one module of a toolkit for Data-Driven Urban Placemaking. I am also using geopandas to import shapefiles. Also, I am using osmnx to download OSM data as Shapefiles and leafmap and voila to have interactive webapps from Sverchok nodes. Maybe we have different goals, I understand that you are interested in explored different GIS visualisation methods in Sverchok for Geology, and I am trying to explore how to integrate different datasets to assist Urban Design process, but maybe is worth to join forces, what do you think? Moreover, I had the same question that you had about how to pack up everything as a Sverchok plugin, but I was expecting to solve it later, because at the moment I am still writing the framework to justify the development of the tools. 2022-04-12-08_50_1649719257-1920x1080 2022-04-12-08_49_1649719178-1920x1080 2022-04-12-08_48_1649719134-1920x1080

Cheers

Marcus-Richmond commented 2 years ago

@victorcalixto very cool! Please feel free to jump into Sverchok gis nodes: https://github.com/Marcus-Richmond/sverchok-gis-nodes. I've been testing code in the sgn_test1 branch, which is working at the moment, so if you want to test out the nodes maybe try that branch first. I've also saved scripts with essentially the same functions (to be used with sn lite node) here: https://github.com/Marcus-Richmond/sverchok-gis-scripts.

I think there is a huge potential for gis data to be used in blender/Sverchok for fields other than geology (my background is in geology so I usually come up with examples with that in mind). The way I see it, our goals start with bringing in data to blender via sverchok, and then branch off into separate analyses with the tools already in Sverchok.

So, yes let's work together. I sent a collaborator invite for Sverchok gis nodes, please feel free to check it out, add/critique code or start discussions there. Thanks,

victorcalixto commented 2 years ago

Many thanks @Marcus-Richmond. I will check it out this week and see how we can combine the functionalities of the nodes. You are right, from that perspective our goals are the same. Thanks for the invitation. Cheers.