search

GeoNode / geonode

GeoNode is an open source platform that facilitates the creation, sharing, and collaborative use of geospatial data.
https://geonode.org/
Other
1.45k stars 1.12k forks source link

GNIP-78: GeoNode generic "Apps" model to include pluggable entities into the framework #6684

Closed afabiani closed 3 years ago

afabiani commented 3 years ago

GNIP-78: GeoNode generic "Apps" model to include pluggable entities into the framework

Overview

The scope of this GNIP is to present a technical proposal for the integration of generic geospatial entities into GeoNode as part of the available Resource Bases.

The main goal is to provide a way for a GeoNode User to manage and share "Geo Applications" similarly to the other GeoNode entities, such as Documents, Layers and Maps.

Such Geo Applications could be any kind of object provided by a plugged in client library, which is not a Layer nor a Map. As an instance Dashboards, GeoStories, Charts, Geospatial Analysis and so forth.

Before moving forward with the proposal, we need to take into account some prerequisites and constraints:

We want to manage such “Geo Applications” by using the same interfaces we already have in place for Layers, Maps and Documents.

  1. This new entity needs to be managed somehow like one of the GeoNode’s ResourceBase ones.

    • It must be exposed through the GeoNode APIs
    • It must be integrated with the GeoNode security subsystem
    • It must be searchable through the GeoNode metadata model

  2. We don’t want to change the GeoNode core model in order to add a new custom resources. Those “Geo Applications” must be somehow “pluggable” into GeoNode by adding external dependencies. image

  3. It is not possible, at this stage, a complete refactoring of the GeoNode model and APIs v1. Moreover we want those changes to be compatible with old GeoNode versions that, hopefully, should be upgradable to the new version without data losses.

Proposed By

@afabiani @giohappy @allyoucanmap

Assigned to Release

This proposal is for GeoNode 3.2.

State

Proposal

Given the prerequisites depicted above, the technical proposal is to introduce into GeoNode the concept of “Geo Application” (aka GeoApp).

A GeoApp in GeoNode, will be a container integrated into the core model able to be queried through both the standard (Tastypie) and REST APIs frameworks of GeoNode.

The GeoApp module, would be a new application of the GeoNode core project, containing the base abstract classes allowing any external plugin to define as many concrete applications they need.

Every GeoApp concrete instance, must define a unique “GeoApp Type” which will be translated into a polymorphic_ctype allowing GeoNode to seamlessly manage the concrete models and distinguish the GeoApp instances between the common ResourceBases and other GeoApp instances.

By default, GeoNode will enable a new menu on the navigation toolbar, whose name could be customized through the settings.

This new section will allow admins to present on the frontend all the GeoApp instances plugged into GeoNode.

It will also be possible to disable such menus, allowing the developers to define their own customized menus, each for every GeoApp concrete instance.

GeoNode Core Model: GeoApp django application

The new GeoApp model, extends the ResourceBase model by adding few more generic fields for any further GeoApp implementation:

  1. Geospatial fields, like, projection, zoom and center coordinates. The BBOX polygon is already part of the base model.

  2. data; a generic JSON blob field which will contain any kind of raw configuration associated with the app.

GeoApp Model

...

class GeoApp(ResourceBase):
    """
    A GeoApp it is a generic container for every client applications the
    user might want to create or define.
    """

    PERMISSIONS = {
        'write': [
            'change_geoapp_data',
            'change_geoapp_style',
        ]
    }

    name = models.TextField(_('Name'), unique=True, db_index=True)

    zoom = models.IntegerField(_('zoom'), null=True, blank=True)
    # The zoom level to use when initially loading this geoapp.  Zoom levels start
    # at 0 (most zoomed out) and each increment doubles the resolution.

    projection = models.CharField(_('projection'), max_length=32, null=True, blank=True)
    # The projection used for this geoapp.  This is stored as a string with the
    # projection's SRID.

    center_x = models.FloatField(_('center X'), null=True, blank=True)
    # The x coordinate to center on when loading this geoapp.  Its interpretation
    # depends on the projection.

    center_y = models.FloatField(_('center Y'), null=True, blank=True)
    # The y coordinate to center on when loading this geoapp.  Its interpretation
    # depends on the projection.

    urlsuffix = models.CharField(_('Site URL'), max_length=255, null=True, blank=True)
    # Alphanumeric alternative to referencing geoapps by id, appended to end of
    # URL instead of id, ie http://domain/geoapps/someview

    data = models.OneToOneField(
        "GeoAppData",
        related_name="data",
        null=True,
        blank=True,
        on_delete=models.CASCADE)
...

class GeoAppData(models.Model):

    blob = JSONField(
        null=False,
        default={})

    resource = models.ForeignKey(
        GeoApp,
        null=False,
        blank=False,
        on_delete=models.CASCADE)

GeoApp URIs

The generic GeoApp URIs will allow us to query and access the instances from GeoNode. Those can also be overridden for specific apps if needed.

js_info_dict = {
    'packages': ('geonode.geoapps', ),
}

apps_list = register_url_event()(TemplateView.as_view(template_name='apps/app_list.html'))

urlpatterns = [
    # 'geonode.geoapps.views',
    url(r'^$',
        apps_list,
        {'facet_type': 'geoapps'},
        name='apps_browse'),
    url(r'^new$', views.new_geoapp, name="new_geoapp"),
    url(r'^preview/(?P<geoappid>[^/]*)$', views.geoapp_detail, name="geoapp_detail"),
    url(r'^preview/(?P<geoappid>\d+)/metadata$', views.geoapp_metadata, name='geoapp_metadata'),
    url(r'^preview/(?P<geoappid>[^/]*)/metadata_detail$',
        views.geoapp_metadata_detail, name='geoapp_metadata_detail'),
    url(r'^preview/(?P<geoappid>\d+)/metadata_advanced$',
        views.geoapp_metadata_advanced, name='geoapp_metadata_advanced'),
    url(r'^(?P<geoappid>\d+)/remove$', views.geoapp_remove, name="geoapp_remove"),
    url(r'^(?P<geoappid>[^/]+)/view$', views.geoapp_edit, name='geoapp_view'),
    url(r'^(?P<geoappid>[^/]+)/edit$', views.geoapp_edit, name='geoapp_edit'),
    url(r'^(?P<geoappid>[^/]+)/update$', views.geoapp_edit,
        {'template': 'apps/app_update.html'}, name='geoapp_update'),
    url(r'^(?P<geoappid>[^/]+)/embed$', views.geoapp_edit,
        {'template': 'apps/app_embed.html'}, name='geoapp_embed'),
    url(r'^(?P<geoappid>[^/]+)/download$', views.geoapp_edit,
        {'template': 'apps/app_download.html'}, name='geoapp_download'),
    url(r'^', include('geonode.geoapps.api.urls')),
]

Notice that the views templates will invoke the client_tag_library tags in order to dynamically render the final template. Such mechanism allows the plugged-in client library to, eventually, override the templates by editing the hooksets.

As an example, the app_list template will be defined as follows:

{% load i18n %}
{% load base_tags %}
{% load client_lib_tags %}

{% block head %}
    <style>
        #paneltbar {
            margin-top: 90px !important;
        }
    </style>

    {% get_geoapp_list %}
{% endblock %}

The generic client hooksets are defined as below instead:

    # GeoApps
    def geoapp_list_template(self, context=None):
        return 'apps/app_list_default.html'

    def geoapp_detail_template(self, context=None):
        return NotImplemented

    def geoapp_new_template(self, context=None):
        return NotImplemented

    def geoapp_view_template(self, context=None):
        return NotImplemented

    def geoapp_edit_template(self, context=None):
        return NotImplemented

    def geoapp_update_template(self, context=None):
        return NotImplemented

    def geoapp_embed_template(self, context=None):
        return NotImplemented

    def geoapp_download_template(self, context=None):
        return NotImplemented

The client library must implement the methods above in order to return its own templates, specific to the GeoApp implementations (see later how the mapstore-client library can plug in the GeoStories to GeoNode).

GeoApp REST APIs (v2)

:warning: Depends on GNIP-79: GeoNode REST APIs (v2)

Similarly to the models and client tags, the REST APIs too will be fully pluggable. The generic GeoApp application, will expose a set of predefined and generic api endpoints along with JSON serializers. Those ones could be overridden and extended in order to expose the APIs for the specific GeoApp instance.

The GeoApp attaches its own generic endpoint to the GeoNode REST API default router.

from geonode.api.urls import router

from . import views

router.register(r'geoapps', views.GeoAppViewSet)

To complete the bundle, the GeoApp generic app will provide a set of predefined viewsets and serializers, which can be eventually extended by the specific implementation.

Use Case: MapStore GeoStories

ℹ️ demo instance available at https://dev.geonode.geo-solutions.it/

The MapStore client library for GeoNode, currently implements the concept of “GeoStory” as a concrete instance of the GeoApps.

GeoStories App Model

As explained above the library should override and extend several methods and templates in order to achieve that.

First of all, we will need to include the MapStore library extension as part of the GeoNode INSTALLED_APPS bundle.

"""
To enable the MapStore2 REACT based Client:
1. pip install pip install django-geonode-mapstore-client==1.0
2. enable those:
"""
if GEONODE_CLIENT_LAYER_PREVIEW_LIBRARY == 'mapstore':
    GEONODE_CLIENT_HOOKSET = os.getenv('GEONODE_CLIENT_HOOKSET', 'geonode_mapstore_client.hooksets.MapStoreHookSet')

    if 'geonode_mapstore_client' not in INSTALLED_APPS:
        INSTALLED_APPS += (
            'mapstore2_adapter',
            'mapstore2_adapter.geoapps',
            'mapstore2_adapter.geoapps.geostories',
            'geonode_mapstore_client',)

The new model must extend the GeoApp one and define a concrete app_type which will be translated by GeoNode as a new polymorphic_ctype.

from django.utils.translation import ugettext_noop as _
from geonode.geoapps import GeoNodeAppsConfig

class GeoStoryAppsConfig(GeoNodeAppsConfig):

    name = 'mapstore2_adapter.geoapps.geostories'
    label = "geoapp_geostories"
    default_model = 'GeoStory'
    verbose_name = "GeoNode App: GeoStory"
    type = 'GEONODE_APP'

    NOTIFICATIONS = (("geostory_created", _("GeoStory Created"), _("A GeoStory was created"),),
                     ("geostory_updated", _("GeoStory Updated"), _("A GeoStory was updated"),),
                     ("geostory_approved", _("GeoStory Approved"), _("A GeoStory was approved by a Manager"),),
                     ("geostory_published", _("GeoStory Published"), _("A GeoStory was published"),),
                     ("geostory_deleted", _("GeoStory Deleted"), _("A GeoStory was deleted"),),
                     ("geostory_comment", _("Comment on GeoStory"), _("A GeoStory was commented on"),),
                     ("geostory_rated", _("Rating for GeoStory"), _("A rating was given to a GeoStory"),),
                     )

default_app_config = "mapstore2_adapter.geoapps.geostories.GeoStoryAppsConfig"
class GeoStory(GeoApp):

    app_type = models.CharField(
        _('%s Type' % settings.GEONODE_APPS_NAME),
        db_column='geostory_app_type',
        default='GeoStory',
        max_length=255)
    # The type of the current geoapp.

That’s it, there’s not much more to do here, unless we need to further specific fields to our model. Currently MapStore is able to store the whole GeoStories configuration as a JSON blob into the schema, therefore the data field of the generic app model is more than enough.

GeoStories Client Hooksets

We will need to render the GeoStories list, details and view templates. As explained before, this can be easily done by implementing the GeoNode client library generic Hooksets methods.

    # GeoApps
    def geoapp_list_template(self, context=None):
        self.initialize_context(
            context,
            callback=ms2_config_converter.convert)
        return 'geonode-mapstore-client/app_list.html'

    def geoapp_new_template(self, context=None):
        self.initialize_context(
            context,
            callback=ms2_config_converter.convert)
        return 'geonode-mapstore-client/app_new.html'

    def geoapp_view_template(self, context=None):
        self.initialize_context(
            context,
            callback=ms2_config_converter.convert)
        return 'geonode-mapstore-client/app_view.html'

    def geoapp_edit_template(self, context=None):
        self.initialize_context(
            context,
            callback=ms2_config_converter.convert)
        return 'geonode-mapstore-client/app_edit.html'

    def geoapp_update_template(self, context=None):
        self.initialize_context(
            context,
            callback=ms2_config_converter.convert)
        return 'geonode-mapstore-client/app_update.html'

    def geoapp_embed_template(self, context=None):
        self.initialize_context(
            context,
            callback=ms2_config_converter.convert)
        return 'geonode-mapstore-client/app_embed.html'

The MapStore client library can now define its own templates by injecting all the specific JavaScript code needed to manipulate a GeoStory.

GeoStories REST APIs v2

:warning: Depends on GNIP-79: GeoNode REST APIs (v2)

The final step is to plug GeoStories specific APIs to the GeoNode router. This requires a few more steps.

Since the MapStore client and adapter are dynamically plugged into GeoNode, we will need to declare at initialization time which new Django urls we want to include into the resolver.

This is possible by including them from the app __init__ class of the MapStore Adapter.

def run_setup_hooks(*args, **kwargs):
    from geonode.urls import urlpatterns
    from django.conf.urls import url, include

    urlpatterns += [
        url(r'^mapstore/', include('mapstore2_adapter.urls')),
        url(r'^', include('mapstore2_adapter.geoapps.geostories.api.urls')),
    ]

class AppConfig(BaseAppConfig):

    name = "mapstore2_adapter"
    label = "mapstore2_adapter"
    verbose_name = _("Django MapStore2 Adapter")

    def ready(self):
        """Finalize setup"""
        run_setup_hooks()
        super(AppConfig, self).ready()

In this way we don’t need to touch GeoNode at all. Everything will dynamically plug in at initialization time.

Now, we can easily define a new api model for the GeoStories which will eventually extend the generic viewsets and serializers provided by the GeoApp module of GeoNode.

from geonode.api.urls import router
from mapstore2_adapter.geoapps.geostories.api import views

router.register(r'geostories', views.GeoStoryViewSet)

urlpatterns = [
    url(r'^api/v2/', include(router.urls)),
]

Backwards Compatibility

Compatible with GeoNode 3.x and above, by using the new bbox model provided by Catalyst.

Future evolution

Feedback

Voting

Project Steering Committee:

Links

gannebamm commented 3 years ago

We will use a mixture of MapStore2 Apps like Dashboard, MapStories and there like and 'third party' applications completely decoupled from GeoNode like shiny applications running on their own server. The third-party applications will most likely just hold some hyperlink to get there but should have some of the ResourceBase metadata info to get them displayed in searches. Since those third-party applications would be out of the scope of the GeoNode system, authentification could not be used - or has to be done by OAuth scopes (TBD).

:edit: To get other developers use this generic model (eg. cartologic), we could provide documentation on how to built a simple generic app.

In the long run, we think about containerizing those third-party applications and host them as part of the GeoNode stack. But that's far-far-away.

giohappy commented 3 years ago

@gannebamm as you will know we already have several options for AuthN of third party apps:

  1. Use an external Oauth2 server. We did it for a client, which needs to authenticate in GeoNode from QGIS using Oauth2 provided by their WSO2 instance. Here GeoNode is configured to "hop" the authentication to WSO2 and create an authenticated session on top of it.

  2. GeoNode already offers on Oauth2 provider, which is internally used by Geoserver to exchange access tokend with GeoNode. One of our customers is using this support to obtain access tokens for a third party app of them. They registered this app along with Geoserver inside GeoNode, and it seems it is working perfectly. On the base of this use case, I want to investigate the support for refresh tokens.

Going in the direction of using GeoNode as a REST backend for multiple third party apps, even served from different domains, the documentation for these scenarios should be extended. And probably there will be some more work to make all this work as smooth as possible...

ahmednosman commented 3 years ago

Cartoview related to this GNIP

Cartoview is managing a couple of app types: App Type A) Resource based, like dashboards, storymaps, viewers etc App Type B) Function based or business based, like data loaders, data processors, disaster management and so on. In this type there is no resource created it just a single app instance

Additionally Cartoview is offering app management from the UI, start, stop, suspend apps and so on There is also an app store for deploying apps at run time which is also configurable if you want to use your own It is good for System Admins, you can maintain the apps without requiring a developer to install or upgrade apps

Having a base App or Generic App type is kept to a minimum, for example a map catalog app, training material app, a pdf maps app are resource based apps however it does not require properties related to maps or layers so you need to keep the app definitions to the bare minimum (just enough to be a resource) which is what CartoView is doing

If the app requires geographic properties like starting extent and layers, then use a map or a number of maps to define the app properties, a number of app property definitions are already available as definition wizards to ease the app development process

deploying story maps or dashboards based on mapstore should be very simple already with Cartoview

Before embarking on this GNIP I suggest spending sometime evaluating the capabilities of CartoView,

Looking forward to hear back from you all

afabiani commented 3 years ago

Dear @ahmednosman the only way to evaluate CartoView apps it to propose a PR to GeoNode.

giohappy commented 3 years ago

@ahmednosman the idea behind CartoView is instereseing, and it is pretty similar at its minimum to the concept that we're proposing here. The key differences are:

In brief, integrate the basic stuff in the core and let everyone build fantastic appls ontop of it :)

ahmednosman commented 3 years ago

@giohappy

ahmednosman commented 3 years ago
giohappy commented 3 years ago

@ahmednosman we have two distinct conecpts here:

From our side we are only committed to support GeoNode core development, and related critical components (geonode project, Geoserver integrations, etc.) I really wish you will be able to contribute to GeoNode core on this side.

ahmednosman commented 3 years ago

image I think we are in agreement, let there be an API first GeoNode with apps treated as resources As such I suggest to limit the proposal of the GNIP to the APP API implementation only.

Referring to the case study GeoStories APP can be deployed as an app outside the GeoNode Core The App will appear as a resource listing in GeoNode with meta data and search etc MapStore library extension or other such libraries should not be included in Core GeoNode

You can consider the following parameters

correction: CartoView is already released for GeoNode 3 https://github.com/cartologic/cartoview/releases/tag/v1.30.0

afabiani commented 3 years ago

@ahmednosman the proposal is already taking into account generic APIs and models.

No specific app will be included into the GNIP. The concept is exactly this one, creating a structure in geonode allowing to plugin any concrete apps implementation from outside geonode.

The "MapStore GeoStory" use case would like to demonstrate how it could be possible to plug in a concrete app from an external package (that said MapStore Client library).

About the cartologic tag, nice, but, again, if you want to backport and include any stuff into GeoNode core you will need to follow the contribution agreement as established by the community rules, i.e.:

  1. Propose a GNIP along with description and technical details
  2. Discuss the GNIP and ask for PSC votes
  3. Attach a PR to the GNIP with the implementation so that developer can inspect and evaluate the code
  4. Make sure there are both documentation and test cases
gannebamm commented 3 years ago

After talking about this feature with some of my colleagues we thought that a simple 'hello world' example would be awesome. Something like an ultra-slim leaflet app maybe? @giohappy @afabiani Would this be possible? I think this would give the concept a big push.

ahmednosman commented 3 years ago

Hello We can develop this template thanks ahmed

http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avg.com http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Tue, Dec 22, 2020 at 12:57 PM Florian Hoedt notifications@github.com wrote:

After talking about this feature with some of my colleagues we thought that a simple 'hello world' example would be awesome. Something like an ultra-slim leaflet app maybe? @giohappy https://github.com/giohappy @afabiani https://github.com/afabiani Would this be possible? I think this would give the concept a big push.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GeoNode/geonode/issues/6684#issuecomment-749481742, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADES3RY3CUBNPD6SZMZIR4LSWB3RZANCNFSM4UJBVMAA .

gannebamm commented 3 years ago

@ahmednosman Hi Ahmed are there any news for the example app?

ahmednosman commented 3 years ago

@gannebamm Hi Florian We took a look at this. What we concluded If the mapstore client is the installed mapping client for geonode then you have to develop all your apps using mapstore and can not use leaflet apps while mapstore is map client for geonode If you want to deploy basic apps on leaflet/other apis, then you have to deploy a mapping client library based on leaflet then all the apps you develop will be based on leaflet maybe we are missing something? @afabiani can confirm this

afabiani commented 3 years ago

@ahmednosman not really, you can hook your client library and override/plugin your custom hookset.

The hookset basically decides which templates to use to render the apps list, details and so on. You will need to provide also leaflet static files with the library of course.

gannebamm commented 3 years ago

I see @afabiani . Do we have any documentation about that? https://docs.geonode.org/en/master/intermediate/viewer/index.html is currently empty. I think we had some documentation about the process somewhere?

gannebamm commented 3 years ago

@afabiani Since there is currently no living example of a non-mapstore based GeoApp I would like to create one as a proof-of-concept for this old GNIP. Since I do not want to do anything complicated the outcome is simple:

Taking a look at the GeoStory implementation which is part of the django-mapstore-adapter I see a lot of boilerplate code(?). I am really struggling to get this very simple concept above to work.

To get started I used the django-mapstore-adapter with it GeoStory as geoapp type and refactored it. You can see my efforts here: https://github.com/gannebamm/geonode/tree/simpleGeoApp The views have to be rewritten to make this work I think and some other parts as well. But for now, I just would like to get some very early concept running.

For some reason, the simple app is not listed if I want to create a new geoapp from the web-UI. Any help is very much appriciated.

giohappy commented 3 years ago

@gannebamm thanks for spending time on this. In the near future we're going to implement a new GeoApp and this would be the right time to review the code and, hopefully, simplify it. We will probably start working on this from the next week. I will ask the guys to have a look to your effort and take your notes into account to review the code.

gannebamm commented 3 years ago

Don´t expect much of my refactoring. It is more like a brute force cutting of everything unrelated. I am happy to hear there will be more geoapps in the near future!

afabiani commented 3 years ago

Be careful, some important changes here will be done through GNIP-89