fBedecarrats
commented
9 months ago This is great news, congratulations! Two questions:
- what about discussions regarding possible further splitting between mapme.* packages? I heard this was in the air and I wonder if this would be a subsequent move we need to anticipate as users or (modest) contributors.
- How would you recommend to handle the transition? We are starting a project in March with short deadlines for the first deliverables based on mapme.biodiversity. Is there a specific package milestone/version number we should refer to before all these changes? Is it recommended to install it via {remotes}?
Thanks in advance for the feedback!
goergen95
Dear MAPME friends,
today we are happy to make some announcements about the future direction of developments for
{mapme.biodiversity}. Before diving into the details let me thank you for choosing this package and for the various contributions we were lucky to receive from you during the last years! :tada:In order to adapt to new challenging and diverse use cases and in the hope to serve an even wider community, some major changes to the packages are in order. This issue is about informing you about the upcoming changes as we walk on our path towards a 1.0 release. It gives a detailed overview of what changes you can expect in the next several months and exactly when to expect them. We encourage you to adopt to these changes early on, and ask you to share your feedback with us along the way.
The overall vision of the proposed changes is to provide you with a package that is versatile in its application environments, delivering clearly structured outputs that can be serialized to different data formats with ease, while providing a clear and idiomatic interface.
To achieve our vision from the current state will mean that we introduce some breaking changes along the way. It is our wish to inform you ahead of time, so that you can plan accordingly.
Schedule of the milestones
We ordered the milestones with the most severe changes happening upfront. That way, once you adopted your workflows to a milestone the adaption to the next should be less severe.
: Schedule for the proposed milestones.
Milestone 1: Cleaner Interface Using Closures
This milestone sets out to provide you with a cleaner interface that provides instant feedback if arguments are wrongly specified. For this, we are going to use closures, e.g. functions that return other functions. The arguments that are important for you as a user to control the functionality are exposed at the outer level. This will make a call to fetch some resources and the subsequent calculation of indicators look something like this:
This also means that it will be easier to access the help pages for a resource/indicator, because these will be associated with fu nction names you are actually using, e.g.:
Arguments will instantly be checked for correctness and inform you about any miss-specifications. This interface will also make it easier to add custom resources/indicators add-hoc, for those that require this functionality.
Milestone 2: Standardized Indicator Output and Serialization Options
With this milestone we will have revised all indicator functions to return a standardized output format. The envisaged output format is inspired by the MovingFeatures standard and differentiates between simple and temporal properties. Simple properties are 1-dimensional attribute values of features that we are most familiar with from various GIS software. However, some of our indicators actually have a temporal axis. We will harness the fact that we already use nested list columns to represent our indicator data in R. However, we will standardize the output of all temporal indicators to a common format, e.g. along the lines of the following example output:
This way we allow for a much better predictability of the indicator output format for downstream applications. Also, it allows us to supply you with seamless serialization functions for GeoPackage, GeoJSON, and even MovingFeatures JSON.
We will also take the opportunity to revise and optimize the indicator functions for better overall performance. Additionally, we will add support for multi-polygon geometries by supplying a mechanism for selecting aggregation functions for indicators.
Milestone 3: Routing data I/O through GDAL
In its current state the package downloads data to the local file system. This severely limits the environments in which the package can be used efficiently. We received several requests to support e.g. different types of cloud storage. Manually maintaining drivers to read/write to commercial cloud storage is not something we are able to do. However, the good news is that GDAL already supports major cloud storage providers via its Virtual Filesystem drivers. With this milestone, we will leverage GDAL's capabilities to read/write geodata from a huge range of formats and sources. This will allow users of mapme.biodiversity to run their applications on a cloud provider of their choice. We will supply configuration options that will ease the process to authenticate e.g. against a cloud storage attached to a compute in the cloud. Thus, the interface to pull data from the internet to an S3 cloud storage will look something like this:
Leveraging GDAL for data transfer will also allow us to translate data to cloud-optimized formats that should increase computation speeds further down the line.
In case a specific resource is already provided in a cloud-optimized data format on a low-latency server, you might also decide to skip the download step altogether. This will most likley be only efficient for small to medium size portfolios and since we do not control how resources are provided, there will be limitations which resources support such a workflow.
Development of new resources/indicators
However, in the meantime, we will work on new resources/indicators at
{mapme.indicators}.After all milestones have been achieved, we will decide if a separation of the backend from concrete resource/indicators implementation makes sense for the future. In the case the answer is yes, all resources/indicators will eventually be migrated to
{mapme.indicators}. In the case the answer is no, we will migrate the newly developed indicators into{mapme.biodiversity}. As always, you are invited to share your feedback along the way.