Question about project status and longer term plans

[songololo]

I'm looking for python bindings for mdal so that I can manipulate meshes for a QGIS plugin.

This led me to this repo, which looks interesting and potentially very useful.

I'm wondering what the current project status is and whether it is envisioned that this will be developed going forward?

[songololo]

A related question, any plans to make this available to pypi? I don't currently see mdal libraries available through e.g. brew so imagine this is some ways down the road.

[runette]

Hi Gareth

Some responses - not in any order:

1. This package is critical for a couple of products that I sell - so it is live and I will continue to maintain it, but
2. At the moment, based on the requirements of the released products, the functionality is fairly basic, but
3. I have a draft of a major update that includes much more functionality - including writing datasets - that I have been working on and meaning to finalize for the last 6 months! Now I have another user chasing me - I will try ti get that finalized and released in the next few weeks!

I see great scope for MDAL in volumetric data analysis - which is where I am moving in my career.

I have also been working - in a relaxed way - on coming up with more of a cookbook for using the bindings. I will pack that up again and see if I can publish something.

As for Pypi - it would be a good idea but I always Conda and have not had time to get the Pypi package worked out and published.

I am moving towards using GH Actions to publish all of my packages - it is cleaner and more secure. The next version of the bindings will include a GH Action to publish the Conda Package. If you are able to create and PR a GH Action to publish to Pypi - that would be great!

Thanks P

On Fri, 25 Nov 2022 at 19:17, Gareth Simons @.***> wrote:

A related question, any plans to make this available to pypi? I don't currently see mdal libraries available through e.g. brew so imagine this is some ways down the road.

— Reply to this email directly, view it on GitHub https://github.com/ViRGIS-Team/mdal-python/issues/7#issuecomment-1327801951, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARC2MYHC7T5L4YVU5R23QDWKEGEVANCNFSM6AAAAAASLSBDZE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[songololo]

@runette that's great news re: future scope for the package.

I don't know much about C++ or packaging C++ packages into PyPI releases but will try to pitch in where I can.

[runette]

OK

1. As I said elsewhere - I had forgotten the status of my own project. Version 1.0.0 and later have the write integrations and are up to date.

2. However, my memory was correct - the problem was that I had not got around to documentation. I will start that now,

3. I have created a pip package! This is shameless stolen from the PDAL package and is based on A GH Actions script - all of which is good. However, like the PDAL and GDAL bindings and for the same reason - the pip package will only work if there is a working version of MDAL on the machine and in the search path. This, after all is the value of Conda - sorting this stuff out.

I have tested the package on my machines but only in a conda environment with MDAL loaded. It works well and should work on any machine where you have correctly built MDAL as well.

HOWEVER - although this package loads correctly into Python in the OSGEO shell and is then accessible in the QGIS python console - it fails because MDAL is not loaded - i.e. it cannot find mdal shared object (i.e. mdal.dll / libmdal.so / libmdal.dylib) - which makes sense since I cannot find it manually either. MDAL is not installed in that way in QGIS. We need to go back to the MDAL Team to find out what is going wrong there!

[runette]

The MDAL Data Model as implemented in Python hs the following entities: Datasource, PyMesh, DatasetGroup and Dataset.

The Datasource represents one URI and can be a file or other datasource depending on the driver. A Datasource can have one or more Mesh's.

The PyMesh is the python instantiation of MDAL Mesh. The Mesh has geometry, CRS, and other Metadata.

The Mesh can have zero or more DatasetGroups that hold data each of which can locate the data on Vertex, Edge, Face OR Volume (again depending on the Driver).

The DatasetGroup has zero or more Dataset's of actual data. Multiple datasets are to be interpreted as a fourth dimension - usually time.

There are lot of ways of creating a new mesh (you can, for instance, do a deep copy of. an existing Mesh or load a mesh in from meshio). But if you want to create one form scratch it is something like this:

ds2 = Datasource("test.ply")
with ds2.add_mesh() as mesh:
    mesh.vertices = <<This is a numpy array dtype=[('X', '<f8'), ('Y', '<f8'), ('Z', '<f8')]>>
    mesh.faces = <<this is a numpy array dtype=[('Vertices', '<u4'), ('V0', '<u4'), ('V1', '<u4'), ('V2', '<u4'), ('V3', '<u4')]>>
    mesh.edges = << This is a numpy array dtype=[('START', '<u4'), ('END', '<u4')]
    with mesh.add_group('name', location=MDAL_DataLocation.DataOnVertices )
    group.add_data(<<numpy array of values of the right size >>>)
    mesh.save()

Volumetric meshes are bit more complicated

That is the general idea. I am very sure that I have forgotten a step or two ..

[caspervdw]

Hi @runette,

First of all, thanks for sharing this project. We'll be using MDAL for interfacing with various result files of hydrodynamic models (first of all, 3Di, as that is the product of my company Nelen&Schuurmans). For that we'd like to use your Python bindings. However, we typically only use open source projects that are community-supported, to avoid single-point dependencies.

Are you open to sharing the maintenance load of this project with us, and hopefully other contributors in the future?

Best, Casper

tagging @elisallenens

[runette]

Hi Casper

I am more than happy for this to be a community project.

The project was started with knowledge and approval (before and after the event) of the MDAL team.

As I said before, thus far the functionality has been primarily what my products have needed but it would be good if we could move to more of a steering group approach.

This code is also a focus of some other work I have been doing related to volumetric 3D GIS. That was the origin of the Open3D beta integration but I actually now think that this should be a separate project - the O3D integration is too big to be part of a bindings project.

Let me know what you think? Maybe we could approach Peter and the core MDAL team about their involvement in a steering group.

[caspervdw]

Great! Let's approach Lutra consulting proposing a steering group approach. Do you have time to reach out? You can include me at casper.vanderwel@nelen-schuurmans.nl

I am also 100% on board with separating the bindings from integration with other formats such as Open3D. I also have the same feeling about PyMesh. But this discussion can be done later.

[PeterPetrik]

mdal-python is officially recognized from MDAL team as official python wrapper around MDAL, and Paul is also MDAL developer. We may consider moving MDAL and mdal-python repositories under OSGeo organization in future, but nevertheless for the moment this project/repository has backing up from MDAL team.

[runette]

Closing this because now further action is required