Command to load a point layer as linear reference records

[leplatrem]

This Django command would take a layer with point geometries and a model name, and create relevant topologies according the path network.

For example:

bin/django loadtopo geotrek.trekking.POI fountains.shp
bin/django loadtopo geotrek.infrastruture.Signage pancartes.shp

For points, it's very easy, since the method Topology.deserialize() already converts lat/lng to topologies.

Use from django.db.models.loading import get_model See https://docs.djangoproject.com/en/1.6/howto/custom-management-commands/

[camillemonchicourt]

Is it possible to use this fonction other ponctual objects (signages, ponctual interventions, ponctual infrastructures...) ? Attributs are taken from the shapefile or only the geometry ?

[camillemonchicourt]

OK it seems like it also works with signages... but in RELEASE NOTE it just mentions POI : "Command to import shapefile with points into POI as topologies (fixes #952)".

[leplatrem]

This is true. I shall reopen this issue, since the implemented command is explicitly made to load POIs only. (bin/django loadpoi ...)

https://github.com/makinacorpus/Geotrek/blob/v0.23.0/geotrek/trekking/management/commands/loadpoi.py#L47-L50

[camillemonchicourt]

I created a SHP layer to test it with only one POI. I uploaded it on the server and launched bin/django loadpoi ../mypois.shp.

It worked well after I transformed the layer into 4326 but it didnt included the name and type even if I had the fields name and type in the layer :

Would be great to fix this, to allow it for other modules with points (signages, infrastructures, interventions...) and to include it in the new IMPORT module as discussed here - https://github.com/makinacorpus/Geotrek/issues/1595

[TheoLechemia]

I have the same problem as Camille using the command loadpoi . The object is created in Geotrek but without his attributes. However my goal was to import 'Infrastructures'... I adapt the 'loadpoi' command to import infrastructures (aménagement), and it works, attributes are also imported. Should I do a pull request to add this new command in the project ?

[camillemonchicourt]

Yes sure, very interested with it !!!!

[camillemonchicourt]

OK so there has been some improvement with a new command in 2.12.0 (https://github.com/GeotrekCE/Geotrek-admin/releases/tag/2.12.0) to load SIGNAGES from a SHP (?) into Geotrek database. There is also a pending PR to load infrastructures (not only signages) but it is not finished and has now some conflicts : https://github.com/GeotrekCE/Geotrek-admin/pull/1630

And now @CynthiaBorotPNV from PnVanoise did an interesting Python to insert her PostGIS datas from a previous database into her Geotrek database : http://geotrek.ecrins-parcnational.fr/ressources/technique/2017-10-13-impFromAigleDB.py

To be clarified in upcoming versions.

[leplatrem]

And now @CynthiaBorotPNV from PnVanoise did an interesting Python to insert her PostGIS datas from a previous database into her Geotrek database : http://geotrek.ecrins-parcnational.fr/ressources/technique/2017-10-13-impFromAigleDB.py

I saw «Aigle» and got curious :)

The script does not seem to link the topologies with the paths (no insertion in e_r_evenement_troncon)

For each point, you must lookup the closest path and insert the «aggregation»... Unless I'm totally wrong :)

Since you do your script in Python, it's a lot easier/safer/shorter to use the Geotrek models and helpers than raw SQL:

        from geotrek.core.helpers import TopologyHelper
        from geotrek.trekking.models import POI, POIType

        poitype, _ = POIType.objects.get_or_create(label="Faune")
        poi = POI.objects.create(name="Guépard", type=poitype)
        # Use existing topology helpers to transform a Point(x, y)
        # to a path aggregation (topology)
        serialized = '{"lng": %s, "lat": %s}' % (geometry.x, geometry.y)
        topology = TopologyHelper.deserialize(serialized)
        # Move deserialization aggregations to the POI
        poi.mutate(topology)
        # Done.

You can see that it is the same code for Signage: https://github.com/GeotrekCE/Geotrek-admin/blob/30a1a30aba79fc841749ec753c460372121c5404/geotrek/infrastructure/management/commands/loadsignage.py#L150-L180

[IdrissaD]

I would like to apply the logic you provided @leplatrem, but can't find, once I created my script, how to execute it in the Django app's context. If I need this script to be long-lasting, and not erased at each upgrade of geotrek-admin, I can't add a new command on the exact same model and place than loadpoi or loadsignage, am I wrong?

[camillemonchicourt]

Maybe this example can help : https://github.com/GeotrekCE/Geotrek-admin/issues/2543#issuecomment-1026928237 ?

[IdrissaD]

Thanks, I didn't knew that! Indeed it helps, but if I'm in the context of Geotrek's shell, I still have to get files outside of this context, e.g. in var/conf, and that is the thing I don't know how to do with the script above.

[LePetitTim]

Here you have the command to load pois : https://github.com/GeotrekCE/Geotrek-admin/blob/master/geotrek/trekking/management/commands/loadpoi.py

In it you can see we use DataSource to load informations from any file (shp, gpx, ....) :

from django.contrib.gis.gdal import DataSource

Here the documentation about Datasources : https://docs.djangoproject.com/en/4.0/ref/contrib/gis/gdal/#vector-data-source-objects

But be careful you should not create as the example in [#2543 (comment)]https://github.com/GeotrekCE/Geotrek-admin/issues/2543#issuecomment-1026928237 this is only a one-shot script because it can have critical effect on your database.

If want to create a new command and think this command can be used by everyone and not dangerous for all the community you could create a pull request. Commands are stored in management directory in every apps. (If there is no management directory there is no command in this app, ask me if you need to add a new management directory) example : https://github.com/GeotrekCE/Geotrek-admin/blob/master/geotrek/trekking/management/

If you want to create a script that can be use more than one time, only by yourself, you can use parsers. To my knowledge there is no other solution.

class CustomParser(Parser):
    def parse(self, filename=None, limit=None):
        # your logic here
        # You can check geotrek/common/parsers.py
        filename = os.path.join('path_of_your_file')
        ds = Datasource(filename, encoding='utf-8')
        for layer in ds:
          pass

[IdrissaD]

Thank you for your answer, we came to the same solutions. For now we surcharge a Parser and its parse() function like you said, because we wanted to avoid any modification to the main code of Geotrek, but adding a new command is certainly the ideal we should aiming for. We'll reach out to you when necessary!