CRUD Layer / Read Content List

[EPajares]

Users would like to upload their own layers to GOAT to work with their own data and use it for visualization and analyses. Meanwhile, there are essentially two types of layers coming as files and layers that are served from another service (like an API). In GOAT we are differentiating between: Feature Layer (File), Imagery Layer (URL) , Tile layer (URL) and Table Layer (File with no geometry).

The upload flow can be as follows

sequenceDiagram
    participant Client
    participant GoatAPI as GOAT API
    participant OrgDB as Organization Database
    participant AppDB as Application Database
    participant Auth as Authorization Endpoint
    Client->>GoatAPI: Sends request with optional file
    GoatAPI->>OrgDB: Uploads file in organization database
    Note over GoatAPI,OrgDB: Optional Step (if the layer source is a file)
    GoatAPI->>AppDB: Creates layer with a random default style if not an indicator/If an indicator it should read the default style for the indicator
    GoatAPI->>Auth: Calls authorization endpoint to set permissions
    Note over AppDB,Auth: Optional Step (if running in SaaS Mode)
    Auth ->> GoatAPI: Return permissions
    GoatAPI->>Client: Returns metadata of layer

[EPajares]

File upload To have a fast file upload it would be the best to use ogr2ogr. Each layer that is uploaded need a primary key that did not exists before. So this is the only column we are adding. Meanwhile we need to make sure to create a spatial index on the geom column in case of geofile. If the file is non spatial (xlsx or csv) it should no geometry needs to be created.

Authorization Call: We need to implement the call to authorization using the webhook logic. We are having the webhooks in the auth repo in the files keycloak.py and webhooks.py. We can coordinate this further.

[metemaddar]

A note to Class Diagram for the Layer which you have designed: It seems the Diagram that the chat GPT Provided is not right.

Layer types:

• Feature Layer (File)
• Imagery Layer (URL)
• Tile layer (URL)
• Table Layer (File with no geometry)

[metemaddar]

Layer schema hirearchy by ChatGPT:

classDiagram
    class LayerType
    class FeatureLayerType
    class IndicatorType
    class AnalysisType
    class ScenarioType
    class FeatureLayerExportType
    class ImageryLayerDataType
    class TileLayerDataType
    class TableDataType
    class LayerUpdateBase
    class LayerBase
    class GeospatialAttributes
    class LayerProjectAttributesBase
    class ContentUpdateBase
    class ContentUpdateBase
    class BaseModel
    class FeatureLayerBase
    class FeatureLayerReadBase
    class FeatureLayerUpdateBase
    class FeatureLayerProjectBase
    class FeatureLayerStandardCreate
    class FeatureLayerStandardRead
    class FeatureLayerStandardUpdate
    class FeatureLayerStandardProject
    class FeatureLayerIndicatorAttributesBase
    class FeatureLayerIndicatorCreate
    class FeatureLayerIndicatorRead
    class FeatureLayerIndicatorUpdate
    class FeatureLayerIndicatorProject
    class FeatureLayerScenarioAttributesBase
    class FeatureLayerScenarioCreate
    class FeatureLayerScenarioRead
    class FeatureLayerScenarioUpdate
    class ImageryLayerAttributesBase
    class ImageryLayerCreate
    class ImageryLayerRead
    class ImageryLayerUpdate
    class ImageryLayerProject
    class TileLayerAttributesBase
    class TileLayerCreate
    class TileLayerRead
    class TileLayerUpdate
    class TileLayerProject
    class TableLayerCreate
    class TableLayerRead
    class TableLayerUpdate
    class TableLayerProject

    LayerType --> LayerBase
    FeatureLayerType --> FeatureLayerBase
    IndicatorType --> FeatureLayerIndicatorAttributesBase
    AnalysisType --> LayerUpdateBase
    ScenarioType --> FeatureLayerScenarioAttributesBase
    FeatureLayerExportType --> FeatureLayerBase
    ImageryLayerDataType --> LayerBase
    TileLayerDataType --> LayerBase
    TableDataType --> LayerBase
    LayerUpdateBase <|-- FeatureLayerUpdateBase
    LayerBase <|-- FeatureLayerBase
    LayerBase <|-- ImageryLayerCreate
    LayerBase <|-- TileLayerCreate
    LayerBase <|-- TableLayerCreate
    ContentUpdateBase <|-- LayerUpdateBase
    ContentUpdateBase <|-- FeatureLayerUpdateBase
    BaseModel <|-- LayerUpdateBase
    BaseModel <|-- LayerProjectAttributesBase
    FeatureLayerBase <|-- FeatureLayerReadBase
    FeatureLayerBase <|-- FeatureLayerProjectBase
    FeatureLayerBase <|-- FeatureLayerStandardCreate
    FeatureLayerBase <|-- FeatureLayerIndicatorCreate
    FeatureLayerBase <|-- FeatureLayerScenarioCreate
    FeatureLayerReadBase <|-- FeatureLayerStandardRead
    FeatureLayerReadBase <|-- FeatureLayerIndicatorRead
    FeatureLayerReadBase <|-- FeatureLayerScenarioRead
    FeatureLayerUpdateBase <|-- FeatureLayerStandardUpdate
    FeatureLayerProjectBase <|-- FeatureLayerStandardProject
    FeatureLayerProjectBase <|-- FeatureLayerIndicatorProject
    FeatureLayerProjectBase <|-- FeatureLayerScenarioProject
    FeatureLayerIndicatorAttributesBase <|-- FeatureLayerIndicatorCreate
    FeatureLayerIndicatorAttributesBase <|-- FeatureLayerIndicatorRead
    FeatureLayerIndicatorAttributesBase <|-- FeatureLayerIndicatorProject
    FeatureLayerScenarioAttributesBase <|-- FeatureLayerScenarioCreate
    FeatureLayerScenarioAttributesBase <|-- FeatureLayerScenarioRead
    FeatureLayerScenarioAttributesBase <|-- FeatureLayerScenarioProject
    ImageryLayerAttributesBase <|-- ImageryLayerCreate
    ImageryLayerAttributesBase <|-- ImageryLayerRead
    ImageryLayerAttributesBase <|-- ImageryLayerProject
    TileLayerAttributesBase <|-- TileLayerCreate
    TileLayerAttributesBase <|-- TileLayerRead
    TileLayerAttributesBase <|-- TileLayerProject
    TableLayerCreate <|-- TableLayerCreate
    TableLayerRead <|-- TableLayerRead
    TableLayerUpdate <|-- TableLayerUpdate
    TableLayerProject <|-- TableLayerProject

[metemaddar]

I think this hierarchy by chatGPT is wrong and useless.

[EPajares]

I think having all the schemas for Update, Read and Create in this overview might be bit too much. So maybe like this it can be slimmed down a bit

[metemaddar]

So I start by creating the layer by user and then in the next step we can talk about uploading the layer's file.

[metemaddar]

@EPajares May I‌ cleanup all the endpoints at endpoints/layers.py or keep them?

[EPajares]

Can you put the old file also in case of crud to a folder called legacy inside the respective folder? So we can take parts of the code later where needed.

[metemaddar]

Okay. Sure.

[metemaddar]

@EPajares Or maybe we can start the v2 endpoints.

[metemaddar]

This can also reduce the load over the loaded endpoints while development

[metemaddar]

We can add an env variable ENABLE_V1_ROUTES that if it was True then we can include V1 as well.

https://github.com/goat-community/goat/blob/1c89b6522dcc3677ed8b00883212818082596e46/app/api/src/main.py#L114

[metemaddar]

@EPajares The Base class is removed from models._base_class but it is needed by some CRUDs. Do we have it in other places? Can we add it? or is there any idea to handle it?

https://github.com/goat-community/goat/blob/1c89b6522dcc3677ed8b00883212818082596e46/app/api/src/db/models/_base_class.py#L6-L13

[metemaddar]

Among all examples. We have the following fields with error which need change either in examples or in Schemas:

{
    "type_error.str": [
        "extent"
    ],
    "value_error.missing": [
        "url",
        "data_type",
        "payload",
        "scenario_id",
        "indicator_type",
        "scenario_type",
        "legend_urls",
        "size",
        "style_id",
        "feature_layer_type"
    ],
    "type_error.enum": [
        "data_type"
    ]
}

[metemaddar]

The problem with using Union as Body data type shows in enum values like the "data_type" here. As the "data_type" field is different among the schemas.

[EPajares]

Some update: We need to convert the read layer into a more generic endpoint: "Read content". This endpoint can return all content types and might include some functionality to filter by content type and folder. Accordingly we also need to consider the folder structure we will have in the new version.

[EPajares]

This task now should only focus on the CRUD layer. I will create a separate issue for content list.