Workflow: Sanjay

[anderslogg]

Workflow for supporting Sanjay's work. See workflows/workflow_sanjay.py in DTCC module.

Questions to be answered:

Overview: Explain what the overall idea is

Input data: List input data, file formats etc in detail. How to obtain the data? Also attach example data to this issue.

Output data: List input data, file formats etc in detail. How is the output data used?

How is this done today: Explain the workflow as it is today with existing tools.

Comments: Any comments? How would this work in an ideal world?

[snjsomnath]

My thoughts on translating the FME script to python with some help from ChatGPT!

Overview

The primary goal of this script is to automate the generation of Unreal Engine (UE) tiles from geospatial datasets. The datasets include a Digital Elevation Model (DEM), land use vector data (MY polygon layer), and a road network vector data (VL polygon layer). The process involves converting these datasets into specific tile resolutions suitable for UE and applying specific preprocessing steps like resampling, merging, and convolution filtering.

Input Data

1. Digital Elevation Model (DEM) in GeoTIFF format

• Description: DEM represents the Earth's surface and includes all objects on it. It's a raster dataset where each pixel value indicates the elevation of the ground at that location.
• Source: The dataset originates from Lantmäteriet, which is the Swedish mapping, cadastral, and land registration authority.
• Acquisition: This data can be obtained from the official Lantmäteriet portal or zeus SLU server.

2. Landuse Layer in Shapefile format

• Description: Vector dataset describing land use categories, e.g., water, forest, farm, built-up areas.
• Attributes: Primary attribute of interest is DETALJTYP which indicates the type of land use.
• Source: Lantmäteriet
• Acquisition: This data can be obtained from the official Lantmäteriet portal or zeus SLU server.

3. Road Network Layer in Shapefile format

• Description: Vector dataset showing road networks. Roads are classified by width, this should allow the script to perform custom buffers to represent realistic road widths
• Source: Lantmäteriet
• Acquisition: This data can be obtained from the official Lantmäteriet portal or zeus SLU server.

4. Optional Clipping Boundary

• Description: This could be a polygon shapefile used to clip the data to a specific region of interest. If not provided, then the script should identify the smallest bounding box of each of the files provided and use that as the clipping boundary.

Output Data

1. UE Tilesets for DEM

• Format: PNG at valid UE tile resolutions
• Naming Convention: Tile_X_<tile_column>_Y<tile_row>.png
• Usage: These tiles are used to render terrain in Unreal Engine with appropriate elevation details.

2. UE Tilesets for Roads

• Format: PNG with binary values (252 for True and 0 for False)
• Naming Convention: Tile_X_<tile_column>_Y<tile_row>.png
• Usage: Represent road networks within the Unreal Engine environment.

3. UE Tilesets for Land Use Categories

• Format: PNG with binary values (252 for True and 0 for False)
• Naming Convention: Tile_X_<tile_column>_Y<tile_row>.png
• Folders: Each land use category (e.g., water, forest) has its dedicated folder.
• Usage: Provide thematic land use layers in Unreal Engine, allowing for categorically distinct visual representation of different land uses.

How is this done today?

Currently, the conversion process relies on FME, a comprehensive software designed to manage spatial data. The workflow in FME consists of:

1. Importing the datasets.
2. Applying necessary transformations such as resampling, convolution filtering, and vector operations like buffering.
3. Exporting the processed data into the desired format, in this case, PNG tiles for UE.

While effective, this process requires manual intervention and lacks the flexibility and customization a dedicated Python script can provide. Additionally, FME may not be readily available to all users, making the Python alternative more universally accessible.

Mechanism of the Script

The core of this script revolves around three major functions:

1. HeightMapGeneration - Concerned with processing the Digital Elevation Model (DEM).
2. GenerateRoadMask - Dedicated to the generation of a rasterized road network.
3. GenerateLandUseMask - Used for converting land use vector data into raster tiles.

Pipeline Breakdown:

1. Input Validation:

   • Before processing, input data goes through a validation mechanism.
   • This ensures data integrity, correct formats, and resolution (e.g., 2m cell spacing for the DEM, there is no overlap between the bounding boxes of each of the input files, if DETALJTYP attribute does not contain any of the valid values we later look for etc).
   • If inconsistencies or errors are detected, they're logged for user clarity.

2. HeightMapGeneration:

   • Resampling of DEM to maintain a consistent resolution.
   • Multiple DEM tiles, if present, are merged into a single mosaic.
   • The composite raster is then clipped using the specified boundary.
   • The clipped raster is divided into tiles of the specified resolution.
   • Tiles are saved with a standardized naming convention in a dedicated output folder.

3. GenerateRoadMask:

   • The road vector data is buffered to provide a cushion around each road (nice to have - Seperate buffer distances per road class under the DETALJTYP attribute).
   • Overlapping buffers are dissolved to create a continuous shape.
   • The buffer shapes are rasterized to generate a road mask (1 True and 0 False).
   • A convolution filter blurs the raster to prevent sharp contrasts.
   • The processed raster is tiled, named, and saved similarly to the DEM.

4. GenerateLandUseMask:

   • Land use data is categorized based on the DETALJTYP attribute.
   • Each category (e.g., water, forest) is processed separately.
   • Each category is clipped, with the road network subtracted from it.
   • Like the road data, a convolution filter is applied to blur the raster.
   • The final rasters for each category are tiled, named, and saved in dedicated folders.

Considerations:

1. Data Size and Processing Time: Handling large datasets might result in substantial processing times. Parallel processing could be explored for optimization.

2. Memory Management: Operations like rasterizing or convolution can be memory-intensive especially if a raster mosaic needs to be loaded into memory. This needs to be handeled elegantly.

3. Boundary Handling: When using a clipping boundary, users need to ensure it is appropriately defined to prevent unwanted data exclusions.

Specifying landuse output categories:

landuse_mapping = {
    "WATER": ["VATTEN"],  # The repeated "VATTEN" values are consolidated under WATER
    "GLACIER": ["ÖPGLAC"],
    "BUILDINGS": ["BEBSLUT", "BEBHÖG", "BEBLÅG", "BEBIND"],
    "FARMING": ["ODLÅKER", "ODLFRUKT"],
    "OPEN AREAS": ["ÖPMARK", "ÖPKFJÄLL", "ÖPTORG"],
    "FOREST": ["SKOGBARR", "SKOGLÖV", "SKOGFBJ"],
    "UNMAPPED": ["MRKO"] # Unused as of now
}

• WATER: Represents areas designated as "VATTEN".

• GLACIER: Represents glaciers.

• BUILDINGS: Captures all types of buildings and developments.

• FARMING: Includes areas designated for farming and fruit-growing.

• OPEN AREAS: Represents various open lands.

• FOREST: Comprises different forest types.

• UNMAPPED: Denotes areas that have not been mapped.

Note: Something similar can be used for the road buffers.

---

1. Primary Functions

1.1. validate_input_data

Validates the input data for correctness, resolution, and the presence of required attributes.

Parameters:

• dem_directory: Path to the DEM geotiff directory.
• landuse_path: Path to the landuse MY layer shapefile.
• road_path: Path to the road VL layer shapefile.
• optional_clipping_boundary: Optional clipping boundary.

def validate_input_data(dem_directory, landuse_path, road_path, optional_clipping_boundary=None):
    pass

1.2. generate_heightmap

Processes DEM data to produce heightmap tiles suitable for Unreal Engine.

Parameters:

• dem_directory: Path to the DEM geotiff file.
• clipping_boundary: Clipping boundary for processing.
• output_folder: Folder to store the generated tiles.
• ue_cell_resolution: Unreal Engine cell resolution (default set to 1009 other valid values include 8129,4033,2017,5055,253,127 see specific tile resolutions suitable for UE).

def generate_heightmap(dem_path, clipping_boundary, output_folder, ue_cell_resolution=1009):
    pass

1.3. generate_road_mask

Creates a rasterized road mask with buffer and optional convolution filtering.

Parameters:

• road_path: Path to the road VL layer shapefile.
• clipping_boundary: Clipping boundary for processing.
• output_folder: Folder to store the generated tiles.
• ue_cell_resolution: Unreal Engine cell resolution (default set to 1009 seespecific tile resolutions suitable for UE).

def generate_road_mask(road_path, clipping_boundary, output_folder, ue_cell_resolution=1009):
    pass

1.4. generate_landuse_mask

Processes landuse data to generate raster tiles for each land-use type, utilizing a provided mapping.

Parameters:

• landuse_path: Path to the landuse MY layer shapefile.
• clipping_boundary: Clipping boundary for processing.
• output_folder: Folder to store the generated tiles.
• landuse_mapping: Dictionary mapping of landuse types.
• ue_cell_resolution: Unreal Engine cell resolution (default set to 1009 see specific tile resolutions suitable for UE).

def generate_landuse_mask(landuse_path, clipping_boundary, output_folder, landuse_mapping, ue_cell_resolution=1009):
    pass

---

2. Utility Functions

Utility functions can be added for:

• Raster operations such as resampling, mosaicking, tiling, renaming, calculating z_scale value see calculation method here.

• Vector operations like buffering, dissolving, and rasterizing.

• File operations to check data integrity, file existence, and handle output.

• ---

3. Main Execution

if __name__ == "__main__":
    # Define paths to input files
    dem_directory = "path_to_dem_directory"
    landuse_path = "path_to_landuse.shp"
    road_path = "path_to_road.shp"

    # Define output folder
    output_folder = "path_to_output_folder"

    # Validate input data
    validate_input_data(dem_path, landuse_path, road_path)

    # Process data
    generate_heightmap(dem_path, None, output_folder)
    generate_road_mask(road_path, None, output_folder)
    generate_landuse_mask(landuse_path, None, output_folder, landuse_mapping)

---

[vbassn]

@dwastberg Perhaps now decide how to import dtcc and use out data structures/methods etc.

[snjsomnath]

Some updates here @vbassn @dwastberg The unreal tiles workflow is now complete. It can generate unreal tiles from DEM, Landuse, Street network.

How to get buildings to work There is a "subworkflow" that brings in the buildings. This also works*. Something worth an explanation here is how to translate the buildings mesh to line up with UE.

The UE workflow generates a minimum overlapping bounding box.

These bounds are passed to Builder. From the resulting mesh we need to apply the following translation

translationx = -bounds.xmin **(Negative)_* translation_y = -bounds.ymin+ ((unreal_resolutioncellresolution)-bounds.height) **(Negative)_ translation_z = centroid of 3D bounding box for combined building meshes (Positive)**

Note: translation_x may be incomplete since I have only tested with a single tile. Further testing required.

Where: unreal cell resolution is the valid UE cell resolution of a single UE tile (1009 pixels per tile by default) cell_resolution is the m/per pixel resolution of DEM that is passed to the workflow (2m if coming via Lantmäteriet)

*What doesn't work Unreal can consume several 3D formats via Datasmith however, none of them are directly compatible with the supported builder formats. FBX is a good option but requires some extra work as of this comment. I have been testing using manual conversions to 3DM via rhino.

Once this is fixed, the buildings may be included in the main workflow.

What is next Like @vbassn mentioned, the next step is to replace io functions with dtcc_io. This requires some reworking in the landuse and street network model. Replace the rasters with the DTCC data model Replace the LM DEM raster with a raster generated through builder using LiDAR