Add StagedTo3DConverter class with initial methods

[robyngit]

This PR includes some methods that I've been using to convert a batch of staged vector tiles into 3d tiles. I've packaged these all into a StagedTo3DConverter class, because I think it might be useful for both testing and for the viz-workflow. It essentially replicates what the RasterTiler class in the viz-raster repo does, except converts to 3d tiles rather than to raster files.

Using this new class enables:

• passing the same config object that is used by viz-staging and viz-raster
• deduplicating the vector files before conversion first if necessary
• using the same z/y/x tile path naming scheme convention that's specified in the config

It also includes an optional create_parent_json method that can be used to combine the individual tileset.json files into one. However, this is not yet working properly, which is why this PR is still a draft. I think that the calculation for the boundingVolume or geometricError is wrong. In any case, this method probably won't work for millions of files. It's just a temporary measure for testing purposes until https://github.com/PermafrostDiscoveryGateway/viz-3dtiles/issues/9 is ready.

@laurenwalker is viz-3dtiles the right repo for this class, or do you think that it should be part of viz-workflow, or is this class redundant to something that you are already working on? Also, if you have a minute to look over my StagedTo3DConverter.create_parent_json() method, I think you might be quicker than me at figuring out what's going wrong. Everything loads in Cesium without errors, but no tiles are visibile. I have an example up on sandcastle. The parent tileset.json is here.

Examples

Here are some examples on how this class can be used:

import viz_3dtiles
import logging

# Log to the console
logging.basicConfig(level=logging.INFO)

# An example config. Can alternatively pass `path/to/my/config.json`
my_config = {
  "dir_staged": "path/to/staged/files",
  "dir_3dtiles": "path/where/we/should/save/3dtiles",
  "deduplicate_at": ["raster", "3dtiles"],
  "deduplicate_method": "neighbor",
  "deduplicate_keep_rules": [["staging_filename", "larger"]],
  "deduplicate_overlap_tolerance": 0,
  "deduplicate_overlap_both": false,
  "deduplicate_centroid_tolerance": null
}

converter = viz_3dtiles.StagedTo3DConverter(my_config)

# Find all of the 3d tiles from the staged directory, and convert them to b3dm
# using the same tile-path naming convention but in the 3dtiles directory.
# Because parent_json is true, create one main tileset.json rather than individual json files
converter.all_staged_to_3dtiles(parent_json=True)

We can alternatively convert each of the staged files in parallel

import parsl
from parsl import python_app

# Set up parsl here...

@python_app
def convert(path, config):
    import viz_3dtiles
    converter = viz_3dtiles.StagedTo3DConverter(my_config)
    converter.staged_to_3dtile(path)

converter = viz_3dtiles.StagedTo3DConverter(my_config)
staged_file_paths = converter.tiles.get_filenames_from_dir('staged')

for path in staged_file_paths:
    convert(path, config)

# When conversions are complete, can create a parent tileset.json with:
converter.create_parent_json()

[robyngit]

The tileset.json created by create_parent_json() was not working because the geometric error was too small. Taking the sum of the geometric error for each child tile, rather than the max, seems to do the trick.

[laurenwalker]

Hi Robyn, this looks great! Thanks for coding this. Yes, this is redundant with what I was working on, but since you've got something complete, let's roll with it!

@laurenwalker is viz-3dtiles the right repo for this class, or do you think that it should be part of viz-workflow

I'd say viz-workflow would be a better place for this since your new class acts as an orchestrator of the 3D Tile classes as part of the overall workflow. I'm especially concerned about pdgstaging becoming a dependency of viz-3dtiles since so much of the pdgstaging interface is not used by viz-3dtiles and I've tried to keep viz-3dtiles generic enough that it doesn't need to be just for the PDG workflow, it is simply a Python package that creates tiles and tilesets.

However I do like the create_parent_json method and I wonder if we could incorporate that into the Cesium3DTileset class without depending on pdgstaging.

[robyngit]

I'd say viz-workflow would be a better place for this since your new class acts as an orchestrator of the 3D Tile classes as part of the overall workflow. I'm especially concerned about pdgstaging becoming a dependency of viz-3dtiles since so much of the pdgstaging interface is not used by viz-3dtiles and I've tried to keep viz-3dtiles generic enough that it doesn't need to be just for the PDG workflow, it is simply a Python package that creates tiles and tilesets.

These are very good points! I think I will follow the same philosophy for viz-raster.

However I do like the create_parent_json method and I wonder if we could incorporate that into the Cesium3DTileset class without depending on pdgstaging.

I've removed the pdgstaging dependency, and added the create_parent_json method to Cesium3DTileset. The StagedTo3DConverter class is now in the viz-workflow repo. It still has a create_parent_json method, but it organizes the relevant file paths and then calls Cesium3DTileset.create_parent_json

[laurenwalker]

Looks great, thanks Robyn!