Reduce computation time of Tilers when adding textures

[LorenzoMarnat]

Adding textures to the 3DTiles with CityTiler multiply the computation by x50.

• [x] Look for Python profiling tools
• [x] Try to reduce computation time

[LorenzoMarnat]

This PR reduces the computation time of textured 3DTiles by querying the images only once per tile. We used to query the images for each surface.

[LorenzoMarnat]

Python standard module cProfile allows to profile Python code.

In code

Import modules:

import cProfile
import pstats

Profile the code between enable() and disable():

cp = cProfile.Profile()
cp.enable()  # Start profiling

# code here

cp.disable()  # Stop profiling
p = pstats.Stats(cp)
p.sort_stats('tottime').print_stats()  # Sort stats by time and print them

In command line

cProfile can be run in the shell with:

python -m cProfile script.py

Results with CityTiler

Command (cProfile used directly in code):

citygml-tiler --db_config_path py3dtilers/CityTiler/CityTilerDBConfig.yml --type water --with_texture > log.txt

10 first lines of stats:

6046598 function calls (4748843 primitive calls) in 65.330 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1195   27.187    0.023   27.187    0.023 {method 'encode' of 'ImagingEncoder' objects}
     1863   11.867    0.006   11.867    0.006 {method 'decode' of 'ImagingDecoder' objects}
1237415/3170    7.818    0.000   13.338    0.004 /mnt/c/Users/lmarn/Documents/GitHub/py3dtilers/py3dtilers/Texture/atlas_node.py:30(insert)
  1244871    2.176    0.000    2.176    0.000 /mnt/c/Users/lmarn/Documents/GitHub/py3dtilers/py3dtilers/Texture/atlas_node.py:20(isLeaf)
       39    1.452    0.037    1.453    0.037 {function NamedTupleCursor.execute at 0x7f54378e5550}
   305759    1.101    0.000    1.622    0.000 /mnt/c/Users/lmarn/Documents/GitHub/py3dtilers/py3dtilers/Texture/atlas_rectangle.py:46(perfect_fits)
   626385    1.062    0.000    1.062    0.000 /mnt/c/Users/lmarn/Documents/GitHub/py3dtilers/venv/lib/python3.8/site-packages/PIL/Image.py:556(size)
   302597    1.056    0.000    1.564    0.000 /mnt/c/Users/lmarn/Documents/GitHub/py3dtilers/py3dtilers/Texture/atlas_rectangle.py:37(fits)
    63720    0.630    0.000    1.296    0.000 /mnt/c/Users/lmarn/Documents/GitHub/py3dtilers/venv/lib/python3.8/site-packages/numpy/core/numeric.py:1341(normalize_axis_tuple)
    31860    0.586    0.000    2.248    0.000 /mnt/c/Users/lmarn/Documents/GitHub/py3dtilers/venv/lib/python3.8/site-packages/numpy/core/numeric.py:1404(moveaxis)

When adding textures to 3DTiles, most of time is taken by Pillow encoding/decoding

[LorenzoMarnat]

According to this Pillow's issue, changing the compression level when saving images allows to reduce computation time.

atlasImg.save(Texture.folder + '/tiles/ATLAS_' + str(tile_number) + '.png', compress_level=3)

Compress level	Computation time (s)	Atlas size (mo)
6	68.65	76.0
3	60.90	77.0
1	49.45	89.4

[jailln]

Is it a lossy or a lossless compression ? Maybe it could be a parameter offered by py3dtiles with a default value ?

[LorenzoMarnat]

Pillow uses the lossless data compression library zlib to compress images

[LorenzoMarnat]

Pillow image compression will be handled with the compression issue. Closing