ErixenCruz
commented
3 years ago I have a branch that is useing OpenMP and a thread safe vector to process elevation tiles in parallel (and keep track of availability bits for implicit): https://github.com/CesiumGS/cdb-to-3dtiles/tree/multithreadElevation . The problem is that you only get around x3 speed up with 12 threads. The threads do fine on their own with mesh_opt'ing and writing the gltfs and b3dms. They don't need to talk to each other. The problem comes when they need to convert all of the imagery from jp2 to jpeg. Because we use GDAL to do the conversion, a lot of their time is wasted waiting on a lock. GDAL uses a global cache to store data (imagery), so when a thread requests to read block 1-10 of its jp2, GDAL will lock blocks 1-10 for all other threads, even if those other threads are reading blocks from their own jp2! Confirm this by running the branch and randomly stopping in the debugger. You can see that almost every time you stop, most of the threads will be waiting on GetLockedBlockRef. Read more about it here: https://gdal.org/development/rfc/rfc47_dataset_caching.html . I tried giving each thread their own GDALDriver, but it doesn't help. I think the solution is to abandon GDAL for this case. Perhaps use https://github.com/uclouvain/openjpeg directly.
Process CDB tiles in parallel. Not too much synchronization should be required.