CesiumGS / cesium-native

Apache License 2.0
415 stars 210 forks source link

Implement Staged Tile Loading Pipeline #779

Open csciguy8 opened 9 months ago

csciguy8 commented 9 months ago

When loading tiles, implement a new staged pipeline where network requests are dispatched separately from processing work.

This achieves a 25-32% loading time reduction in some test cases.

Closes #746. Also closes #473 as a side effect.

Performance Tests Summary (Cesium for Unreal Performance Tests)

SampleLocaleDenver (cold cache) - 21% slow down
SampleLocaleDenver (warm cache) -15% slow down
SampleLocaleMelbourne (cold cache) - 5% improvement
SampleLocaleMelbourne (warm cache) - about the same
GoogleTiles.LocaleDeathValley (cold cache) - 32% improvement
GoogleTiles.LocaleDeathValley (warm cache) - 15% slow down
GoogleTiles.LocaleChrysler (cold cache) - 25% improvement
GoogleTiles.LocaleChrysler (warm cache) - 9% slow down
Complete Testing Data ``` cesium-native main -------------------------------------------------------------------------- SampleLocaleDenver (27 MB) Cold cache - 2.02, 2.03, 2.08 | 2.96, 2.45, 2.09, 2.15 - 2.04 avg (best 3) Warm cache - 0.81, 0.90, 0.91 | 0.98, 0.98, 0.93, 0.94 - 0.87 avg (best 3) SampleLocaleMelbourne (89 MB) Cold cache - 3.38, 3.38, 3.45 | 3.54, 4.10, 4.26, 3.54 - 3.40 avg (best 3) Warm cache - 1.75, 1.76, 1.81 | 1.96, 2.09, 1.81, 1.83 - 1.77 avg (best 3) GoogleTiles.LocaleDeathValley (33 MB) Cold cache - 4.63, 4.75, 4.91 | 5.17, 5.08, 4.96, 4.96 - 4.76 avg (best 3) Warm cache - 1.92, 1.96, 1.98 | 5.07, 2.00, 2.03, 2.02 - 1.95 avg (best 3) GoogleTiles.LocaleChrysler (62 MB) Cold cache - 6.69, 6.84, 6.84 | 7.04, 7.09, 6.98, 6.96 - 6.79 avg (best 3) Warm cache - 3.06, 3.16, 3.21 | 3.62, 3.26, 3.25, 3.26 - 3.14 avg (best 3) this PR -------------------------------------------------------------------------- SampleLocaleDenver (27 MB) Cold cache - 2.43, 2.46, 2.52 | 2.99, 2.86, 2.77, 3.05 - 2.47 avg (best 3) Warm cache - 0.97, 1.01, 1.04 | 1.12, 1.08, 1.09, 1.05 - 1.00 avg (best 3) SampleLocaleMelbourne (89 MB) Cold cache - 3.19, 3.25, 3.27 | 4.14, 5.17, 3.31, 3.84 - 3.23 avg (best 3) Warm cache - 1.78, 1.81, 1.88 | 1.94, 2.03, 2.06, 1.95 - 1.82 avg (best 3) GoogleTiles.LocaleDeathValley (33 MB) Cold cache - 3.22, 3.24, 3.24 | 3.25, 3.32, 3.38, 3.57 - 3.23 avg (best 3) Warm cache - 2.24, 2.25, 2.26 | 2.28, 2.29, 2.37, 2.38 - 2.25 avg (best 3) GoogleTiles.LocaleChrysler (62 MB) Cold cache - 5.05, 5.16, 5.17 | 5.20, 5.20, 5.37, 5.17 - 5.12 avg (best 3) Warm cache - 3.39, 3.42, 3.47 | 3.51, 3.53, 3.60, 3.57 - 3.42 avg (best 3) ```

In Depth

... Simplified view of existing tile loads ![image](https://github.com/CesiumGS/cesium-native/assets/130494071/e9fc0024-ad22-4443-87ae-4fa0be3c5bc5) This PR introduces this structure ![image](https://github.com/CesiumGS/cesium-native/assets/130494071/82b81a62-9eaf-4063-aa87-669c31f6e8ef) The best place to start looking at the code changes would be in `Tileset:: _processWorkerThreadLoadQueue`. Previous load requests were handled immediately, but now they are now passed to `TilesetContentManager`, which transforms the requests into new data that `TileWorkManager` can work with, a generalized `TileLoadWork`. `TileWorkManager` handles the critical logic to queue / throttle / and execute `TileLoadWork` objects.

TO DO

csciguy8 commented 6 months ago

All major restructuring is done and have posted (in the PR description) the latest Google P3DT performance numbers using Cesium for Unreal performance tests.

A snippet...

GoogleTiles.LocaleDeathValley (cold cache) - 25% load time improvement
GoogleTiles.LocaleDeathValley (warm cache) - 44% load time slow down
GoogleTiles.LocaleChrysler (cold cache) - 17% load time improvement
GoogleTiles.LocaleChrysler (warm cache) - 48% load time slow down

@kring , keeping you in the loop. Oddly enough, looks like we're still seeing roughly the same improvements as originally guessed back in November, when the original issue was written, 17%-25% improvements for Google P3DT scenes in Unreal. And we're doing that without increasing the max simultaneous loads (still 20).

Unfortunately, the warm cache numbers got worse. The reason is not obvious, but there may be something we can do to make this better. Will add a check box to the TO DOs.

kring commented 6 months ago

This is a total guess, but one thing to watch out for is that the warm cache case is especially suceptible to slowdowns caused by waiting to run tasks. For example, a simple thenInMainThread somewhere in the load processing can cause a delay of 16ms (at 60FPS) before that part of the task can run, which is quite a long time with a warm cache.

csciguy8 commented 6 months ago

Finished performance testing and updated the missing numbers.

A snippet

SampleLocaleDenver (cold cache) - 16% load time slow down
SampleLocaleDenver (warm cache) -22% load time slow down
SampleLocaleMelbourne (cold cache) - about the same
SampleLocaleMelbourne (warm cache) - about the same

A little odd that Denver slowed down a bit. It is our smallest test, and already pretty fast. First guess is that this PR is indeed batching content requests better as shown for the google levels. But for cached (or very fast) content requests, it may be introducing delay (or more overhead) for processing. Like @kring said, this could be an errant thenInMainThread, or something similar that is fixable.

I think this is ready for review and to talk a bit more about performance numbers.

Making this a non-draft PR...

csciguy8 commented 6 months ago

I think I have a solid lead for the slowdown in the Google warm cache case...

If I turn off the time budget in Tileset::_processMainThreadLoadQueue, the loading times in Google Chrysler (warm) between main and this PR become almost the same (within .1s).

This function is throttling the final stage of tile loading based on a time budget, to keep a reasonable frame rate.

In the context of this PR, we can think of it as the final 'stage' in our staged loading pipeline. And taking into account what this PR is doing, it seems likely that it's pushing the same amount of work through this stage, but with bigger batches of work, later in time, ultimately leading to longer load times.

Will think on this a bit more...

csciguy8 commented 6 months ago

Found the slowdown in the Google warm cache case! (finally).

TLDR, the reference branch (based on main) I'm comparing to had a bug where it completed too early. Fixing the bug and doing a quick retest showed me that this PR doesn't seem to introduce any weird slow downs in this case anymore. Will retest to verify.

I've fixed it in my branch here, but it could optionally be its own PR as well, since it's a bug in main too.

More in depth, the problem is in ::computeLoadProgress and is reporting 100% done before we are actually done. It uses this code to determine if there's any work in the any of our loading queues... int32_t queueSizeSum = static_cast<int32_t>( this->_workerThreadLoadQueue.size() + this->_mainThreadLoadQueue.size());

Unfortunately, _mainThreadLoadQueue gets cleared every frame, so this is always 0, and any progress that is computed is ignoring any main thread loading that happened, or was ever queued.

This behavior coincides with what I saw in the original tests. Inexplicably fast performance compared to my PR, where I had inadvertently fixed this bug through refactoring. It also coincides with my observation of equal performance when turning off time budget. Doing this clears the main thread queue every frame, which essentially makes it always 0 when ::computeLoadProgress is called.

csciguy8 commented 6 months ago

Another update to the performance test numbers...

Good news: the performance wins from this PR went up! (now 25-32% improvements for the google tests)

Mediocre news: although the inexplicable "warm cache is really slow" problem is gone, warm cache results are still slightly slower in this PR (9-21% slower). Denver cold cache falls into this category too. I have a pretty good reason to believe that the problem is here, which coincides with @kring 's recommendation to look out for main thread continuations.

The fix would be to dispatch processing work immediately, rather than waiting to dispatch on the main thread. Not a trivial change, but not an unreasonable amount of work either.

A snippet

SampleLocaleDenver (cold cache) - 21% slow down
SampleLocaleDenver (warm cache) -15% slow down
SampleLocaleMelbourne (cold cache) - 5% improvement
SampleLocaleMelbourne (warm cache) - about the same
GoogleTiles.LocaleDeathValley (cold cache) - 32% improvement
GoogleTiles.LocaleDeathValley (warm cache) - 15% slow down
GoogleTiles.LocaleChrysler (cold cache) - 25% improvement
GoogleTiles.LocaleChrysler (warm cache) - 9% slow down
kring commented 4 months ago

I'm seeing frequent crashes just by press Play and flying around in the 01_CesiumWorld level.

Here's one example:

    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!CesiumUtility::ReferenceCounted<CesiumRasterOverlays::RasterOverlayTileProvider,0>::addReference() Line 68   C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!CesiumUtility::IntrusivePointer<CesiumRasterOverlays::RasterOverlayTileProvider>::addReference() Line 187    C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!CesiumUtility::IntrusivePointer<CesiumRasterOverlays::RasterOverlayTileProvider>::IntrusivePointer<CesiumRasterOverlays::RasterOverlayTileProvider>(CesiumRasterOverlays::RasterOverlayTileProvider * p) Line 24 C++
>   UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TilesetContentManager::dispatchRasterWork(Cesium3DTilesSelection::RasterProcessingData & processingData, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & responseDataMap, Cesium3DTilesSelection::TileWorkManager::Work * work) Line 1802 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TilesetContentManager::createWorkManager::__l2::<lambda>(Cesium3DTilesSelection::RasterProcessingData & processingData, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & responseDataMap, Cesium3DTilesSelection::TileWorkManager::Work * work) Line 913   C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::invoke<void <lambda>(Cesium3DTilesSelection::RasterProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *) &,Cesium3DTilesSelection::RasterProcessingData &,std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> const &,Cesium3DTilesSelection::TileWorkManager::Work *>(Cesium3DTilesSelection::TilesetContentManager::createWorkManager::__l2::void <lambda>(Cesium3DTilesSelection::RasterProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *) & _Obj, Cesium3DTilesSelection::RasterProcessingData & _Arg1, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & <_Args2_0>, Cesium3DTilesSelection::TileWorkManager::Work * && <_Args2_1>) Line 1587 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::_Invoker_ret<void>::_Call<void <lambda>(Cesium3DTilesSelection::RasterProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *) &,Cesium3DTilesSelection::RasterProcessingData &,std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> const &,Cesium3DTilesSelection::TileWorkManager::Work *>(Cesium3DTilesSelection::TilesetContentManager::createWorkManager::__l2::void <lambda>(Cesium3DTilesSelection::RasterProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *) & _Func, Cesium3DTilesSelection::RasterProcessingData & <_Vals_0>, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & <_Vals_1>, Cesium3DTilesSelection::TileWorkManager::Work * && <_Vals_2>) Line 674    C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::_Func_impl_no_alloc<void <lambda>(Cesium3DTilesSelection::RasterProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *),void,Cesium3DTilesSelection::RasterProcessingData &,std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> const &,Cesium3DTilesSelection::TileWorkManager::Work *>::_Do_call(Cesium3DTilesSelection::RasterProcessingData & <_Args_0>, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & <_Args_1>, Cesium3DTilesSelection::TileWorkManager::Work * && <_Args_2>) Line 834 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::_Func_class<void,Cesium3DTilesSelection::RasterProcessingData &,std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> const &,Cesium3DTilesSelection::TileWorkManager::Work *>::operator()(Cesium3DTilesSelection::RasterProcessingData & <_Args_0>, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & <_Args_1>, Cesium3DTilesSelection::TileWorkManager::Work * <_Args_2>) Line 875 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TileWorkManager::transitionProcessing(std::shared_ptr<Cesium3DTilesSelection::TileWorkManager> & thiz) Line 558  C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TileWorkManager::transitionRequests::__l23::<lambda>() Line 476  C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!async::detail::invoke_fake_void<void <lambda>(void),void>(Cesium3DTilesSelection::TileWorkManager::transitionRequests::__l23::void <lambda>(void) && f) Line 103 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!async::detail::invoke_fake_void<void <lambda>(void)>(Cesium3DTilesSelection::TileWorkManager::transitionRequests::__l23::void <lambda>(void) && f, async::detail::fake_void __formal) Line 113   C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!async::detail::continuation_exec_func<CesiumAsync::CesiumImpl::ImmediateScheduler<CesiumAsync::CesiumImpl::QueuedScheduler>,async::task<void>,async::detail::fake_void,void <lambda>(void),std::integral_constant<bool,1>,0>::operator()(async::detail::task_base * t) Line 550  C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!async::detail::task_func<CesiumAsync::CesiumImpl::ImmediateScheduler<CesiumAsync::CesiumImpl::QueuedScheduler>,async::detail::continuation_exec_func<CesiumAsync::CesiumImpl::ImmediateScheduler<CesiumAsync::CesiumImpl::QueuedScheduler>,async::task<void>,async::detail::fake_void,void <lambda>(void),std::integral_constant<bool,1>,0>,async::detail::fake_void>::run(async::detail::task_base * t) Line 398    C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!async::task_run_handle::run() Line 134   C++

The immediate problem here appears to be that in TilesetContentManager::dispatchRasterWork, pRasterTile->getLoadingTile() is returning an invalid pointer.

image

Here's another example:

>   UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::_Func_class<CesiumAsync::Future<Cesium3DTilesSelection::TileLoadResult>,Cesium3DTilesSelection::TileLoadInput const &,Cesium3DTilesSelection::TilesetContentLoader *>::operator()(const Cesium3DTilesSelection::TileLoadInput & <_Args_0>, Cesium3DTilesSelection::TilesetContentLoader * <_Args_1>) Line 873   C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TilesetContentManager::dispatchTileWork(Cesium3DTilesSelection::TileProcessingData & processingData, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & responseDataMap, Cesium3DTilesSelection::TileWorkManager::Work * work) Line 1692 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TilesetContentManager::createWorkManager::__l2::<lambda>(Cesium3DTilesSelection::TileProcessingData & processingData, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & responseDataMap, Cesium3DTilesSelection::TileWorkManager::Work * work) Line 905 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::invoke<void <lambda>(Cesium3DTilesSelection::TileProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *) &,Cesium3DTilesSelection::TileProcessingData &,std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> const &,Cesium3DTilesSelection::TileWorkManager::Work *>(Cesium3DTilesSelection::TilesetContentManager::createWorkManager::__l2::void <lambda>(Cesium3DTilesSelection::TileProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *) & _Obj, Cesium3DTilesSelection::TileProcessingData & _Arg1, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & <_Args2_0>, Cesium3DTilesSelection::TileWorkManager::Work * && <_Args2_1>) Line 1587 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::_Invoker_ret<void>::_Call<void <lambda>(Cesium3DTilesSelection::TileProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *) &,Cesium3DTilesSelection::TileProcessingData &,std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> const &,Cesium3DTilesSelection::TileWorkManager::Work *>(Cesium3DTilesSelection::TilesetContentManager::createWorkManager::__l2::void <lambda>(Cesium3DTilesSelection::TileProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *) & _Func, Cesium3DTilesSelection::TileProcessingData & <_Vals_0>, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & <_Vals_1>, Cesium3DTilesSelection::TileWorkManager::Work * && <_Vals_2>) Line 674    C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::_Func_impl_no_alloc<void <lambda>(Cesium3DTilesSelection::TileProcessingData &, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> &, Cesium3DTilesSelection::TileWorkManager::Work *),void,Cesium3DTilesSelection::TileProcessingData &,std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> const &,Cesium3DTilesSelection::TileWorkManager::Work *>::_Do_call(Cesium3DTilesSelection::TileProcessingData & <_Args_0>, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & <_Args_1>, Cesium3DTilesSelection::TileWorkManager::Work * && <_Args_2>) Line 834   C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!std::_Func_class<void,Cesium3DTilesSelection::TileProcessingData &,std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> const &,Cesium3DTilesSelection::TileWorkManager::Work *>::operator()(Cesium3DTilesSelection::TileProcessingData & <_Args_0>, const std::map<std::string,CesiumAsync::ResponseData,std::less<std::string>,std::allocator<std::pair<std::string const ,CesiumAsync::ResponseData>>> & <_Args_1>, Cesium3DTilesSelection::TileWorkManager::Work * <_Args_2>) Line 875 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TileWorkManager::transitionProcessing(std::shared_ptr<Cesium3DTilesSelection::TileWorkManager> & thiz) Line 554  C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TileWorkManager::TryDispatchProcessing(std::shared_ptr<Cesium3DTilesSelection::TileWorkManager> & thiz) Line 249 C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::TilesetContentManager::processLoadRequests(std::vector<Cesium3DTilesSelection::TileLoadRequest,std::allocator<Cesium3DTilesSelection::TileLoadRequest>> & requests, Cesium3DTilesSelection::TilesetOptions & options) Line 963   C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::Tileset::_processWorkerThreadLoadQueue() Line 1454   C++
    UnrealEditor-CesiumRuntime-Win64-DebugGame.dll!Cesium3DTilesSelection::Tileset::updateView(const std::vector<Cesium3DTilesSelection::ViewState,std::allocator<Cesium3DTilesSelection::ViewState>> & frustums, float deltaTime) Line 396 C++

Here we're in dispatchTileWork and the immediate problem is that loaderCallback is empty. The assert didn't trigger, though, probably because they're disabled in Unreal.

image

csciguy8 commented 4 months ago

I'm seeing frequent crashes just by press Play and flying around in the 01_CesiumWorld level.

Those crashes look pretty severe and not obvious.

I can't seem to reproduce them though. I checked several of the sample levels in PIE mode and everything seems to run fine. Ran in these 4 configurations...

(this PR with cesium-unreal main)

Could there be something wrong with your set up?

kring commented 4 months ago

Could there be something wrong with your set up?

I don't have any reason to think so @csciguy8. But I'll try main to make sure it doesn't happen there. They look like race conditions to me, and I was able to reproduce various crashes in both Debug Editor (with Debug build of cesium-native) and Development Editor (with RelWithDebInfo build of cesium-native).

kring commented 4 months ago

I switched to main, flew around a whole bunch, and never saw a crash. Switched back to this branch and saw it within seconds. I can easily make it happen at will. I recorded a video, which I'll send to you separately.

csciguy8 commented 4 months ago

I switched to main, flew around a whole bunch, and never saw a crash. Switched back to this branch and saw it within seconds. I can easily make it happen at will. I recorded a video, which I'll send to you separately.

@kring Ready for another look regarding those crashes.

Thanks for the test. Looks like I never tested in cases where tiles would get to the unloading state faster than they would load.

electrum-bowie commented 1 month ago

Any progress on this ?

I can't wait for these improvements to come !

kring commented 1 month ago

This is on hold for the time being, due to other higher-priority work and because the performance we saw with this approach was a bit mixed (some things got notably faster, but others got slower).

electrum-bowie commented 1 month ago

Is there any other work being done for performance ?