Servers slow to (cold) start with persistent boards

[dmurfet]

When nine full persistent boards are in TRS, cold server startup times exceed 30 seconds.

Baseline: Server startup times for TRS with empty boards, measured from tapping the "Join" button on an iPad Pro and from the Roblox loading dialog first appearing on a Macbook Pro, to the loading screen disappearing. Times are reported in seconds both for warm start (somebody already in the server) and cold start (nobody else in the server).

iPad Pro warm start {10, 4, 4} avg = 6
iPad Pro cold start {16,8,10} avg ~= 11
Macbook Pro warm start {11,7,7} avg ~= 8
Macbook Pro cold start {12,12,12} avg = 12

With nine full persistent boards: using the 8-fold board to replicate a full board (shown above) so that there are nine full persistent boards, the startup times are as follows:

iPad Pro warm start {7,6,7} avg ~= 7
iPad Pro cold start {32, 33, 32} avg ~= 32
Macbook Pro warm start {8,14,8} = 10
Macbook Pro cold start {34,34,38} ~= 35

The good news is that warm startup is not significantly longer. However cold startup times more than double with the persistent boards in the world.

I do not understand why; if the GetAsync calls are slow it seems like the server should still spawn in the player, so maybe there is a holdup somewhere in the client-side code that can be fixed. Ideally, the player should spawn into the world and be active ASAP and the boards can asynchronously load in (remaining locked until they have finished). But none of this should delay the player appearing in the server. This appears to be the case on warm startups but not on cold startups.

[dmurfet]

Incidentally the memory usage for 9 full boards was 1900Mb on my Macbook Pro.

[sheepish12]

That's strange. Are those numbers from one test or an average of multiple? Maybe this is a one off

[dmurfet]

Yeah an average of three experiments in all 8 configurations. I don't think they are a one off.

[sheepish12]

Could you test this with a previous release of metaboards where loading was synchronous? This would help us determine if the asynchronous loading is the issue

[dmurfet]

Yeah I can do that, probably tomorrow though.

[dmurfet]

Mostly resolved in v0.4.1 so I'm closing for now.

Under the same conditions as the earlier experiments (and v0.4.1). With empty board:

iPad Pro warm start {10, 5, 5} avg ~= 7
iPad Pro cold start {9, 10, 13} avg ~= 11

With full persistent board and 8 subscriber boards:

iPad Pro warm start {8, 11, 8} ~= 9
iPad Pro cold start {19, 16, 17} ~= 17

You'll note the cold start time has been nearly halved.

[dmurfet]

Startup.server.lua now contains task.delay( N, Persistence.Init ). That is, loading the persistent boards is pushed somewhat out of server startup with task.delay. Here are the effects of various values of N.

delay 1
Text on first board completely loaded
Mac cold start {22, 20, 12, 20} avg ~= 19
iPad Pro cold start {20, 16, 16, 17} ~= 17

delay 2.5
One or two lines of text loaded on board
Mac cold start {20, 19, 18, 10} ~= 17
iPad Pro cold start {14, 7, 9, 9} ~= 10

delay 5
Text starts to load in immediately
Mac cold start {10, 10, 12, 16} = 12
iPad Pro cold start {11, 11, 9, 10, 10} ~= 10

delay 10
Text loads in after a delay
Mac cold start {10, 11, 10, 12} ~= 10
iPad Pro cold start {9, 11, 9, 11, 9} ~= 10

iPad Pro warm start {6, 7, 6, 6} ~= 6

In v0.4.2 the delay is set to 5 since this seems to be the sweet spot.