History persistence lets .sfs files get very large

[lpgagnon]

from discord discussion. Not a big problem so far, and too early to be proposing "fixes"; but worth tracking and gathering more data.

rp-1 career, ~15 active vessels including a ~5-year-old first orbit vessel; .sfs size was up to 280MB. Deleting just the first-orbit vessel reduced it by 60MB. Cleaning up all vessels from completed contracts brings it down to 24MB.

Main impact is on save/load speeds; saving was up to ~5s on SSD (presumably it's generating the data, not writing the file that's the bottleneck). Particularly annoying because this includes autosaves, so this can introduce long pauses the player doesn't control.

Secondary, minor impact is that big .sfs files are less easy to edit, share, etc.

[eggrobin]

More data:

[22:04]RurouniDonut:Deleting old vessals just made my save go from 252 mb to 6 mb

[lpgagnon]

https://drive.google.com/open?id=15bLvwZiRsIiOOg602r4tfFR7fna_4v4P

162MB sfs, of which ~155MB is from 2 vessels ~1500d old.

[pleroy]

I have managed to read the save and produce a .wl file for the vessel trajectories. The code to do this is in https://github.com/pleroy/Principia/tree/2400 (some of that is throw-away, but some will ultimately be checked in).

The output file (776 MiB...) is on Google Drive and shared with @eggrobin. Good luck trying to read it in Mathematica.

[lpgagnon]

https://drive.google.com/open?id=1RDNwA2KRJU1miBiLIhgaaRehjXtOVdVf 133MB sfs, reduces to ~25MB after deleting ~15 vessels aged 0.5y-3y in various orbits around Earth, Mars & Venus.

Notable (in both linked saves): simple zipping reduces the size by a good third. So if nothing else, different compression algorithms could be worth investigating (though higher compression might come at the cost of slower saves & loads).

[pleroy]

Notable (in both linked saves): simple zipping reduces the size by a good third. So if nothing else, different compression algorithms could be worth investigating (though higher compression might come at the cost of slower saves & loads).

I noticed the same, but it mostly comes from the fact that the save uses base64, i.e., 6 bits per bytes. It stands to reason that zipping would save 25%. A better compression algorithm (zstd?) would probably not save more than ~5%.

[damianreds]

hypervelocity from KSP forums here, providing a hefty persistent file for your analysis. I tried uploading it to gist but website kept crashing before finishing upload. Let me know if anything else is needed or if there's any other way I can help.

https://drive.google.com/file/d/1pxOinC2T4251Q8no2SX1sds2FfulqMsU/view?usp=sharing

[pleroy]

@damianreds: Thanks, I might try to see how the code in #2485 helps with your save.

[pleroy]

Running #2485 on the save dated Jan 31 above. Looking only at the Principia part, i.e., the part that's encoded in base64.

Before (gipfeli, base64): 156'075'134 bytes. After (zfp+gipfeli, base64): 42'282'108 bytes. Gain: 3.7x.

[pleroy]

Running #2485 on the save provided by @damianreds. Looking only at the Principia part, i.e., the part that's encoded in base64.

Before (gipfeli, base64): 83'633'693 bytes. After (zfp+gipfeli, base64): 20'934'426 bytes. Gain: 4.0x.

Curiously this save is very slow to deserialize (135 s), irrespective of zfp. To be continued...

[pleroy]

@damianreds: Your pain comes from the fact that your game is in the year 2023 (I suspect that you started by warping all the way to 2020 or some such). On every scene change, Principia recomputes the history of the solar system from 1951 to 2023. That takes about 2 minutes.

Obviously that's not very smart of us, but then the alternative of explicitly storing 72 years of history has its own drawbacks (it runs out of memory on my machine, for instance).

This will need to be addressed in a future release, but it will take time. In the meantime, I recommend playing closer to the origin date of 1951 if possible.

[damianreds]

@pleroy thank you Pascal! I very much appreciate your insights. You are correct in your assumption, I did start by warping all the way through 2020. Don't worry about it, the fantastic experience Principia provides significantly outweighs a few mins of wait time. I will be looking forward for a future fix but this will not prevent me from enjoying your work. Thank you & the team for your awesome work!

EDIT: @pleroy now that I think about it, is this issue first noticed now? I have been using Principia for a while now and always fast forwarded to roughly our present time and this is the first time I have experienced such significant delays.

[pleroy]

@damianreds: The "problem" was introduced in פרנקל. In that version we started to plot the trajectories of celestials. At the same time, we dropped the capability to forget the past histories; instead the setting of the history length hides the histories but doesn't delete them. Overall this seems useful because, well, all these trajectories look nice.

We were aware that it would make saves bigger/slower, but the situation is not all that bad if you are a few years after 1951. We just didn't anticipate your use case, where 70 years would need to be integrated on each scene change.

[damianreds]

got it, thank you, very much appreciated! :)

[Peter-JY]

游戏加载的时候能直接从.sfs加载当前时间飞船的位置，而不是重新计算一遍吗？还有，能不能说让这些历史记录不放到内存里（如果必须的话放到页面文件里）来节约一些内存？我只有8G内存，加载RP-1就很耗时间、很占内存了，我不希望这个也占用我的内存，让我的游戏更卡了。

[cloudeecn]

Is it possible to provide an option to forget the history over 2 weeks of some specified vehicle? I have some long-running vehicles, such as communication satellites around bodies. I don't necessarily care about their trajectories. Maybe making the history of them transient could make the size more manageable. (I would appreciate even a crappy one like making the history transient if the vehicles have some prefix in their names.)

[Myshiko]

Slowing down scene changes is actually extremely harmful for the game play, as KSP does require a lot of scene changes to do even the most basic things. Having more memory usage instead of slow down is preferable. 32 / 64 GB machines are normal this days. 2-minute scene change waits quickly add-up to the time when it becomes more money efficient to simply buy more RAM (lost time is money).

How about the following idea: have a configurable number of GB to allocate to the in-memory persistence of system state. If that number is exceeded, recompute the missing portion and drop a message somewhere "tune that particular number to get faster scene changes".

This would allow people with sufficiently powerful machines(64 - 128 GB) allocate as much as they want and not worry about slowdowns, while smaller machines (16 gb or so) still both don't crash and people know how to address the problem (get more RAM).

[pleroy]

If you come here to comment on this issue, please provide facts: on my computer which has characteristics X, the attached save takes Y seconds to load and Z seconds to write. And please give us your save.

Unless you are an astronomer or a Squad developer, your suggestions are not going to tell us anything that we didn't know about the problem.

[RurouniDonut]

https://drive.google.com/file/d/1GB6LWSBCE67M0YhpufxOs5Gi44uhzTbK/view?usp=sharing

This save is around 16 years into the game. It has a mix of interplanetary probes, earth sats, lunar sats, and lunar surface sats.

[RCrockford]

Extra data point, this save is largely cleared of long term vessels (my PC can't cope), but there is one called Venus Orbiter 1 which has 934 days of history, mostly in a fairly boring Venus orbit. https://1drv.ms/u/s!ApWZXOEZJxA5gpolNVKFVgmh2VA17g?e=7qgPcj

[szundi]

Slowing down scene changes is actually extremely harmful for the game play, as KSP does require a lot of scene changes to do even the most basic things.

Hello guys, I can tell you these scene change delays are getting worse here too. It is barely playable now. I keep on playing though because Principia is great, but I think in 2 weeks when loadings become like 5 minutes, it'll be 30 minutes to try a launch.... I happily give up some history data for this.

I have a lot of serialized_plugin values that takes up most of my sfs file. (could not be attached, the link is here: https://idat-my.sharepoint.com/:u:/g/personal/andras_szabo_idat_onmicrosoft_com/EVK6vm8QbNFHqP_N3pc4hLUBnU-hStc-6Bp-BYyAY73cjQ?e=GDtLlS)

Principia gives me lots of joy though! :) Please keep up your good work, and tell me where can I help with this maybe.

[sohmien]

I'm running into this issue too, and it's very frustrating. RP1 career in 1967, ~15 active vessels, scene load times are a couple of minutes and the game is no longer playable :(
edit: Not a hardware issue, KSP is on SSD and the machine has 32Gb RAM.

[pleroy]

Copying here a post from Discord:

Unfortunately, despite considerable efforts trying to find a solution to #2400, the results so far have been underwhelming.

The basic idea, largely derived from [Kud07], was to construct a compressed representation of trajectories using Poisson series. Ignoring the fact that there are significant shortcomings to the numerical techniques presented by [Kud07], the compression algorithm can be made to work. The problem is that it cannot be made to work in a way that is suitable for use in a game. Astronomers have all the time they need to compute compressed ephemerides. In KSP, we have 1/50 second to compress as much as 1 hour of evolution of the solar system, assuming a warp factor of 100 000×.

For large bodies (read: planets) or bodies that are reasonably isolated from the rest of the solar system (e.g., Pluto/Charon) the compression algorithm is actually quite fast and yields fairly accurate results. Problems start to appear, though, when dealing with satellites. Consider for instance Phobos. It moves in the fairly bumpy gravitational field of Mars, so it is really bobbing around like a cork in water. This means that its motion is not at all regular, being affected by Olympus Mons and other lumps of matter. Therefore, compressing the trajectory of Phobos requires an inordinate amount of work. Roughly, it takes 1 second of CPU to compress 1 day of trajectory. Since there are around 20 satellites in the solar system, it's easy to see that the maximum warp speed that could be reached would be around 4000×. Even with superhuman optimization efforts, it's hard to see how we could exceed warp 10 000×. Obviously, that's not workable. And that's only for celestials. Vessels would have all the problems of small satellites and more.

We have ideas on less exotic (but less risky) approaches, but they will take time to develop.

Whoever at Squad decided that scene changes would be best effected by (1) writing all the state to disk (2) throwing it away (3) reloading it from disk has a comfy little hot spot waiting for them in hell.

[pleroy]

2930 is preparatory work for another attempt at solving this issue.

[sohmien]

I'm thinking about this: "On every scene change, Principia recomputes the history of the solar system from 1951 to 2023. That takes about 2 minutes."

Maybe the issue could be partially fixed as follows:

1. When the game starts for the first time, compute the positions of all bodies in the solar system for 100 years in the future.
2. Save the results for the first day of each year (e.g. 1951-2051).
3. On scene change, recompute the history of the solar system not from 1951, but from the beginning of the previous year. E.g. on 02.03.1967, load the state of the solar system as of 01.01.1966 and compute from there.

This should give a substantial performance benefit, especially for later dates, with very little implementation effort, compared to the complicated solution described above. It should work for any solar system (RSS, stock or whatever). The only downside is that the history is limited to one year, but it's probably not a major problem, and one year could even be an adjustable setting.

[eggrobin]

As has been said above,

Unless you are an astronomer or a Squad developer, your suggestions are not going to tell us anything that we didn't know about the problem.

A shortage of ideas has never been the issue, and estimates of difficulty from those unfamiliar with the innards of Principia have little value.

On the other hand, half-thought-through suggestions on how to address a problem we have been studying for more than a year are very effective at making us want to work on something more interesting instead.

I am locking this issue; we have all the data we need, and we will post updates as they come.

[pleroy]

A progress report, in case anyone is following this issue.

I just merged #2972, which is the last step in trying to alleviate the cost of scene changes due to celestials. It does nothing for vessels. It won't go in Green (May 11) because it will need extensive testing, but if all goes well it should be in Gröbner (June 10).

No fancy math here (sigh), just pedestrian programming to improve the user experience:

• When integrating the celestials, Principia periodically takes a checkpoint, which is an object containing enough information to reconstruct the future. Checkpoints are relatively compact (100 KiB uncompressed for RSS) and are taken at least every 180 days, and sometimes more frequently.
• When loading a save, the last checkpoint before the current game time is used to reconstruct the trajectories of the celestials up to the present. This blocks the UI thread, but it covers at most 180 days.
• At the same time, an asynchronous thread it started to reconstruct the past by processing the checkpoints backwards: the 180 days immediately before the present are first recomputed, then the 180 days before that, etc. This recomputation stops when enough history has been produced to cover the Max history length chosen by the user.

This approach has two benefits:

• After a scene switch, the UI becomes available quickly (seconds) and the trajectories of the celestials reappears progressively (about 1 segment per second). This make the game much more responsive at the expense of some flickering.
• It is virtually never necessary to recompute the trajectories all the way back to 1950 (in RSS). A user playing around 2020 will probably only display a few years of history, which should reconstruct in seconds.

A similar approach could be used for vessels, but there the situation is complicated by burns. Be patient.

[pleroy]

We've done some measurements to understand where the time is spent when loading saves with long vessel histories.
As usual with benchmarks, the numbers are "on my machine", but the essence of the problem is independent from hardware details. It's worth noting though that storage is using SSD. The numbers are for stock KSP 1.11.2 and the times are in KSP years of 2556.50 hours.

The overall speed for loading a save is 9.3 MiB/s. KSP itself loads at 12.6 MiB/s, and Principia at 34.9 MiB/s. In other words, KSP accounts for 75% of the time, Principia for 25%. To add insult to injury, KSP seems to have a fixed cost of 3 s irrespective of the size of the save.

The Principia speed is not great, and could possibly be improved by 2×, but that would only be an improvement of 10% overall and nobody would notice.

As is well-known (to us, at least), the orbits with high curvature are way more expensive than the nearly-hyperbolic ones. To quantity this we compared a middle altitude Kerbin orbit and an orbit escaping the Kerbol system. The former takes about 250 KiB/year, the latter 1 KiB/year.

It's interesting to compare the load time to the time it would take to recompute the trajectory from scratch. A 300-year middle altitude orbit takes about 11 s to load. To reconstruct that orbit in the same time we would need to be able to warp at 250×10⁶. In practice we barely reach 10⁶, so recomputing would be 250 times slower than loading.

A few inescapable conclusions:

1. Don't try to speed up the read/deserialization code of Principia, there is nothing to be gained.
2. Try to make the save smaller by any means: better compression, storing less data, storing data on the side, etc.
3. Don't try to recompute the past; it might work for satellites in low orbit, where one would want to only see a short segment, but it won't work for long-distance probes. Note that the situation is different from that of celestials because we are able to recompute the entire real solar system at 16×10⁶.

It is not exactly clear what is the way forward, but this analysis tells us at least what approaches can be discarded.

Oh, and don't delete your deep space probes, they cost nothing.