support custom room counts from yaml

[Rnd-Guy]

Updated 07/10/24

This top post will be kept up to date and should always provide a summary of the current state of the PR

Adds custom room count as a yaml option, as well as agate knife percentage

In order to do this, the following were done:

• Connecting to the server is now forced before any gameplay begins, to get the room count before gen
• Added connection loading screens which also say when connection has failed
• Removed requirement for dungeon to have max_dist >= 50
• Room distance is multiplied by 50 / max_dist for enemy scaling, audio, horde room treasure calculations.
  • max_dist is reduced by ~3% to allow for a bit of overshooting, to mimic how vanilla sometimes goes above 50. This is to keep enemy proportions roughly similar
• Crystal scaling is increased by 10000 / total enemies to balance the reduced gem count from having fewer enemies
  • If gem_count * crystal_scaling < 250000 (which is roughly the minimum vanilla expects to get) then we redo generation, that way we don't get affected by the low lows.

Testplays (of latest changes):

Note that with these testplays, no crystal efficiency was picked up and no bosses were killed until the very end, so even vanilla cannot get 18 caches with this strategy hence it's not a consideration in this PR.

Expected total gems from enemies after multiplier (calculated at gen time):

Other notes:

• Testing has been limited to specifically 300 rooms, 750, and 1500 room counts (dubbed small, medium and large) to make it easier to balance correctly.
  • The min room count that does not crash is 15, but there's not enough horde rooms to ensure 18 chests. A safe min room count is 100.
• knife option is really OP. Should it even be an option? Should we hide it? Should it even exist on smaller dungeons?
• Should be backwards compatible
• Some stats were added to estimate how many gems can be picked up, which I used to estimate crystal calculations. These have been added to the stats screen but are not visible unless the flag is set to true in code.

Gif of finding agate knife at room 100, with small dungeon displayed in map: (Note gif has a bug where if you pick up the knife then quit then picking the knife again does nothing but this has since been fixed)

Changelog / Full test history

• A baseline vanilla test was done on 0.5.0, which is the main target to compare against
• First was a number of 100 room tests back when crystal scaling was done by room_count, which was giving too many crystals
• A full test (300, 750, 1500) was done after this was changed to enemy count, but showed great variance in crystal count. The biggest amplifier however was a bug where the multiplier affected crystals dropped from death, heavily inflating fewer riin count runs. Was subsequently fixed afterwards.
• Added average gem count by counting up the estimated gems dropped by each enemy spawned in the dungeon as actually playing through games takes a very long time
• Repeatedly genned dungeons and measured up the average gem count between the different sizes to try to match vanilla
  • 16 gens of small medium large and vanilla showed that currently medium and large were genning with too few crystals
  • Adjusted max_dist formula to max_dist - 2 to allow overshooting and tried again, now all sizes had too many crystals
  • Adjusted max_dist formula to max_dist * 0.95, and regenerated dungeon if calculated gems < 250000. Was a bit on the low side.
  • Did 50 more gens of vanilla to get an average overshoot distance, gem stats and enemy count
  • Adjusted max_dist formula to max_dist * 0.97, expected enemies to 10480. Was a bit too high
  • Adjusted expected enemies to 10300. Was a bit too high.
  • Adjusted expected enemies back to 10000. Seemed to be ok, however this was getting more crystals than the previous 95% test with 10000. Not sure why.
• Did another full test (300, 750, 1500) and looks much more ok

[Rnd-Guy]

In case it's helpful, I'll attach the apworld changes I've used to get this to work, feel free to change them as desired

__init__.py

@@ -57,6 +57,8 @@ class MeritousWorld(World):
        self.include_psi_keys = False
        self.item_cache_cost = 0
        self.death_link = False
+       self.rooms = 3000
+       self.agate_knife_percent = 100

    @staticmethod
    def _is_progression(name):

@@ -101,6 +103,8 @@ class MeritousWorld(World):
        self.include_psi_keys = self.options.include_psi_keys.value
        self.item_cache_cost = self.options.item_cache_cost.value
        self.death_link = self.options.death_link.value
+       self.rooms = self.options.rooms.value
+       self.agate_knife_percent = self.options.agate_knife_percent.value

    def create_regions(self):
        create_regions(self.multiworld, self.player)

@@ -177,5 +181,7 @@ class MeritousWorld(World):
        return {
            "goal": self.goal,
            "cost_scale": cost_scales[self.item_cache_cost],
-           "death_link": self.death_link
+           "death_link": self.death_link,
+           "rooms": self.rooms,
            "agate_knife_percent": self.agate_knife_percent,
        }

Options.py

from dataclasses import dataclass

import typing
-from Options import Option, DeathLink, Toggle, DefaultOnToggle, Choice, PerGameCommonOptions
+from Options import Option, DeathLink, Toggle, DefaultOnToggle, Choice, PerGameCommonOptions, Range, NamedRange

cost_scales = {

@@ -53,6 +53,37 @@ class ItemCacheCost(Choice):
    default = 0

+class Rooms(NamedRange):
+   """
+       Sets the total room count for the dungeon generated, which changes the length of the game.
+       Crystal gain will be multiplied to compensate for the reduced room count.
+
+       - **Original (default):** 3000 rooms, ie base game amount
+       - **Small:** 300 rooms
+       - **Medium:** 750 rooms
+       - **Large:** 1500 rooms
+   """
+   display_name = "Rooms"
+   range_start = 15
+   range_end = 3000
+   default = 3000
+   special_range_names = {
+       "original": 3000,
+       "small": 300,
+       "medium": 750,
+       "large": 1500,
+   }
+
+class AgateKnifePercent(Range):
+   """
+       How much % of rooms to discover to count as 100% room completion.
+       Once this % is reached, the agate knife will spawn
+   """
+   display_name = "Agate Knife %"
+   range_start = 0
+   range_end = 100
+   default = 100
+
@dataclass
class MeritousOptions(PerGameCommonOptions):
    goal: Goal

@@ -60,3 +91,5 @@ class MeritousOptions(PerGameCommonOptions):
    include_evolution_traps: IncludeEvolutionTraps
    item_cache_cost: ItemCacheCost
    death_link: DeathLink
+   rooms: Rooms
+   agate_knife_percent: AgateKnifePercent

[Rnd-Guy]

Found a bug with the branch, same as the bug shown in the gif of this PR:

• If a special item (eg seal or agate knife) is picked up that was gotten already (eg getting item then quitting before saving) in the slot, then the special item may fail to be given locally. This can be quite serious if the cursed seal is unable to be picked up a second time. This is likely due to the reordering of the setup before gameplay begins but not sure what it is yet. Tested once in normal branch and it didn't happen

[Rnd-Guy]

Can indeed confirm this bug is indeed related to the order of operations for game start, possibly because by waiting for the connection we trigger the on_connect handlers too early.

The specific difference happens when we pick up the special item. We correctly identify that we need to pick it up, but we fail the if condition shown below whereas in normal branch we correctly go inside and receive our cursed seal.

void InternalCollectAPItem(uint64_t location)
{
  if (apStores[location / 24]->HasItemStored(location % 24))
    ReceiveItem((t_itemTypes)(apStores[location / 24]->GetStoredItem(location % 24) % AP_OFFSET),
                storeNames[location / 24]);
  else {
    std::list<int64_t> check;
    check.push_back(location + AP_OFFSET);
    ap->LocationScouts(check);
    ap->LocationChecks(check);
  }
}

Through trial and error it looks like it's specifically the case for when the connection is successful before we call ReadStoreData(), which makes sense given it's store related. However a lot of the load game functions cannot be reordered because map data needs to be read in a specific order. That is, ReadStoreData() must be called after RandomGenerateMap(), but connection must happen before RandomGenerateMap() and yet we need connection handlers to run after ReadStoreData().

This bug is really punching above it's weight class here D:

[Rnd-Guy]

My theory is that ReadStoreData() was resetting the store state, which is the second time it gets reset during initialisation (the first is during InitStores() which is usually after generate map but is now done at the start).

Normally this is harmless, as it gets reset twice before anything is added, but I think because we now finish connecting and handle connection handlers between the first and second store resets, funny business occurs. The bug no longer happens when I remove the second reset (ie remove CreateAPStores() below)

void ReadAPState()
{
  CreateAPStores();

  int progress = 0;
  nextCheckToGet = FRInt();
  for (auto store: apStores) {
    store->SetCostFactor(FRInt());
    for (size_t x = 0; x < store->GetLength(); x++) {
      auto collected = FRChar();
      if (collected) store->MarkCollected(x);
      auto hasItem = (bool)FRChar();
      if (hasItem) store->StoreItem(x, FRInt64());
    }
    UpdateLoadingScreen((float)(++progress + 1) / (IS_MAX + 1));
  }

It makes sense because

• The reason I had to move InitStores to the start is because the on_connect handlers need to connect with the stores, so it definitely does something in that area
• ReadAPStores() is only called on game load, and the bug doesn't happen on a new game.

However the theory is broken because that should mean I can replicate it by starting a new game on a slot that has picked up a a special item, save game, quit and load game (to trigger the bug) and then try picking up the item again but it's working fine. Thankfully this is intended behaviour so it's not too bad of a thing if I can no longer replicate it but leaves uncertainty.

For now I think I'll just remove the call to CreateAPStores() and just monitor if this is an issue during further testplay.

[Rnd-Guy]

Just realised crystal counts shouldn't be scaled by room count but instead by monster count. Potentially can scale it by 10000 / monster count, as base game usually creates just over 10000 monsters, though I've seen sub 10k before.

For example, in the first gif above we have 100 rooms and 771 monsters. 30x less rooms gives me a crystal multiplier of 30 despite only having ~13x less monsters, off by more than a factor of two, which helps explain a bit of the surplus crystals that I'm noticing

[Rnd-Guy]

Did some more testing but unfortunately looks like there's an issue with crystal counts 🙁

Here's a chart of my crystals on a few runs with this PR, compared to a 3000 room count run I did on the sep 2024 big async. To ensure my comparison runs are consistent:

• I plando'd crystal efficiency onto alpha chest 18 (new to plando so I couldn't figure out how to put it to 24), to match how I did not get this on my big async run.
• Only defeated bosses as the last few rooms, thus 0 evolutions.

• Naturally the large 1500 room count is a major issue and that's enough to block this PR from being mergeable. Maybe rounding issue causing values to be halved? Though I had added code to handle partial crystals...
• The medium 750 room may be similar to the 1500 room issue.
• Small room count may be highly susceptible to variance (eg psi key room at edge of dungeon) but perhaps the horde rooms being a larger proportion of the overall dungeon is inflating their value to above normal? Probably needs more tests.

The three run sizes here (300, 750, 1500) were sorta arbitrary room counts that I chose to represent small, medium and large choices to be used as yaml options. Not much thought was particularly done to pick those numbers, but playtime wise these took about 40min, 1hr10 and 2 hours respectively to 100% which I think is really nice! It was suggested that possibly as a first approach, we stick to a limited set of room sizes to make the first implementation easier.

As for other testing results, I've not encountered any other major issues. I've confirmed the "not picking up agate/cursed seal" issue doesn't happen anymore after removing the CreateAPStores() call for example.

Once I figure out the crystal count issues, I think I'll be happy with how it is, but for now I'll put this to draft to reflect how it's not ready to be merged in this current state.

Edit: Found a bug where the crystals we drop on death (we lose 1/3, then 95% of that is turned into crystals we can pick back up) also get multiplied so we gain back more than we lost. Whoops! That might explain why 300 rooms gained so many more crystals.

[Rnd-Guy]

Made some changes to the crystal scaling algorithm to try to match vanilla a bit more, as well as a couple failsafes.

Summary The two main factors that currently govern how many crystals smaller dungeons get are:

• room proportions. Having more rooms considered "far" results in more harder enemy spawns which drop much more gems.
• expected vanilla enemy count, used to calculate crystal scaling (ie expected vanilla enemy count / total enemies)

Overshoot distance - aim to match room and thus enemy proportions Vanilla calculates stuff based on room distance from center, with a cap of 50. I've made the assumption that if a dungeon has a distance higher than 50, then it likely has more rooms considered "far away" enough to spawn harder enemies, which in turn affects gems dropped. Therefore by letting smaller dungeons also overshoot a little, I can mimic this behaviour to keep enemy proportions similar to vanilla.

I genned 50 vanilla dungeons and got an average max distance of 51.4, so I've multiplied the max distance of smaller dungeons by 50/51.4 (~97.2%) to mimic this behaviour, to keep enemy proportions roughly the same as what vanilla might have.

Min gem check - at least the minimum that vanilla can generate with I calculate the average number of gems dropped (for each monster add (min+max)/2) and then multiply that by the expected monster count. If this is below what vanilla can generate (~250000) then I redo the dungeon generation. This then prevents the smaller dungeons from being too volatile and not genning with enough crystals.

Since this skews the average, I've kept the expected monster count to be 10000, when the average is more like 10480 for vanilla, to keep the smaller dungeons from being too volatile and genning with too many crystals

• Average of 98,99,100,101,102 and 80,90,100,110,120 is 100, but removing entries less than 98 skews the right average to now be 110 for example

Before:

After:

After with testplays:

This is significantly better than the first attempt! The 750 and 1500 are still a tiny bit lower than I expected them to be but perhaps small sample size issue. I'm much more happier with this now.

Note these runs were done with the worst crystal strats (no efficiency, no bosses until the very end) and so even vanilla doesn't get 18 caches. As a result we only need to aim to be close enough to vanilla.

With this, I'm reopening this PR!

[Rnd-Guy]

I've made the changes according to your review but right now I can't actually compile it to see if it works D: I completely scuffed my c dev environment trying to figure out a good way to debug the game ^^;

I'll try to fix it this weekend

[FelicitusNeko]

If nothing else, it builds in Linux. I can try to finagle building for Windows, then give this a run in one or the other.

[Rnd-Guy]

Got my environment working I think, might still be scuffed though

However it looks like there's a bug when generating a dungeon that needs to regenerate due to my new crystal safety check. It looks like it's not cleaning up the enemies sufficiently and it's causing enemies to appear outside walls and cause horde rooms to be impossible, as well as glitching up the palette

To reproduce (I think), spam pressing new game until the loading bar noticeably resets (this means it's recreating the dungeon as it's not meeting some criteria which for small dungeons should only be my new min crystal check)

I've never seen this during all of my testing, so I dunno if this is due to my current scuffed environment or not but I'll do some more testing tomorrow

[Rnd-Guy]

Thankfully appears to be a simple fix, just needed to clean up enemies after each generate attempt

Successful 300 room run with latest version which reset gen a few times before starting, got 2 of the 17th caches which matches with my previous testing results when having 0 evo traps + no crystal efficiency