Starvation doesn't target pop that consume exactly 0 food

[squealingpiggies]

Though my many simulation runs, I have come to notice a lot of goblins continuing to live with 0.00 of their food needs being met month after month. After so code diving, I found two piece of code that create an interesting edge case in code that prevents some starving pops from being culled. It all starts in the starvation check: https://github.com/Calandiel/SongsOfGPL/blob/033b9607f33bb1073339db0d420dc487046faaf7/sote/game/society/pop-growth.lua#L32 Where a caveat of Lua seems to be evaluating pp.need_satisfaction[NEED.FOOD] or 0.5 to 0.5 for when need_satisfaction resolves to a 0 instead of a nil (is a nil simple a x0 pointer?). This leads to pops that have consumed exactly 0 food to not be checked when starving. This effectively compounds goblin growth as many of their pops never even attempt to buy food since their life span prevents building up large amounts of wealth, specially in foreign provinces. This leads them to indefinitely live as an underclass consuming any excess food and hindering province growth of non-goblins (#175).

The simplest solution would be to make sure that the need_satisfaction is always resolved to at least 0.001 so that the nil check in pop growth should only ignore starvation for the first (initializing) satisfy_needs call. Another, not necessarily mutually exclusive, option is the modify foragers to consume some of their before they sell it so at least some of their food check is capture.

The other quick fix with foragers attempting to cover their some of their food needs before selling only excess food to the market to buy other goods. In theory, this should also prevent anyone from having 0 food satisfaction as long as they spend some of their free time foraging but won't catch working pops that are too busy to forage.

Other approaches I have considered require a lot of rework with the production-consumption algorithm; specifically setting up a two pass through the pops where the first would collect(read) information and the second produce and consume (write) what is now in the stockpile based weighted so that at least all pops buy some food. I also know that this could be easily parallelized to increase simulation performance, if love threads are easy enough to use.

Perhaps along with any of these, it would be an opportune time to separate the life and basic needs from just making the pop attempt to buy 3x needs. Something along the line of relegating life needs to food and water, maybe cloths too (although I think a substitution system is need so that hides could satisfy this need) and have the basic goods only calculate on known goods (tech, if the pop know its possible to produce said good) or available in market (the pop is regularly in contact with said good).

[Calandiel]

@ineveraskedforthis can you chime in? I don't remember that part of the codebase

[Calandiel]

At a glance seems like an oversight and almost definitely not the intended behavior

[ineveraskedforthis]

I think the simplest solution would be a proper nil check.

[ineveraskedforthis]

Also needs should be eventually replaced with use cases, anyone wanting to implement more efficient market clearing should keep it in mind.

[squealingpiggies]

use cases

What would the goal be and how would you use it? I could potentially be interested in implementing as I have found the satisfy_need function lacking a bit in how it fails to prioritize life needs. It seems that goblin growth simply outpace starvation as they never make enough food to satisfy more than a few of the wealthiest pop and then everyone else is starving. I tried adding nibbling (taking part of the production and consuming immediately) to forage and when producing goods that satisfy life needs from buildings. Additionally, I added a check to satisfy_needs where it buys all food and then forages until 20% food satiation before attempting to satisfy any other need. It turns out these together still weren't enough to support goblin tribes do to their quarter foraging efficiency and most goblins were relegated to 3 to 14 percent after the first few ticks. I wonder if it is an priority issue where pop don't prioritize food or simply goblins not being able to support themselves by foraging alone.

Edit: After messing around with the buy_use function (and fixing it to work for needs, so meat counts as food) and making foragers, gatherers, and hunters consume needed food before selling, I was able to implement a decent prototype needs function that attempted to buy from the market and then cottage the remaining needs. This gave the pops a some of all their basic and life needs covered and kept everyone (but poorer goblins) from the starving threshold consistently. Needs a lot of work but it definitely is a step in the right direction to get the population simulation running.

[ineveraskedforthis]

What would the goal be and how would you use it?

Use cases are much more flexible than needs: needs are basically use cases where each good has weight 1.

[squealingpiggies]

So is the intention to completely remove needs and just have a list of use cases and demand values the pop tries to satisfy? Or should needs just be mapped to a use case and have the satisfy_need function be used to call the buy_use before attempting to work to buy and/or cottage the remaining? How much work should be done to fit use cases in ideally into the production and consumption tick?

[ineveraskedforthis]

It should work basically the same way. For example, instead of loop like

for _, good in pairs(need.goods) do
                local index = RAWS_MANAGER.trade_good_to_index[good]
                local c_index = index - 1

                local available = market_data[c_index].available
                local price = market_data[c_index].price

                ---@type number
                local demand = math.min(need_amount, buy_potential * market_data[c_index].feature / total_exp)
                local consumption = math.max(0, math.min(demand, available))

                expense = expense + record_consumption(index, consumption) * POP_BUY_PRICE_MULTIPLIER
                record_demand(index, demand)

                total_bought = total_bought + consumption

there should be a loop related to use cases (copied from buy_use, so it should be adjusted):

for good, weight in pairs(use.goods) do
            local c_index = RAWS_MANAGER.trade_good_to_index[good] - 1

            local consumed_amount = potential_amount / weight * market_data[c_index].feature / total_exp
            if consumed_amount > market_data[c_index].available then
                consumed_amount = market_data[c_index].available
            end
            local demanded_amount = demanded_use / weight * market_data[c_index].feature / total_exp

            -- we need to get back to use "units" so we multiplay consumed amount back by weight
            total_bought = total_bought + consumed_amount * weight

            spendings = spendings + record_consumption(c_index + 1, consumed_amount)
            record_demand(c_index + 1, demanded_amount)
        end

You could notice that all code related to needs basically mirrors code related to use cases but doesn't use weights.

[Calandiel]

I feel like theres some confusion over what goods, needs and use cases are.

A good is a single type of tradable item. An iron sword. A bronze cutlass. Etc.

A need is a thing on an agent interacting with the market that drives it to make purchases. Officers in an army have a need for weapons.

A use case is a set of goods that fulfill the same purpose, alongside a number defining how well they fulfill it. For example, a use case of weapons would include both an iron sword and a bronze cutlass.

Use cases are the youngest feature and as such not all code has taken them into account when it should to make the system both more flexible and less hardcoded.

[squealingpiggies]

loop related to use cases

what goods, needs and use cases are

These help a lot. The basic idea is simply modify the satisfy_need function used by satisfy_needs to call the buy_use function for each good in the need. This seems like the minimalist code implementation I had in mind but makes me wonder if NEED should have it's own list of goods or be sifted into the races. Shouldn't races just have a list of uses for each need they have and let the buy_use choose the best goods and drop the need.goods loop between the needs loop and buy_use use_case loop? This would save on a lot of ticks for each satisfy_needs call. The other process saving option is to simply have a base use_case for each need, which is almost already implemented in the need names except for mismatches between clothing-clothes and storage-containers, and lacking of a luxury good.

[ineveraskedforthis]

I think adding/using according use_case for every old need would be enough for now. It would be nice to eventually rebalance "food" needs into race-specific diets. Probably when we will get rid of "food" trade good and replace it with something more specific.

[squealingpiggies]

I managed to convert the NEED.goods list into NEED.use_cases weighted list in this branch. It also changes satisfy_need to call buy_use in a similarly weighted way. This alone didn't have much effect on the simulation. But I noticed something about satisfy_need, after weighting whether it was more efficient to cottage or forage and buy, it would only do one regardless of availability and then waste the rest of the time doing nothing. I changed this to now potentially do all three. If the pop thinks it can only buy so much from the market and has enough money or only needs a little more foraging income to buy all of it, it will use the remaining time to cottage the last few bits. To go with with time management, the satisfy_needs now parses out the remaining time to basic needs after using potential all money and time for life needs; finally if there is time remaining, the pop forages to help the next pop food purchase. The pop seem much better at staying alive in some cases almost every pop (except a fair amount of poor goblins) gets at least some food. Humans are between 40-60% life needs with about 20% average food, the richest goblins manage about 50% life needs but only 15% food, and the richest few elves manage to satisfy almost half of their basic needs. Unfortunately, this only last first the first decade or so before pop growth and deaths begin to drastically change the population.

• The first thing I noticed was that while unemployed and even children were doing nicely trying to survive, gatherers were failing to get any food (probably because of their low income) even though they were producing most of it. This was leading to them dying off and killing any potential at a stable food surplus
• The second was an interesting development with elves, turns out that they were so successful and long lived that they continuously made babies until they were mostly children. With the slow development of children, it wasn't long before most of the tribe could only consume the surplus food from the few working pop, dragging the rest of the tribe down not unlike goblin babies.

I did add a quick hack to let food and meat goods be siphoned off by building workers before selling to market. It seemed to help to keep them alive, grow food supply steadily once the more jobs become available and maintain a consistent enough food supply that even the bigger goblins seem to grow in one they got enough gatherers working. But other than that, every tweak seemed to lead to pops being successful enough to overrun themselves with children before falling into a death spiral or live in abject poverty to prevent reproduction from happening more than the death rate.

[ineveraskedforthis]

Good job. You could probably also tweak how needs satisfaction influences reproduction rate, so instead of 0, reproduction cuts off at 0.3 or some other magic number, or try out different curves.

[squealingpiggies]

I spend a while watching too many simulations go by and think I finally have a solid base for the pop AI on this branch.

There are a lot of changes but there are only a few main points:

• I actually implement use cases properly for needs, like meat and food for food (where meat can be used for food but not the other way around)
• needs are halved each month instead of zeroed, this with initialization in new pops prevents the need to ever check for a nil or worry about a zero. It also create a more stable simulation by reducing major oscillations in food production from forage efficiency calculations (providing a lower, more stable forager count, month-to-month)
• parents take care of their children by attempting to satisfy their needs (with time and money) before their own and then donating some money instead of having the child siphon wealth from them
• pop can only reproduce when over 20% food, this creates a buffer zone between it and 10% where pop doesn't starve but doesn't reproduce until enough food become available for another pop
• I disabled buildings, warriors, court, and war to make sure the simulation works just as hunter-gatherers rummaging around, I am not sure how it will react to those being enables yet

Overall, I think I got a decent simulation that has everyone, more or less, growing at the start. There is a major performance cost to it as I can no longer tell the difference between 8, 9 and 10 speed.

Unfortunately, I am still having issues getting golbins to work right but I beleive I am getting somewhere. I find the real problem is that goblins just start with too many people and always start declining. Sometimes this even ends the tribe since no pop gets enough food to reproduce until everyone is outside breeding age or unlucky starvation rolls leave only one gender left. This can also happen to other races but it's funny to see a 0 pop realm with a single elf ruler who is only 300 years old. The other issue I am concerned about is price determination as I was sometimes seeing the price of food (the only good available in my runs) continue to rise despite the stockpile increasing. Some runs I would see a couple goblin markets with 60+ food available at a cost of over 2000 and no pop had more than a few units of money. I'm also definitely sure there are at least 2 edge case that are giving NaNs from the buy_use call in satisfy_need, very rarely from induced-demand and/or price_expectation that I haven't solved yet.

Regardless, I did get a run that finally passed the 100 year mark as I continued to put error guards in (and fiddled with magic numbers) trying to find the edge cases and took the following screenshots:

High end of human pops: Low end of human pops: Low end of beaver pops: High end of dwarf pops:

I'm at the point where I could start enabling and balancing buildings and hiring warriors but I wonder if this is not the opportune time to address #118 and get characters needs to work with the pop needs rework at the hunter-gathering stage first. Before that, I need to look at the tribe generation and ensure that each tribe has a viable pop distribution of age and sexes once I rebase it.

[Calandiel]

I will take a look at it in more detail today after work. I think first thing we absolutely need to merge in is making starvation treat 0s properly

[squealingpiggies]

I am closing this since it has been fixed for a while now.