feat: RBGen - Region Based Generator for industries and probably even cities

[winex]

i'm currently working on "Region Based Generator" (aka rbgen) similar what rso-mod has (Resource Spawner Overhaul)

WIP branch: winex:rbgen (might be force-pushed frequently) -- edit: WIP-12 is new

current progress:

• [ ] WIP again: split all the work to (almost) isolated commits
• [x] prepare grounds and fixes to be merged from winex:pr-2
• [x] defer tag placement winex@adb9184
• [x] call RBGen from on_chunk_generated() and do less in very frequently called on_chunk_charted()
• [x] segmentation of PRNG values to check against chunk positions still needs some statistical analysis (not good at it), but should work a bit better
• [x] Primary Industries: choose Region size (in chunks) -- 8 (min: 4), 256x256 tiles
• [x] Primary Industries: choose probabilities -- farms: 0.5, fisheries: 4.0
• [x] Primary Industries: switch to use RBGen
• [x] Cities: switch placement to use RBGen (in explore mode, future milestone?) -- works nicely!
• [x] Cities: limit placement by RBGen in total (current: 6)
• [ ] Cities: choose Region size (in chunks) -- 32 (min: 32), 1024x1024 tiles. looks fine, see below
• [ ] Cities: choose probability -- 0.5 looks good
• [ ] repeatability of generation independent of explore direction even after save/load probably not going to happen (see in first comment)
• [x] WARN players with increased difficulty message on specific map_gen_settings combinations: water 0%/disabled = unplayable, sand deserts bias, etc... (see reports in comments below)
• [x] NEW: support multiple surfaces in some way -- name blacklisting by few rules

mod support:

• [ ] TODO: #309 -- need to add some similar tiles
• [x] NEW: mod support: space-exploration, blacklist non-solid SE surface types -- preliminary support: planet, moon

almost done, WIP, ?:

• [ ] Cities: in discussion of overall city limit increase
• [ ] Fishery: tweak a bit more - still too many mishits increasing radius may cause some lags and looks too off-shore (use float and switch to BBox from chunks?)
• [ ] prove that adding more types of industries works ;)

testing

• [x] DONE: test some variations of params (see reports in comments below)
• [x] DONE: test different map_gen_settings (see reports in comments below)
• [x] DONE: test natural exploring (non-chart()) -- looks even better!
• [x] DONE: test singleplayer -- found and hit again some non-related bugs, lol
• [ ] test singleplayer after fixing non-related bugs
• [ ] test multiplayer after fixing desyncs
• [ ] test again
• [ ] test again again
• [ ] run FFA (public) Tycoon multiplayer server with minimal set of mods (>3 cities and 4K or 2K map limit)
• [ ] test adding more types of industries (even better: insane amounts)

gather feedback

• [x] DONE: first WIP testing streamed live (screenshots: #271), thanks @bahrmichael
• [ ] different params(not for players, though)
• [ ] different map_gen_settings (might missed some combinations)
• [ ] different play styles

description

it's actually very simple and look really universal, uses simple configuration and supports any (well, if fits into region) amount of entries that doesn't depend on each other's numbers

it even supports filters in a single table. for each rbgen there should be region size, then a table of entity probabilities, tile names, etc.

@bahrmichael pls, label: enchancement, assign: winex

early alpha screenshot, looks very nice to me (notice close industries, but overall randomness):

[winex]

reserved for code description

configuration

this is how i banned wheat-farms from spawning in sands and deserts, for ex:

RBGEN_CHUNKS=256/32
PARAMS_PRIMARY_INDUSTRIES = {
    ["tycoon-apple-farm"] = { prob = 1.0,
    },
    ["tycoon-wheat-farm"] = { prob = 1.0, min = 0.75, max = 1.00,
        tiles = {
            "grass-1", "grass-2", "grass-3", "grass-4",
            "dirt-1", "dirt-2", "dirt-3", "dirt-4", "dirt-5", "dirt-6", "dirt-7",
        },
    },
    ["tycoon-fishery"] = { prob = 1.0, min = 0.25, max = 0.90, radius = 1,
        tiles = { "water", "deep-water" },
    },
}

-- rbgen itself is called like this:
local function on_chunk_generated(event)
    for _, name in pairs(Constants.PRIMARY_INDUSTRIES) do
        region_based_generator(event, PARAMS_PRIMARY_INDUSTRIES[name], name)
    end
end

adding new industries like potato and rice farms (wanna them :)) is just adding an entry into that table

tags

tag placement is deferred for calls from on_chunk_charted() (as in current release), by introducing global.tycoon_tags_queue, which could be used for tagging cities as well or any other future type

repeatability (low priority)

repeatability of generation independent of explore direction even after save/load probably not going to happen as it should keep random seed for every region or seed it with coordinates XOR'd w/ something

[bahrmichael]

I love that you started on this! Let me know if there's anything I can help with, or where some of my code/intentions may be unclear

[bahrmichael]

switch Cities placement to use RBGen (in explore mode, future milestone?)

I'll try to get some more clarity on the path forward with cities. There has been some feedback on custom placement on cities, as well as multi-surface support. Will share more as ideas get clearer, and am happy to receive more input :)

[winex]

4Kx4K map (with town generation enabled!) by rbgen:

• industries: 256x256 (8 chunks) regions. apple=72 (prob=0.5), wheat=76 (0.5), fishery=40 (4.0, omg)
• towns: 1024x1024 (32 chunks) regions. (prob=0.5, radius=2)
• seed: 1633975416, water = 1.0 (normal)

as game.forces.player.chart() go spiral, placement seems too close when just checking random() < probability. checking if it hits into range of adjacent chunk positions inside this region should work better like: 0 <= rnd < 0.5 <= rnd < 1, will see... otherwise, i'll just go for repeatability using region coordinates

as one may noticed - fishery probability is set to 4.0, which is kinda big - my tests shows that it just can't generate enough of them, because of very strict conditions (radius=1 makes 3x3 chunks search). this tweak helps

will try to compare this against rbgen-disabled map

[winex]

here it is! RED - old (A:54 W:54 F:56), WHITE - rbgen (A:72 W:76 F:40, same as above):

[bahrmichael]

This looks great! I can't wait to test it soon :)

Did you notice any positive/negative impact on map startup times or lag when discovering new chunks?

[winex]

This looks great! I can't wait to test it soon :)

me too, better in multiplayer :) it took more time, than i thought - lots of little creepy things! also, i've ran into some issues, which bothers me like hell - figured out the numbers and parameters for rbgen (to be alike the old gen), but the result just is not fine, yet (see https://github.com/bahrmichael/factorio-tycoon/issues/269#issuecomment-1976898462 )! still working on that...

Did you notice any positive/negative impact on map startup times or lag when discovering new chunks?

oooh, that's a hard question, actually! but as i always think about speed (50% optimization inplace) when writing anything, i think rbgen is even faster, here is why:

• on_chunk_charted() was lagging - i put warning in the code - biters trigger that VERY frequently! and it was always/on each call searching for nearby entities of the same type trying to place another one(!) while it wasn't needed
• while testing old gen with "easy params" - i got lots of industries spawn nearby just by looking at the biters for a couple of minutes, not even exploring! that looked weird -- so i dug into that and we're here now! ;)
• rbgen: doesn't do anything after chunk has been generated already
• rbgen: on_chunk_charted() just places pending tags if there are any, nothing more!
• rbgen: there are still tiles counting/searches, but it's done in actual generation step and only once per each chunk!

[winex]

@bahrmichael okay, if you can't wait anymore to play with it - i've just pushed WIP version winex:rbgen with lots of logging to track placement and errors, it's based off winex:pr-2 (#271)

• it's just for testing right now, but you can tweak params and check how it works (+cities there, too)
• disabling anything is just removing that entry from Constants table
• tried to doc the params -- don't know if that works with your IDE (i use Kate - it just highlights some)
• as was mentioned before - rbgen is very simple (except for number crunching there) and just called for every entry from param tables
• it should work for any amount of entries, even if it's more than region_size^2 (but not recommended) - maybe try to place chests/inserters to prove it, haha? :)
• added towns/cities as proof-of-concept with larger region size. any other type might be added just the same way
• water coverage is also honored almost as before - that setting selects between [water_lo_mult; water_hi_mult] and prob is multiplied then.

water coverage needs some more testing, but my regens (from the same point) on 2Kx2K map did good for A, W, F respectively:

• 17%: 19 29 9
• 100%: 13 23 3
• 600%: 20 20 12

this corresponds to mult params. maybe we need to lower it to 1.5 ~sqrt(2) for the numbers to be close to what 100% shows (except for fisheries - they're already struggling hard with prob=4.0)

btw, players with sand or desert maps could become sad, so it's probably going only into v0.5.0: "biomes" in Factorio were almost never used at all - i always thought map could be more useful (hate cliffs, though!)

looking forward to discuss your findings or even watch a stream (yeah, i've found your last one - it was cool :) )... have fun!

[winex]

just for history, already fixed in https://github.com/winex/factorio-tycoon/commit/1e065bcfafe7a809d75f09e6a87ca3332a2f846c (make sure to use latest winex:rbgen)

tile filtering, 2Kx2K map

were testing moisture and sand-desert biases... found that players would definitely rage at me for: by adding dirt tiles (finally found them :)) as well as red-desert for apples (it's like clay, so i'ts fine) - map shouldn't be so empty as above:

• red-desert is not for wheat though (future changelog: this should be stated)
• if there is no grass, there should be dirt tiles somewhere!

it should be fine as wheat farms aren't starting tech (currently) - it just makes game more interesting, though i see future mod conflicts here, ex: alien-biomes... i'd be happy to get some feedback for mod compatibility requests

+ water and 2 bias map settings (by vanilla)

we should inform players with a message of higher difficulty overall as it might be harder to find industries if map_gen_settings has:

• Water Coverage set to 0% => disabled
• Moisture Bias set to -0.5 => desert
• Terrain type Bias set to -0.5 => sand
• even worse: any combination of these at the same time, ex: 4Kx4K map - A: 4 W: 6 F: 0 - about 10 times lower than expected! 8Kx8K map - A: 37 W: 27 F: 0 - huge 49MiB empty save! - Cities: 10 (names wrapped, lol)

[winex]

feedback no. 1, let's go...

ok, i've got some great feedback (but it's misposted in #271), those are better to be discussed here:

notes

• to see overall picture you need to run pseudo-rand() exactly its bits-times (2^31 how many we got there?)
• it is designed to have every possible value and only ONCE (almost true, even distribution - it's LCG, if i'm not wrong) and that it should not repeat. should never output: 0, 0, 0
• while real random generator (but we don't have such!) does not have this property as it should have exactly the opposite one - numbers, distance between them and distribution should be truly random. could output: 0, 0, 0 of very low probability, but it is >0
• filters in params table smash PRNG even more... ex: fisheries with insane 4.0 probability

let's track these:

1. Fisheries very close to each other, and on an island. Maybe not ideal as it's hard to access. I'm fine with rare cases of similar industries nearby each other, but in general it should have a good distribution.
   • [ ] TODO: tweak last-step placement checks. will revise this (not my code, though)
   • generator itself doesn't know whether it is on island now or not - it's just too expensive to search for this
   • it's PRNG - i'm trying to use all the bits of it, but there's almost nothing left we can do more about it

this is how rbgen works (almost the same as oldgen): 8x8 chunks = 64. chunk was generated, let it be number 42 in that region, check if random(0, 64) is inside [42; 42+prob) range, check filtering with radius and then try to place. placement itself (it's same 0.4.5 code) has some checks (last-step) and we should tweak that more for cliffs and other stuff ! i haven't pushed the latest WIP version, yet !

btw, i personally see such batches of industries as really nice feature (different + same, too). never aimed for that and even couldn't - there is no code for this! ex: resource patches in Factorio are soooo evenly distributed -- they just used perlin/simplex noise of few levels (maybe only 1) and that looks very bad to me after almost 10 years of development... that's why we like rso-mod, not to mention that it has same problem (for bigger maps), though :P

2. City placed on water; may prevent the city's early expansion if nearby water or cliffs or resource patches.

   • [ ] TODO: didn't touch that, yet. callback needs position argument or some BoundingBox at least just used CityPlanner:: addCity() function as a proof-of-concept - it has no checks at all, lol :)

3. This feels like there are a lot of factories close to each other in batches. Noticed this, but not sure if I'm for or against it. Just a different spawning behavior that we should think about.

   • [x] see no.1 PRNG. in overall, you get a good distribution (see my large screenshots), but sometimes you'll be just out of luck
   • ex: one lake might get no fisheries at all, but the one next to the right of the same size -- could get 3 fisheries around it
   • i've been choosing params for rbgen to be statistically the same (for large maps, ofc) this takes so much time, omg! it sometimes 2x than before by water_*_mult, though -- should change that to about 1.5
   • you need to generate at least 1024x1024 (or 2048x2048) map to see much better overall picture

4. Industry spawned very close to resource patch, which made me wonder if they could now spawn ON resource patches. I don't like the latter, but spawning close to patches is okay.

   • [x] TODO: tweak last-step placement: resources

5. Industry spawned next to cliff. That blocks access to the industry, until the player has cliff explosives.

   • [x] TODO: tweak last-step placement: cliffs

few ideas (for myself mostly)

• think about checker-board ((y%2 + x)%2, iirc) segmentation instead of 2D (y*size + x), but i don't think it will change anything
• even repeatability by encoding chunk position as hash for each industry type using map.seed (reimplementing the wheel rand()) MAY NOT help the overall picture of batches -- it is still just pseudo-random numbers!
• think about overlaying rand() over bigger areas somehow, may only be possible with repeatability

sht, this is looong

[winex]

latest test with PRNG segmentation... as i thought before - it doesn't do much better :(, but a bit - yes natural exploring made this feel more interesting, though (but it takes more time)!

same as above https://github.com/bahrmichael/factorio-tycoon/issues/269#issuecomment-1979307705 4Kx4K map, A:101 W:69 F:40 T:6 starting area minimal, moisture/red-desert bias +0.5, +0.5

• PRNG sometimes hit too many times in a row, but empty parts seem to balance this. it feels more natural as no repetitive pattern is seen here, just bunch of: circles, lines, clusters, etc... even single ones are present in some places - this is good!
• as wheat is banned on sand (see left top corner), it's ~0.7x that of apples on this part of map
• proof-of-concept town generation: towns look too close at the bottom, but they are inside their respective regions (of 4x4=16 town-regions here), limit is 1 town per region and 6 total
• proof-of-concept town generation: works not that bad for only a few lines of code added :)

conclusion

• i suppose this algo is about 95% complete in its internals, 5% goes to externals/last-step checks
• only real playtests will show what PARAMS should be tweaked, but it should feel almost the same as before
• maybe even productivity bonus curves should be lowered instead because of new industry generation possibilities

future work, optional

• future tests may include increasing region size together with increasing probability (for the same overall count), but whether it can result in less super-close industries spam because of more random-bits used - is debatable!
• drop current PRNG, write something better - debatable even more
• drop current PRNG, go for exact repeatability by multiplying chunk positions with seed to overflow fast (LCG again, lol) Lua 5.2 has no integers, so it would be double-precision FPU stress - way worse than C-rand()

without real cases feedback this is going to be pure research :P

[bahrmichael]

generator itself doesn't know whether it is on island now or not - it's just too expensive to search for this

Right, I see that now, too. Let's drop that part of feedback. It was just a thought when looking at the map, but I'm happy to not add that restriction :)

just used CityPlanner:: addCity() function as a proof-of-concept - it has no checks at all, lol :)

Gotcha! I wasn't sure how much you already changed. Will look into this myself!

see no.1 PRNG. in overall, you get a good distribution (see my large screenshots), but sometimes you'll be just out of luck

Sounds good to me! Thanks for reviewing my feedback.

it feels more natural as no repetitive pattern is seen here, just bunch of: circles, lines, clusters, etc... even single ones are present in some places - this is good!

I agree!

i suppose this algo is about 95% complete in its internals, 5% goes to externals/last-step checks

Awesome!

maybe even productivity bonus curves should be lowered instead because of new industry generation possibilities

We can do that. If you want you can suggest a rebalance after/with the PR for this change.

---

Let me know when I should give it another try or when you want to schedule a multiplayer session :)

[winex]

@bahrmichael you can try any time! :) i've pushed latest "WIP-9" version (see at the top) few days ago.

• towns are still placed weirdly on water sometimes, even enlarging town-center-virtual to 14x14 didn't help much

i must review all city placement functions and sync them to be alike industry placement

• resources check can fail in rare cases - when generating map really fast, an industry might be placed just before resources are

possible fix would be to add some delay, but ideally it should be async. otherwise, we need to ensure somehow that our on_chunk_generated() is called only after resources are actually placed, but idk if that is possible.

[bahrmichael]

resources check can fail in rare cases - when generating map really fast, an industry might be placed just before resources are

Rare cases are fine, don't worry about hardening this too much.

[bahrmichael]

Some more feedback :) Love the work so far! I see that you have disabled town generation, so I didn't look at that, but just at the primary industries.

This is the map I got with some running around, default settings:

I noticed that tags show up outside of the range that I should have discovered. Maybe that's related to tag placement, which we may want to defer to the on_chunk_charted handler?

I also noticed that the coloured backgrounds aren't always matching the selection area. It would be nice to have them matching, or at least be consistent around the icons.

[bahrmichael]

Wanted to share some explanation on how I (try to) prevent cities on water:

When placing a city, we generate a bunch of locations, and check the tiles in the area.

https://github.com/bahrmichael/factorio-tycoon/blob/ad46472f0f390412920fa7edb505182f441824b3/city-planner.lua#L86-L111

Water (and other less desirable tiles) are counted and with the power of 2 added to the weight of all possible locations.

https://github.com/bahrmichael/factorio-tycoon/blob/ad46472f0f390412920fa7edb505182f441824b3/city-planner.lua#L134

We then finally sort the options by weight, and pick the best result.

This can lead to towns on water, but it's unlikely. If you didn't touch this code, then don't worry about rare towns on water. Sorry for not pointing that out earlier!

[winex]

Some more feedback :) Love the work so far! I see that you have disabled town generation, so I didn't look at that, but just at the primary industries.

?! no, i didn't disable town generation. should work still, it's just rbgen_stats_... don't support multi-surface/surface_index, yet can you point what else you've spotted?

I noticed that tags show up outside of the range that I should have discovered. Maybe that's related to tag placement, which we may want to defer to the on_chunk_charted handler?

feature, same as in rso-mod by forcing game.forces.player.chart() (why it can't show code here?!): https://github.com/winex/factorio-tycoon/blob/18d1ecd337bd25d3e7d5fbcf77dc7acac4166053/chunk-charted-handler.lua#L206-L208 i think it's nice that it pops out while you're exploring - you'll never miss what you're trying to find

I also noticed that the coloured backgrounds aren't always matching the selection area. It would be nice to have them matching, or at least be consistent around the icons.

feature - apples are red, wheat is yellowish, town halls are green (saturated) -- it helps to distinguish them in the map view zoomed out and when it's uncharted. it looks a bit strange with icons, but it's because entities are large ~14x14. we can color treasury entity so it's easier to find it on map, too :)

Wanted to share some explanation on how I (try to) prevent cities on water:

thanks for pointing this, i'll try to use this function with limited radius (to be inside region) somehow... it's just rbgen works the other way around - it checks each chunk if conditions are met to be filled w/ something - similar to autoplace thing. towns are rare, though, so we can do more stuff

[bahrmichael]

?! no, i didn't disable town generation. should work still, it's just rbgenstats... don't support multi-surface/surface_index, > yet can you point what else you've spotted?

This is what I meant: https://github.com/winex/factorio-tycoon/blob/rbgen/city-planner.lua#L491-L493

I commented this out and everything worked fine.

[winex]

This is what I meant: https://github.com/winex/factorio-tycoon/blob/rbgen/city-planner.lua#L491-L493

I commented this out and everything worked fine.

this is build_initial_city() as per Spawn Initial City option. town generation by rbgen should work fine

[winex]

multiple surfaces was not intended to be supported by my (should-be-) fast Util::chunkToHash() function, so moving from number to string type was required. it is slower, but until FactorioLua provides integers or at least >32-bit bitwise operators (bit52, lol), this will work in some way...

global.tycoon_rbgen_stats on 2Kx2K map now looks like this, cool isn't it?:

4119.323 Script log(serpent.block(global.tycoon_rbgen_stats)):1: {
  true,
  ["tycoon-apple-farm"] = {
    ["0001fffd0002"] = 1,
    ["0001fffefffd"] = 1,
    ["0001fffefffe"] = 1,
    ["0001ffff0000"] = 1,
    ["0001ffff0003"] = 2,
    ["0001ffff0004"] = 1,
    ["0001ffffffff"] = 1,
    ["00010000fffc"] = 2,
    ["00010001fffc"] = 1,
    ["00010002ffff"] = 2,
    ["00010003fffd"] = 2,
    ["00010003ffff"] = 1,
    ["000100000003"] = 1,
    ["000100010000"] = 1,
    ["000100010005"] = 1,
    ["000100020000"] = 1,
    ["000100020001"] = 2,
    size = 8
  },
  ["tycoon-fishery"] = {
    ["0001fffcfffc"] = 1,
    ["0001ffff0002"] = 2,
    ["0001ffff0006"] = 1,
    ["00010001fffc"] = 1,
    ["00010002fffe"] = 1,
    ["000100000001"] = 1,
    ["000100010004"] = 1,
    ["000100030003"] = 1,
    size = 8
  },
  ["tycoon-town-hall"] = {
    ["0001ffff0000"] = 1,
    ["0001ffffffff"] = 1,
    ["000100000001"] = 1,
    size = 32
  },
  ["tycoon-wheat-farm"] = {
    ["0001fffc0002"] = 2,
    ["0001fffcfffc"] = 1,
    ["0001fffcfffd"] = 1,
    ["0001fffd0000"] = 1,
    ["0001fffd0001"] = 1,
    ["0001ffff0002"] = 1,
    ["0001fffffffc"] = 1,
    ["00010000fffc"] = 1,
    ["00010000fffe"] = 1,
    ["00010000ffff"] = 1,
    ["00010001fffd"] = 1,
    ["00010001fffe"] = 1,
    ["00010001ffff"] = 1,
    ["00010003fffc"] = 1,
    ["00010003fffd"] = 1,
    ["00010003ffff"] = 1,
    ["000100000005"] = 1,
    ["000100010006"] = 2,
    ["000100020001"] = 1,
    ["000100020003"] = 1,
    ["000100020004"] = 1,
    size = 8
  }
}

iirc, this is count of named entities per region first is global.tycoon_rbgen_stats[surface.index] = true to allow rbgen on this surface -- idk if it bugs any for ... in pairs() loops, yet. maybe should move it deeper

region size is in chunks, i thought we should remember it for future possibility of changing it, then a RBGen.rescan() could be easily called just by comparing values

this stats table is also used to limit anything internally... 48-bit fit into 12 chars means:

• 65535 surfaces
• -32768..32767 ^ 2 chunks == 4M chunks, 4096M tiles area (when size=1) if someone is able to bug this limit (teleport not allowed!) -- blame @winex, lol

i think we should use this for surface-modified TagsQueue also, but it's not necessary

[bahrmichael]

@winex Just wanted to check in with you about your plans for the rbgen. I'm thinking about releasing 0.5 within 2-3 weeks, shall we include rbgen there?