smooth out estimated time to depletion

[wchristian]

These three changes mainly affect how ore per minute is calculated, primarily to jump over updates where no ore was consumed, and to smooth out spikes (both up or down) in ore consumption.

Might possible be done more elegantly, but i don't know Lua that well.

[narc0tiq]

I'm somewhat concerned about ignoring updates where no ore was consumed; there can be any number of reasons why this is the case, and some of them are ones the player would want to know about -- e.g., if a mine is idling because a biter ate a power pole or because a train ran out of fuel. Plus, it looks strange to claim that a site is going to run out in 30 minutes when it's actually been idling for the last two hours and will likely keep idling for the foreseeable future.

It also seems to me like the smoothing is based on that same skipping of updates where the delta amount is zero, so there wouldn't be a material change if I accepted the one, but not the other -- however, I admit I probably haven't read the code closely enough, so maybe I missed something. Please let me know if I did.

---

My impression (as a not-very-mathy person) is that, for the best smoothing, we should we storing multiple (10ish?) older records (amount and game tick) per site, and calculate our rate of change using that full history:

• either as a straight delta between the oldest and newest records,
• or as an averaging of the rate-of-change between each pair of records.

I'm not actually sure which would be best, but in either case, I wouldn't really want to ignore idle times, because that does actually impact the rate of depletion.

I'm also not sure about the number of records to save; theoretically, the longer the recording window, the more accurate the estimate, as the rate of change should trend toward zero without reaching it until we're "pretty sure" the site is going to idle for a long time (read: has already idled a long time). Maybe, if the newest record is the same amount as what we're about to add, we should just update its timestamp? And then we can kick off the oldest record only if it's more than 30 minutes old?

I tried an implementation like this (if not as refined) in my prodmon mod which I never finalized; I definitely remember having had trouble with zero values and zero deltas, but that code is not very suitable for inspiration because it's also trying to solve a systematic sampling error that's somewhat related. Nonetheless, I'm sure we can come up with something better than a simple (old_value - new_value) / (new_tick - old_tick), which is the current implementation in YARM.

---

In any case, I'd appreciate your thoughts on this proposed alternative -- whether positive or negative -- or any other means we could use to reach the same goal.

[narc0tiq]

Also, I'm sorry I somehow forgot this earlier, but thank you for the contribution! Whether I take it or not, I really appreciate that you took the time to help!

[wchristian]

Yeah, no worries, i just shared it because the implementation is useful for myself. Also, thanks for the work on the mod, because it's super useful for playing with pymods. :D

As for the skipping, i believe your understanding doesn't match what it actually does, which would however be understandable, as the matter is a little tricky.

It doesn't actually ignore the update and keep the same value going forward. If something changed, it uses that immediately to calculate the new ore-per-minute AND stores the new current ore value and the timestamp of that value to be used next iteration. If no ore was consumed it discards the values and instead uses the last stored ones to calculate ore-per-minute between that earlier timestamp and now. Thus, it won't keep showing 30 minutes remaining, but on each iteration will display a longer and longer ETD which roughly works out.

So maybe skip is the wrong word and it's more of a "stretch the last interval with change out".

The reason i need that personally is that i'm playing with a mod that has a LOT of ores and some of them are used sparingly, so it'll sometimes go 15 minutes without drawing any ore from that patch, then a blip of activity, etc. The ore-per-minute going 0, 0, 0, 0, 200, 0, 0, 0, 0 is fairly useless there.

As for "site was taken offline", that'll still show up, as the ore-per-minute will keep drop and approach zero over time, just not immediately.

As for stretching + smoothing or just smoothing: I haven't done simulations, but i believe just the smoothing would make the values on very low flow deposits spike up then drop to zero over time, particularly if the cycle on low flow deposits is longer than the timeframe the smoothing covers, while the stretching will always provide some amount of useful data.

As for storing multiple values: I've done things like that more often in the past, and simply put: It's not needed. I understand your thought process since i've been there, but really, you don't. It can be useful for more elaborate schemes, but in this particular case simple easing is perfectly fine.

Using the algorithm of:

display_value += (smoothing_factor * (new_value - display_value))

You can adjust how much history you get by decreasing or increasing the smoothing factor.

If you'd like to try things out more viscerally, maybe try and see if you can generate some series via excel and graph them.

I could be more verbose, but at this time i'll just let you read for now and see how you feel. :)

[narc0tiq]

Sounds awesome, actually, thanks for the description! I'm happy to let your expertise take over, then.

For now, I'm going to test this more thoroughly in my dev build and probably merge it into the next release; depending on available time, could be in just a few hours or up to 24. I'll ping back here when it's ready.

Thank you very much for your help!

On Mon, May 13, 2019, 18:04 Christian Walde (Mithaldu) < notifications@github.com> wrote:

Yeah, no worries, i just shared it because the implementation is useful for myself. Also, thanks for the work on the mod, because it's super useful for playing with pymods. :D

As for the skipping, i believe your understanding doesn't match what it actually does, which is however understandable, as the matter is a little tricky.

It doesn't actually ignore the update and keep the same value going forward. If something changed, it uses that immediately to calculate the new ore-per-minute AND stores the new current ore value and the timestamp of that value to be used next iteration. If no ore was consumed it discards the values and instead uses the last stored ones to calculate ore-per-minute between that earlier timestamp and now. Thus, it won't keep showing 30 minutes remaining, but on each iteration will display a longer and longer ETD which roughly works out.

So maybe skip is the wrong word and it's more of a "stretch the last interval with change out".

The reason i need that personally is that i'm playing with a mod that has a LOT of ores and some of them are used sparingly, so it'll sometimes go 15 minutes without drawing any ore from that patch, then a blip of activity, etc. The ore-per-minute going 0, 0, 0, 0, 200, 0, 0, 0, 0 is fairly useless there.

As for "site was taken offline", that'll still show up, as the ore-per-minute will keep drop and approach zero over time, just not immediately.

As for stretching + smoothing or just smoothing: I haven't done simulations, but i believe just the smoothing would make the values on very low flow deposits spike up then drop to zero over time, particularly if the cycle on low flow deposits is longer than the timeframe the smoothing covers, while the stretching will always provide some amount of useful data.

As for storing multiple values: I've done things like that more often in the past, and simply put: It's not needed. I understand your thought process since i've been there, but really, you don't. It can be useful for more elaborate schemes, but in this particular case simple easing is perfectly fine.

Using the algorithm of:

display_value += (smoothing_factor * (new_value - display_value))

You can adjust how much history you get by decreasing or increasing the smoothing factor.

If you'd like to try things out more viscerally, maybe try and see if you can generate some series via excel and graph them.

I could be more verbose, but at this time i'll just let you read for now and see how you feel. :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/narc0tiq/YARM/pull/113?email_source=notifications&email_token=AAEQNVOEVA7ZVSQGW7WFJOTPVF7ONA5CNFSM4HMLE6V2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVITD3I#issuecomment-491860461, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEQNVKMLXQLATGZREKBPB3PVF7ONANCNFSM4HMLE6VQ .

[wchristian]

Cheers. Looking forward to the results of that, and no hurry needed. :)

I would like to note though that my middling experience is mainly with statistics, less with Lua.

[narc0tiq]

It's looking great in my testing:

I did cut down the display precision to just one decimal (I don't think anyone will miss it), and I managed to get that column right-aligned, so I think it looks pretty good now. I'm going to go ahead and tag it as v0.8.16 and push it out.

[wchristian]

Very nice work on that. Looks excellent, thanks. :)

Also, fwiw, i think it might be good to right-align all of the number-containing columns if that's as easy to do as that.

[narc0tiq]

I'm tempted, but at the same time, it's kind of a fragile setup -- if the column ever gets wider than the minimal_width I set, they'll get misaligned again. I just gave enough space that you'd have to get a ridiculous mining rate before that happens.

What I really wanted was for the label to auto-stretch to take up the full column, but for some reason that makes it not render anymore; it's probably a bug and I'll have to make a bug report when I have some time. Once that gets fixed, I'll make the change in the other relevant columns.

[wchristian]

That's reasonable. Thanks for explaining. :)

[narc0tiq]

And actually, thanks to @Choumiko, it turns out there's a much easier way to do it: edit the table's https://lua-api.factorio.com/latest/LuaStyle.html#LuaStyle.column_alignments. Example (yellow highlight):

So I guess I'll be fixing that for the next release. 😂

[wchristian]

Turns out manuals can be helpful. Looking forward to that. :D