Duration being under reported?

[NeilJed]

I noticed in utils.js the method of extracting the duration seems to pull the Duration value and not the Billed Duration value:

https://github.com/alexcasalboni/aws-lambda-power-tuning/blob/2baf19fc14f0bcbef2903abc3e17a0f7f1130a14/lambda/utils.js#L514

I think it's better to get the actual Billed Duration as the Init (Cold-Start) phase of a Lambda is not always free depending on which packaging method and runtime you are using: When is the Lambda Init Phase Free, and when is it Billed?

I think this might be really relevant in regard to #176 as currently it might be under-reporting cost as a result.

A simple regex such as /Billed Duration: (?<billed_duration>\d+) ms/ would probably fix it rather than the splits.

[alexcasalboni]

Hi @NeilJed, thanks for reporting this!

I believe it's an interesting observation and it can impact cost considerations for functions that have a very high, non-free initialization time, depending on how you use Lambda Power Tuning.

For historical reasons (before 1ms-billing was introduced in late 2020), the tool used Duration because it represented invocation time much better than Billed Duration - which was rounded at 100ms intervals.

Today this isn't true anymore, but my intuition says the power-tuning impact should be fairly low. Because the "durations" are averaged and outliers are excluded by default (see here and here), using Duration should not have impacted power-tuning results considerably so far. Unless you use a very low num, parallelInvocation=true, discardTopBottom=0, and the init time is considerably long & not free.

Let me explain 😄 (very long thinking-out-lout moment - skip to bottom for final results)

---

As pointed out in the article you shared:

The init phase gets two unthrottled vCPUs, even at very low memory configurations

So for the sake of power-tuning itself, the init duration does not depend on the memory/power configuration and someone might argue that Lambda Power Tuning shouldn't include init time in the analysis because 1/ it's power-independent, 2/ it's often free, and 3/ the tool excludes cold starts from the analysis by default anyway.

That said, when discardTopBottom=0, cold starts are included in the analysis and using Billed Duration (instead of Duration) would indeed affect the analysis, causing the average duration to be higher. Please note this would happen for all power values, so I'd still expect the final power-tuning results to not be affected (e.g. if 512MB was the optimal value, it probably still is).

Another important parameter to keep in mind is parallelInvocation. When false, you only get one cold start per power value, so it's easy to discard with discardTopBottom>0. When true, you can experience multiple cold starts per power value. Since new Function Versions/Aliases are created each time you power-tune, the number of cold starts should be consistent across many power-tuning executions.

Basically, there are 8 major cases to consider.

If parallelInvocation=false:

1. discardTopBottom>0 - all cold starts are ignored - no big deal (Duration ~= Billed Duration)
2. discardTopBottom=0 and your init time is very short - no big deal
3. discardTopBottom=0 and your init time is considerable but free - no big deal
4. discardTopBottom=0 and your init time is considerable and not free
   • this is probably worth fixing because Duration << Billed Duration, but should have a minor impact on the reported average duration because you only get one cold start per power value

If parallelInvocation=true:

6. discardTopBottom>0 - most cold starts are ignored - no big deal
7. discardTopBottom=0 and your init time is very short - no big deal
8. discardTopBottom=0 and your init time is considerable but free - no big deal
9. discardTopBottom=0 and your init time is considerable and not free
   • this is definitely worth fixing because Duration << Billed Duration and it might have a major impact on the reported average duration

---

Apologies for the long reply, I needed to think out loud and confirm that the current implementation works fine for the largest majority of use cases.

To summarize, it totally makes sense to use Billed Duration in all situations 🚀

Will work on a PR to implement this (with a regex) asap 👌 Or do you feel like submitting a PR yourself?

[NeilJed]

@alexcasalboni - thanks for the reply and thoughts.

Yes I agree that in most cases it probably doesn't matter when things average out. The reason I investigated is that we've been trying to figure out the pros/cons of using ZIP vs Container plus we're also looking at running Rust lambdas both of which charge for the init phase.

For those that run "warm" it's maybe 1 in 10,000 invocations that are cold but I can imagine there are maybe less frequently called event driven lambdas that might have a higher rate of cold starts hence it's important.

But yes, I guess using Billed Duration is best as that afterall is the time you're being billed for.

I'll have a go at a PR - JS isn't my strong suite but you've got to learn sometime. :)

[alexcasalboni]

Closing this :)