search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.29k stars 4.74k forks source link

CPU time vs code size tradeoffs #94334

Open adamsitnik opened 1 year ago

adamsitnik commented 1 year ago

While reviewing some of the stale PRs I’ve noticed that we are frequently discussing the tradeoff between improving CPU time at the cost of increased code size. Example: https://github.com/dotnet/runtime/pull/90459#discussion_r1292257807

We have the tools to measure the former, but I am not sure about the latter.

I believe that we should at least provide a clear definition of what we mean by the code size (size of the managed assembly? size of precompiled native code? both?). If possible, BenchmarkDotNet should also include the metrics we care about, by default when benchmarking local dotnet runtime builds. Of course, this should be documented as well.

Ideally, we would also describe our decision process behind making or rejecting such tradeoffs, with some examples.

@jkotas @stephentoub thoughts?

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-meta See info in area-owners.md if you want to be subscribed.

Issue Details
While reviewing some of the stale PRs I’ve noticed that we are frequently discussing the tradeoff between improving CPU time at the cost of increased code size. Example: https://github.com/dotnet/runtime/pull/90459#discussion_r1292257807 We have the tools to measure the former, but I am not sure about the latter. I believe that we should at least provide a clear definition of what we mean by the code size (size of the managed assembly? size of precompiled native code? both?). If possible, BenchmarkDotNet should also include the metrics we care about, by default when benchmarking local dotnet runtime builds. Ideally, we would also describe our decision process behind making or rejecting such tradeoffs, with some examples. @jkotas @stephentoub thoughts?
Author: adamsitnik
Assignees: -
Labels: `area-Meta`, `tenet-performance`
Milestone: -
stephentoub commented 1 year ago

I believe that we should at least provide a clear definition of what we mean by the code size

Native assembly code plus all the runtime data structures required to keep track of everything. IL is generally less interesting.

adamsitnik commented 1 year ago

all the runtime data structures required to keep track of everything

Could you provide some exact names? It would be easier for me to search for it in the ClrMD APIs.

jkotas commented 1 year ago

We have macro benchmarks to measure the published binary sizes and startup time for different application types that will detect this type of regressions. https://github.com/dotnet/runtime/issues/93072 is an example of a regression detected by these benchmarks.

ClrMD APIs.

ClrMD API does not have APIs for this. It is hard to account for all contributing costs reliably.

For micro benchmarking purposes, I typically create a test that has thousand instances of the construct over different types and then measure the working set or startup time for one instance vs. thousand instances. Here is an example of such test from years ago: https://gist.github.com/jkotas/102dc708cca8d2c85002cb47bdd49870

ericstj commented 3 months ago

There are also linker tests that track code size for common application types. @eerhardt

eerhardt commented 3 months ago

We have a set of ASP.NET benchmark apps where we track the native AOT size. Links to the app code:

These are run a couple times a day on the latest bits and any size changes larger than 2% get automatic issues logged. For example:

Published size regression: todosapipublishaot, todosapipublishaot, todosapipublishaot (dotnet/aspnetcore#56139)

We also have a "dotnet new console" template benchmark that is run in the perf lab and it measures the size of the default "Hello World" app. I'm not sure where that code is and can't find the link to the results right now. cc @LoopedBard3 @MichalStrehovsky

MichalStrehovsky commented 3 months ago

We also have a "dotnet new console" template benchmark that is run in the perf lab and it measures the size of the default "Hello World" app. I'm not sure where that code is and can't find the link to the results right now. cc @LoopedBard3 @MichalStrehovsky

I think it's this one: https://github.com/dotnet/performance/tree/main/src/scenarios/emptyconsolenativeaot Some dashboard is at: https://dataexplorer.azure.com/dashboards/c9dc2cc3-d492-4137-ab16-93db390da316?p-_startTime=30days&p-_endTime=now&p-_queue=v-Ubuntu.2204.Amd64.Tiger.Perf#c8e6c269-7ed1-457f-9225-97e9611e4a8a