search

DataObjects-NET / dataobjects-net

https://dataobjects.net
MIT License
60 stars 23 forks source link

Performance issues (Xtensive.Orm 7.1.1 and net7.0) #358

Open niikoo opened 7 months ago

niikoo commented 7 months ago

Hello!

We have a project running on Xtensive.Orm 6.1.1 and .net Framework 4.8 and we have migrated it to Xtensive.Orm 7.1.1 and net7.0. We experience that database tasks are slower than before, even without any changes to the models. We also get many DeadlockExceptions. Is there anything we need to do or any changes in Xtensive.Orm 7.1.1 that affects the performance?

gkverneland commented 6 months ago

This is definitely a problem, does anyone have information about potential fixes?

ketiovv commented 4 months ago

Bump!

I also noticed a major performance issue with .NET 8 after migrating from .NET Framework.

Do you have any advices on this @alex-kulakov @SergeiPavlov?

alex-kulakov commented 4 months ago

Hello everyone, @niikoo, @gkverneland, @ketiovv

As far as I remember 7.1 is declared and build for NET 5 and 6. So we didn't test it on NET 7 nor NET 8 environment.

Can you provide some exact scenarios where it is noticeable so we could track what possible changes may be the cause. Just regular queries, some of them or all of them? Would be good if you and me had the same benchmark query or something to check it.

Is this include only translation or entire roundtrip from executing Expression to getting results?

Chose some query or something else that bothers you and post some performance numbers for 1) 7.1.1 on .Net Framework 4.8 2) 7.1.1 on .Net 7

I will try to reproduce it all along on latest 6.0, 7.0 and 7.1 versions and post some numbers too. Let's figure it out.

Secondly, I need more info about DeadlockException to investigate it, just saying that deadlock appears gives me zero information about how to at least reproduce scenarios so we could fix it.

gkverneland commented 4 months ago

Hi @alex-kulakov

Thanks for you positive response.

We have noticed worse performance across the system without being able to identify exactly where the difference is.

You mentioned running 7.1.1 on .Net Framework 4.8, but it does not look like 7.1.1 targets that framework. Would it be possible to make version 6 run on .net 8? That would make it easier for us to do a fair comparison.

Thanks in advance.

alex-kulakov commented 4 months ago

Hi @gkverneland,

Pardon me, you are right about 7.1.1 on .Net Framework.

If you say it is all across the system then it might be also new framework contributing to the issue, because, for example, Domain building process wasn't changed drastically, there were some improvements, thanks to Sergei and the company he works for (their project is big and they really fight for performance of DataObjects), but nothing really big happened with domain building.

We can compare domain building for starter, I have a model of 200 entities and around 10 associations per entity, I believe that it would be good for performance measurements. To eliminate variable of network interactions and database actions performance I would choose building domain in Skip mode, another option is to stop domain building right before translated upgrade actions executed. this would also eliminate impact of database almost completely.

Speaking of query translation, there was serious change of last step of translation - when we visit SqlDom expressions to translate them to result string for DbCommand. I remember that I checked performance of those changes on a number of query cases and it was at least not worse than before, memory efficiency increased though.

I have everything I need to compare domain building performance. I will start with it for following cases:

1) latest 6.0 version on .Net Framework it is compatible with 2) latest 6.0 version on .Net Core 2.0 3) latest 7.0 version on .Net Core 3.1 (and on other compatible. NETs, if it is possible) 4) latest 7.1 version on .NET 5 5) latest 7.1 version on .NET 6 4) latest 7.1 version on .NET 7 (should be possible) 4) latest 7.1 version on .NET 8 (if possible, I believe it is)

Query translation is a different story, it really depends on complexity of queries because it affects size of expression tree and also methods that are executed throughout translation. I would like to have an example of queries you usually execute in your project as a benchmark. The provider (sqlserver, postgresql, mysql, etc.) you use is important because translation to result string is in assembly of provider. So I need this info from you.

alex-kulakov commented 4 months ago

Unfortunately, I couldn't run all the benchmarks for now only results for DO v6.0. Projects for different frameworks and DataObjects.Net versions are ready but I haven't got enough time to gather all results. I could publish the projects I use for benchmarking, but later.

Here is a portion of numbers of the same model set and two ways of building Domain 1) in Skip mode - full Domain build, in this mode DO does minimal interaction with database 2) in Recreate mode - I intentionally interrupt building before any upgrade actions executed (table/indexes/foreign key creations) to eliminate network and database performance impact, they are translated to strings though.

Domain model contains 200 of entities similar to this

[HierarchyRoot]
[Index("Int16Field")]
[Index("Int32Field")]
[Index("Int64Field")]
[Index("FloatField")]
[Index("DoubleField")]
public class TestEntity12 : Entity
{
  [Key, Field]
  public long Id{get;set;}
  [Field]
  public bool BooleanField {get;set;}
  [Field]
  public Int16 Int16Field {get;set;}
  [Field]
  public Int32 Int32Field {get;set;}
  [Field]
  public Int64 Int64Field {get;set;}
  [Field]
  public float FloatField {get;set;}
  [Field]
  public double DoubleField {get;set;}
  [Field]
  public DateTime DateTimeField {get;set;}
  [Field]
  public string StringField {get;set;}
  [Field]
  [FullText("English")]
  public string Text {get;set;}
  [Field]
  public TestEntity11 TestEntity11{get;set;}
  [Field]
  public TestEntity10 TestEntity10{get;set;}
  [Field]
  public TestEntity9 TestEntity9{get;set;}
  [Field]
  public TestEntity8 TestEntity8{get;set;}
  [Field]
  public TestEntity7 TestEntity7{get;set;}
  [Field]
  public TestEntity6 TestEntity6{get;set;}
  [Field]
  public TestEntity5 TestEntity5{get;set;}
  [Field]
  public TestEntity4 TestEntity4{get;set;}
  [Field]
  public TestEntity3 TestEntity3{get;set;}
  [Field]
  public TestEntity2 TestEntity2{get;set;}
  [Field]
  public TestEntity1 TestEntity1{get;set;}
}

I used SQL Server 2016 as database (installed on the same PC). My PC specs: Windows 10 (10.0.19045.4046/22H2/2022Update) Intel Core i5-8400 CPU 2.80GHz (Coffee Lake), 1 CPU, 6 logical and 6 physical cores I used BenchmarkDotNet v0.13.12

Benchmarks use cold start and 1 run to eliminate possible cache influence. Every table below is a result of one run of executable file.

Number of projects targeting different framework used the same packages of DataObjects.Net (v 6.0.12).

So...

.Net Framework 4.8 (.NET Framework 4.8.1 (4.8.9181.0))

Skip Mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.545 s NA 25000.0000 19000.0000 4000.0000 134.71 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.539 s NA 25000.0000 19000.0000 4000.0000 134.78 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.549 s NA 25000.0000 19000.0000 4000.0000 134.71 MB

Recreate mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.261 s NA 149000.0000 91000.0000 5000.0000 767.82 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.247 s NA 150000.0000 90000.0000 5000.0000 768.02 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.269 s NA 150000.0000 91000.0000 5000.0000 768.07 MB

.Net Core 2.1 (.NET Core 2.1.30 (CoreCLR 4.6.30411.01)

Skip Mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.828 s NA 37000.0000 12000.0000 2000.0000 216.93 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.820 s NA 37000.0000 12000.0000 2000.0000 216.93 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.797 s NA 37000.0000 12000.0000 2000.0000 216.93 MB

Recreate mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.543 s NA 237000.0000 54000.0000 4000.0000 1.22 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.561 s NA 237000.0000 51000.0000 4000.0000 1.22 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.544 s NA 237000.0000 51000.0000 4000.0000 1.22 GB

.NET 5 (.NET 5.0.17 (5.0.1722.21314))

Skip Mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.821 s NA 35000.0000 11000.0000 1000.0000 206.97 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.825 s NA 35000.0000 12000.0000 1000.0000 206.97 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.822 s NA 35000.0000 12000.0000 1000.0000 206.98 MB

Recreate mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.533 s NA 228000.0000 50000.0000 2000.0000 1.17 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.507 s NA 228000.0000 50000.0000 2000.0000 1.17 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.518 s NA 228000.0000 50000.0000 2000.0000 1.17 GB

.NET 6 (.NET 6.0.25 (6.0.2523.51912))

Skip Mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.890 s NA 35000.0000 12000.0000 1000.0000 204.41 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.876 s NA 35000.0000 12000.0000 1000.0000 204.41 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.873 s NA 35000.0000 12000.0000 1000.0000 204.38 MB

Recreate mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.760 s NA 223000.0000 48000.0000 2000.0000 1.15 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.685 s NA 222000.0000 50000.0000 2000.0000 1.15 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.757 s NA 223000.0000 50000.0000 2000.0000 1.15 GB

.NET 7 (.NET 7.0.14 (7.0.1423.51910))

Skip Mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.787 s NA 37000.0000 14000.0000 3000.0000 203.83 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.802 s NA 37000.0000 14000.0000 3000.0000 203.83 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.796 s NA 37000.0000 14000.0000 3000.0000 203.85 MB

Recreate mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.849 s NA 221000.0000 48000.0000 1000.0000 1.14 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.814 s NA 221000.0000 41000.0000 1000.0000 1.14 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 3.833 s NA 221000.0000 41000.0000 1000.0000 1.14 GB

.NET 8 (.NET 8.0.0 (8.0.23.53103))

Skip Mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.806 s NA 35000.0000 12000.0000 1000.0000 203.36 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.818 s NA 35000.0000 12000.0000 1000.0000 203.36 MB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 1.807 s NA 35000.0000 12000.0000 1000.0000 203.36 MB

Recreate mode results

Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 4.024 s NA 217000.0000 46000.0000 1000.0000 1.13 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 4.010 s NA 217000.0000 46000.0000 1000.0000 1.13 GB
Method Mean Error Gen0 Gen1 Gen2 Allocated
Build 4.064 s NA 217000.0000 45000.0000 1000.0000 1.13 GB
alex-kulakov commented 4 months ago

I have finished domain build benchmarking.

Probably I should've explained why I chose Skip mode and Recreate mode for benchmarking. I'll explain now.

Skip. The key feature of Skip upgrade mode is that it assumes that database has already been upgraded. This assumption allows us to skip extraction of database structure and comparison of current domain model state with the state of database. Because of low overhead, this is great mode to benchmark Domain model building process. This is important because Domain model lives in memory the whole life of Domain itself and Domain is assumed to live almost entire life of application.

Recreate. This mode drops everything it can in database and creates entire structure of database. If it is done on already empty database it benchmarks states comparison process (current state of domain and current state of database). It also benchmarks efficiency of CREATE query translation (tables, foreign keys, indexes, etc.), in other words it benchmarks translation to string (to a certain extend).

Why I didn't use Perform/PerformSafely mode? These modes require to have certain state of database which leads to database structure extraction but I would like to exclude this network and storage interactions from result values.

Disclamer.

For better understanding of how different versions of code perform I will arrange tables of results by Framework where every table is one of versions of DataObjects.Net.

Our base line is .Net Framework 4.8 with DO 6.0.12, which looks like this

Method Mean Allocated Method Mean Allocated
Build 1.545 s 134.71 MB Build 3.261 s 767.82 MB
Build 1.539 s 134.78 MB Build 3.247 s 768.02 MB
Build 1.549 s 134.71 MB Build 3.269 s 768.07 MB

Let's start with .NET 8

.NET 8 (.NET 8.0.0 (8.0.23.53103))

v6.0.12

Method Mean Allocated Method Mean Allocated
Build 1.806 s 203.36 MB Build 4.024 s 1.13 GB
Build 1.818 s 203.36 MB Build 4.010 s 1.13 GB
Build 1.807 s 203.36 MB Build 4.064 s 1.13 GB

v7.0.0

Method Mean Allocated Method Mean Allocated
Build 1.850 s 199.26 MB Build 3.909 s 1.05 GB
Build 1.842 s 199.27 MB Build 3.801 s 1.05 GB
Build 1.817 s 199.26 MB Build 3.771 s 1.05 GB
Build 1.834 s 199.26 MB Build 3.800 s 1.05 GB
Build 1.824 s 199.26 MB Build 3.808 s 1.05 GB
Build 1.833 s 199.27 MB Build 3.813 s 1.05 GB

v7.0.1

Method Mean Allocated Method Mean Allocated
Build 2.200 s 187.88 MB Build 4.412 s 1.03 GB
Build 2.151 s 187.87 MB Build 4.345 s 1.03 GB
Build 2.138 s 187.87 MB Build 4.398 s 1.03 GB
Build 2.190 s 187.88 MB Build 4.454 s 1.03 GB
Build 2.199 s 187.88 MB Build 4.212 s 1.03 GB
Build 2.171 s 187.86 MB Build 4.345 s 1.03 GB

v7.0.3

Method Mean Allocated Method Mean Allocated
Build 2.258 s 187.86 MB Build 4.491 s 1.03 GB
Build 2.241 s 187.88 MB Build 4.345 s 1.03 GB
Build 2.229 s 187.87 MB Build 4.501 s 1.03 GB
Build 2.223 s 187.86 MB Build 4.381 s 1.03 GB
Build 2.253 s 187.87 MB Build 4.262 s 1.03 GB
Build 2.238 s 187.87 MB Build 4.246 s 1.03 GB

v7.1.1

Method Mean Allocated Method Mean Allocated
Build 2.251 s 170.8 MB Build 3.868 s 671.71 MB
Build 2.200 s 170.8 MB Build 3.932 s 671.71 MB
Build 2.248 s 170.81 MB Build 3.876 s 671.7 MB
Build 2.220 s 170.8 MB Build 3.874 s 671.73 MB
Build 2.185 s 170.81 MB Build 3.850 s 671.71 MB
Build 2.230 s 170.81 MB Build 3.979 s 671.7 MB
Build 2.219 s 170.81 MB Build 4.048 s 671.73 MB
Build 2.174 s 170.8 MB Build 3.788 s 671.72 MB
Build 2.201 s 170.8 MB Build 3.892 s 671.71 MB
Build 2.227 s 170.8 MB Build 3.879 s 671.7 MB
alex-kulakov commented 4 months ago

.NET 7 (.NET 7.0.14 (7.0.1423.51910))

v6.0.12

Method Mean Allocated Method Mean Allocated
Build 1.787 s 203.83 MB Build 3.849 s 1.14 GB
Build 1.802 s 203.83 MB Build 3.814 s 1.14 GB
Build 1.796 s 203.85 MB Build 3.833 s 1.14 GB

v7.0.0

Method Mean Allocated Method Mean Allocated
Build 1.900 s 199.74 MB Build 3.784 s 1.06 GB
Build 1.869 s 199.73 MB Build 3.719 s 1.06 GB
Build 1.847 s 199.74 MB Build 3.622 s 1.06 GB
Build 1.829 s 199.74 MB Build 3.670 s 1.06 GB
Build 1.842 s 199.74 MB Build 3.747 s 1.06 GB
Build 1.831 s 199.74 MB Build 3.735 s 1.06 GB
Build 1.835 s 199.75 MB Build 3.698 s 1.06 GB
Build 1.850 s 199.74 MB Build 3.829 s 1.06 GB
Build 1.829 s 199.75 MB Build 3.664 s 1.06 GB
Build 1.858 s 199.74 MB Build 3.734 s 1.06 GB

v7.0.3

Method Mean Allocated Method Mean Allocated
Build 2.082 s 188.44 MB Build 3.924 s 1.05 GB
Build 2.091 s 188.44 MB Build 3.931 s 1.05 GB
Build 2.094 s 188.44 MB Build 3.979 s 1.05 GB
Build 2.105 s 188.45 MB Build 4.001 s 1.05 GB
Build 2.109 s 188.45 MB Build 4.006 s 1.05 GB
Build 2.112 s 188.47 MB Build 4.023 s 1.05 GB

v7.1.1

Method Mean Allocated Method Mean Allocated
Build 2.144 s 171.49 MB Build 3.532 s 686.45 MB
Build 2.142 s 171.49 MB Build 3.518 s 686.24 MB
Build 2.157 s 171.5 MB Build 3.541 s 686.26 MB
Build 2.165 s 171.5 MB Build 3.508 s 686.52 MB
Build 2.114 s 171.49 MB Build 3.541 s 686.24 MB
Build 2.107 s 171.5 MB Build 3.540 s 686.49 MB
Build 2.128 s 171.51 MB Build 3.548 s 686.5 MB
Build 2.123 s 171.5 MB Build 3.550 s 686.51 MB
Build 2.174 s 171.5 MB Build 3.536 s 686.42 MB
Build 2.086 s 171.5 MB Build 3.577 s 686.47 MB

.NET 6 (.NET 6.0.25 (6.0.2523.51912))

v6.0.12

Method Mean Allocated Method Mean Allocated
Build 1.890 s 204.41 MB Build 3.760 s 1.15 GB
Build 1.876 s 204.41 MB Build 3.685 s 1.15 GB
Build 1.873 s 204.38 MB Build 3.757 s 1.15 GB

v7.0.0

Method Mean Allocated Method Mean Allocated
Build 1.871 s 200.62 MB Build 3.607 s 1.06 GB
Build 1.878 s 200.66 MB Build 3.617 s 1.06 GB
Build 1.898 s 200.66 MB Build 3.624 s 1.06 GB

v7.0.1

Method Mean Allocated Method Mean Allocated
Build 2.092 s 188.83 MB Build 3.694 s 1.05 GB
Build 2.058 s 188.86 MB Build 3.839 s 1.05 GB
Build 2.077 s 188.83 MB Build 3.849 s 1.05 GB

v7.0.3

Method Mean Allocated Method Mean Allocated
Build 2.070 s 188.86 MB Build 3.797 s 1.05 GB
Build 2.057 s 188.85 MB Build 3.806 s 1.05 GB
Build 2.083 s 188.85 MB Build 3.812 s 1.05 GB

v7.1.1

Method Mean Allocated Method Mean Allocated
Build 2.131 s 171.3 MB Build 3.449 s 686.64 MB
Build 2.131 s 171.28 MB Build 3.472 s 686.62 MB
Build 2.114 s 171.25 MB Build 3.482 s 686.61 MB
Build 2.137 s 171.27 MB Build 3.479 s 686.66 MB
Build 2.134 s 171.27 MB Build 3.515 s 686.65 MB
Build 2.136 s 171.3 MB Build 3.519 s 686.63 MB
Build 2.101 s 171.27 MB Build 3.460 s 686.63 MB
Build 2.133 s 171.26 MB Build 3.508 s 686.65 MB
Build 2.141 s 171.27 MB Build 3.461 s 686.69 MB
Build 2.148 s 171.27 MB Build 3.433 s 686.68 MB

.NET 5 (.NET 5.0.17 (5.0.1722.21314))

v6.0.12

Method Mean Allocated Method Mean Allocated
Build 1.821 s 206.97 MB Build 3.533 s 1.17 GB
Build 1.825 s 206.97 MB Build 3.507 s 1.17 GB
Build 1.822 s 206.98 MB Build 3.518 s 1.17 GB

v7.0.0

Method Mean Allocated Method Mean Allocated
Build 1.854 s 203.2 MB Build 3.504 s 1.09 GB
Build 1.849 s 203.22 MB Build 3.463 s 1.09 GB
Build 1.838 s 203.21 MB Build 3.523 s 1.09 GB

v7.0.1

Method Mean Allocated Method Mean Allocated
Build 1.985 s 191.13 MB Build 3.627 s 1.07 GB
Build 2.010 s 191.14 MB Build 3.562 s 1.07 GB
Build 2.026 s 191.15 MB Build 3.549 s 1.07 GB

7.0.3

Method Mean Allocated Method Mean Allocated
Build 2.049 s 191.12 MB Build 3.612 s 1.07 GB
Build 2.021 s 191.13 MB Build 3.649 s 1.07 GB
Build 1.985 s 191.12 MB Build 3.672 s 1.07 GB

v7.1.1

Method Mean Allocated Method Mean Allocated
Build 2.111 s 172.21 MB Build 3.415 s 688.68 MB
Build 2.115 s 172.18 MB Build 3.369 s 688.72 MB
Build 2.118 s 172.18 MB Build 3.369 s 688.72 MB
Build 2.126 s 172.19 MB Build 3.371 s 688.72 MB
Build 2.165 s 172.2 MB Build 3.380 s 688.72 MB
Build 2.130 s 172.19 MB Build 3.382 s 688.73 MB
Build 2.117 s 172.2 MB Build 3.537 s 688.7 MB
Build 2.129 s 172.2 MB Build 3.404 s 688.72 MB
Build 2.110 s 172.18 MB Build 3.404 s 688.72 MB
Build 2.147 s 172.22 MB Build 3.369 s 688.7 MB
alex-kulakov commented 4 months ago

.Net Core 3.1 (.NET Core 3.1.32 (CoreCLR 4.700.22.55902, CoreFX 4.700.22.56512))

v6.0.12

Method Mean Allocated Method Mean Allocated
Build 1.856 s 207.38 MB Build 3.615 s 1.17 GB
Build 1.844 s 207.38 MB Build 3.672 s 1.17 GB
Build 1.867 s 207.44 MB Build 3.638 s 1.17 GB

v7.0.0

Method Mean Allocated Method Mean Allocated
Build 1.891 s 203.61 MB Build 3.556 s 1.08 GB
Build 1.874 s 203.61 MB Build 3.523 s 1.08 GB
Build 1.881 s 203.61 MB Build 3.504 s 1.08 GB

v7.0.1

Method Mean Allocated Method Mean Allocated
Build 2.122 s 190.39 MB Build 3.884 s 1.07 GB
Build 2.115 s 190.48 MB Build 3.789 s 1.07 GB
Build 2.116 s 190.41 MB Build 3.816 s 1.07 GB

v7.0.3

Method Mean Allocated Method Mean Allocated
Build 2.128 s 190.43 MB Build 3.776 s 1.07 GB
Build 2.149 s 190.39 MB Build 3.817 s 1.07 GB
Build 2.122 s 190.38 MB Build 3.818 s 1.07 GB

No result for 7.1.1, no support for this framework

alex-kulakov commented 4 months ago

We can see that: 1) changing of framework contributes to performance, newer frameworks are less speedy and far less memory efficient. 2) there is a performance drop somewhere around 7.0.1, but we gained some memory usage improvement. 3) 7.1.1 shows great memory efficiency improvements, which were lost thanks to .NET 8 itself. It is still slower than .Net Framework 4.8, I admit, but not by big value.

Assuming that domain lifetime almost equals application lifetime, we can understand that memory efficiency of domain is more important than build time and we can sacrifice some time to improve memory usage. I'm not saying that we should sacrifice, I'm saying that we've improved the most important side of Domain building. Thanks to Sergei. Basically, it was main target of Sergei, because they have not only great number of Entities, but they have big number of Storage nodes (a way of dynamically scale Domain horizontally) and memory consumption of application startup is always important for his team.

I will revisit changes that we made in 7.0.1 to and try to find out what was the cause of performance drop, we will try to gain it back.

@niikoo, @ketiovv, @gkverneland, Take a look at the results.

alex-kulakov commented 4 months ago

The projects I used for benchmarking I published here https://github.com/alex-kulakov/domain-building-perf-check

gkverneland commented 3 months ago

@alex-kulakov Thanks for your information and testing. It is sad to see that performance is worse with newer frameworks and we're looking forward to your further investigations on why version 7.x.x seems to be slower than 6.x.x.

Btw, our concern is not mainly the startup time, but the performance while the system is running. Have you done testing related to heavy workloads in a multi-threaded application with small and large transactions going on at the same time?

alex-kulakov commented 3 months ago

@gkverneland, I agree that startup is not very important, for startup memory efficiency is important and as you can see we managed to improve it significantly.

Secondly, speaking of performance during normal work, I asked you guys what type of queries (or something else) bothers you. I already told that I can create a query, but it may not represent your usual queries. This is important. Since no one gave me one, query benchmarks will be my fantasy, probably unrealistic. :)

Btw, our concern is not mainly the startup time, but the performance while the system is running. Have you done testing related to heavy workloads in a multi-threaded application with small and large transactions going on at the same time?

No, we haven't. The thing is that applications are different. Some of them use query caching heavily, some don't; some of applications use session tweaking to have better performance on row inserts/updates/deletions, and some don't; some of applications use query batching and some don't. And list of differences goes on and on. How should I test of all variations the user can have in his code? I would do only tests of all combinations instead of making actual changes in one particular part of ORM responsibility.

Probably the most important part is queries. If we speak about query execution there are several layers that contributes to overall performance: 1) hardware layer (cpu, memory, hdd/ssd); 2) database performance; 3) network; 4) performance of database client library (Microsoft.Data.SqlClient, Npgsql, etc.); 5) performance of our Expression to SQL string translation. 6) results materialization

We can't control first 4 layers, we should not be responsible for them. I can benchmark 5th and 6th. The 5th will show how fast and efficient expression is transformed to plain text of query. This is what I'm going to measure, for this purpose I'm not going to execute DbCommands and stop on getting SQL query string.

The 6th layer benchmarks are harder because on the 4th layer each client library implements receiving results differently - some use direct reading of results from DbDataReader, others load data to internal collection and then give results. This makes benchmark results of one storage not applicable to others. But no one answered to my question about provider you, guys, use. I will skip benchmarking this part.

Next part is persisting changes to database. There weren't many changes in this part of DataObjects.net so its performance is mostly dictated by .NET (its improvements and degradations :), we saw them in example of 6.0 in pair with different frameworks). This part depends on relations in model because it uses graphs to order insert/update/delete operations. For now, I skip benchmarking it unless you are interested in this.

Next part - session openings, transactions openings/commits/rollbacks. It is simple here, mostly it is dictated by client library and database. We do have layers of abstraction that wraps DbConnection and DbTransaction but the code is simple and can be represented as a chain of method calls. I would be less concerned about this.

You say about "transactions going at the same time". Correct me if I'm wrong but to me this a database responsibility, not ours. It is also partially your responsibility as developer because you decide how to manage transactions - when to open them and when to close, durations and isolation levels. We can't do much about it. Our job is to call DbConnection.Open(), dbConnection.BeginTransaction(), DbTransaction.Commit()/.Rollback() the most efficient way. Right? What may confuse you is long execution of TransactionScope.Dispose(). I'm not sure whether you familiar with one DataObjects.Net particularity - if there are some unsaved changes and TransactionScope is marked as Complete, then unsaved changes are persisted during Transaction scope disposal. Some people don't know this.

Next, multithreaded performance. First of all, we see Session as a unit of parallelism. One Session one thread, this concept allows asynchronous execution but not parallel execution. We declared it in documentation. We develop with this assumption in mind, I hope you do the same. Sergei's team discovered that our domain-level caches are slow on high number of threads because of locks, these collections were developed long time. We replaced our collections with counterparts from BitFaster library. This happened in 7.1. By doing so multi-threaded performance of domain-level LRU caches was improved significantly. Other global caches use collections from System.Collections.Concurrent namespace, the collections are lock-free so they should be fine.

I believe that if we (me and you guys) start searching for exact cases, we will be more productive and chances of improving overall performance by improving exact weaknesses are higher. This is why I asked about particular cases that stand out on your side and should be addressed.

alex-kulakov commented 2 months ago

Hello,

I'm back with additional results, this time for query translation measurements. I've created following LINQ query (if somebody wants to provide his own query, I don't mind, I would glad to benchmark close-to-life query):

public static IQueryable<VehicleDto> MakeQuery(Session session)
{
  return session.Query.All<IVehicle>()
    .Where(iv => iv.EnergySource == EnergySource.DieselFuel
      || iv.EnergySource == EnergySource.Gasoline
      || iv.EnergySource == EnergySource.Electricity
      || iv.ModelName.StartsWith("A") || iv.ModelName.StartsWith("B") || iv.ModelName.StartsWith("C")
      || iv.ModelName.Contains("C") || iv.ModelName.Contains("D") || iv.ModelName.Contains("E")
      || iv.Type.In(IncludeAlgorithm.ComplexCondition,
           VehicleType.Car, VehicleType.Truck, VehicleType.Ship, VehicleType.PersonalMobilityDevice)
      || (iv.Id > 100 && iv.Id < 1_000_000)
      || (iv.Timestamp.Year - iv.Timestamp.Day) > 1500
      || (iv.Timestamp.AddDays(3) > DateTime.UtcNow)
    )
    .LeftJoin(
      session.Query.All<Person>()
        .Where(p => p.FirstName.Contains("A") || p.FirstName.Contains("B") || p.FirstName.Contains("C")
          || p.FirstName.StartsWith("D") || p.FirstName.StartsWith("Y") || p.FirstName.StartsWith("Z")
          || p.Immovables.Count > 0
          || (p.Id > 100 && p.Id < 1_000_000)
          || (p.Timestamp.Year - p.Timestamp.Day) > 1500
          || (p.Timestamp.AddDays(3) > DateTime.UtcNow)
          )
        ,
      iv => iv.Owner.Id,
      p => p.Id,
      (iv, p) => new { IVehicle = iv, VehicleOwner = p })
    .LeftJoin(
      session.Query.All<VehicleManufacturer>()
        .Where(vm => vm.Country.StartsWith("A") || vm.Country.StartsWith("B") || vm.Country.StartsWith("C")
          || vm.Country.Contains("C") || vm.Name.Contains("D") || vm.Name.Contains("Z")
          || (vm.Id > 100 && vm.Id < 1_000_000)
          || (vm.Timestamp.Year - vm.Timestamp.Day) > 1500
          || (vm.Timestamp.AddDays(3) > DateTime.UtcNow))
        ,
      a => a.IVehicle.Manufacturer.Id,
      vm => vm.Id,
      (a, vm) => new { IVehicle = a.IVehicle, VehicleOwner = a.VehicleOwner, VehicleManufacturer = vm })
    .LeftJoin(
      session.Query.All<VehicleRegistrationInfo>()
        .Where(ri => ri.LicensePlate.StartsWith("B")
          || ri.LicensePlate.StartsWith("C")
          || ri.LicensePlate.EndsWith("Z")
          || ri.LicensePlate.EndsWith("Y")
          || (ri.Id > 100 && ri.Id < 1_000_000)
          || (ri.Timestamp.Year - ri.Timestamp.Day) > 1500
          || (ri.Timestamp.AddDays(3) > DateTime.UtcNow))
        ,
      a => a.IVehicle.RegistrationInfo.Id,
      ri => ri.Id,
      (a, ri) => new {
        IVehicle = a.IVehicle,
        VehicleOwner = a.VehicleOwner,
        VehicleManufacturer = a.VehicleManufacturer,
        VehicleRegistrationInfo = ri })
    .LeftJoin(
      session.Query.All<AddressInfo>()
        .Where(ai => ai.City.Length > 5 || ai.Address.Length > 10 || ai.ContactType == ContactType.Address),
      a => a.VehicleRegistrationInfo.RegistrationAddress.Id,
      ra => ra.Id,
      (a, ra) => new {
        IVehicle = a.IVehicle,
        VehicleOwner = a.VehicleOwner,
        VehicleManufacturer = a.VehicleManufacturer,
        VehicleRegistrationInfo = a.VehicleRegistrationInfo,
        VehicleRegistractionAddress = ra
      })
    .Select(a => new VehicleDto
    {
      VehicleId = a.IVehicle.Id,
      VehicleRecordTimestamp = a.IVehicle.Timestamp,
      VehicleModelName = a.IVehicle.ModelName,
      VehicleVinNumber = a.IVehicle.VinNumber,
      VehicleEngineSource = a.IVehicle.EnergySource,
      VehicleEnginePower = a.IVehicle.EnginePower,
      VehicleType = a.IVehicle.Type,
      VehicleOwner = new VehicleOwnerDto
      {
        OwnerId = a.VehicleOwner.Id,
        OwnerRecordTimeStamp = a.VehicleOwner.Timestamp,
        OwnerFirstName = a.VehicleOwner.FirstName,
        OwnerLastName = a.VehicleOwner.LastName,
        NumberOfVehicles = a.VehicleOwner.Vehicles.Count,
        NumberOfImmovables = a.VehicleOwner.Immovables.Count,
      },
      VehicleManufacturer = new VehicleManufacturerDto
      {
        ManufacturerId = a.VehicleManufacturer.Id,
        ManufacturerRecordTimeStamp = a.VehicleManufacturer.Timestamp,
        ManufacturerName = a.VehicleManufacturer.Name,
        ManufacturerCountry = a.VehicleManufacturer.Country
      },
      VehicleRegistrationInfo = new VehicleRegistrationInfoDto
      {
        RegistrationId = a.VehicleRegistrationInfo.Id,
        RegistrationRecordTimeStamp = a.VehicleRegistrationInfo.Timestamp,
        RegistrationLicensePlate = a.VehicleRegistrationInfo.LicensePlate,
        RegistrationAddress = new RegistrationAddressDto
        {
          AddressId = a.VehicleRegistractionAddress.Id,
          AddressPostCode = a.VehicleRegistractionAddress.PostCode,
          AddressCountry = a.VehicleRegistractionAddress.Country,
          AddressCity = a.VehicleRegistractionAddress.City,
          AddressAddress = a.VehicleRegistractionAddress.Address
        }
      }
    });
}

The idea in the query was to have different statements of SQL query.

I didn't benchmark the whole round-trip (from linq to sql command than execute and materialize) because it would bring network and RDBMS performance to the equation and dilute results.

DataObjects.Net has a built-in service called QueryBuilder that will help to benchmark entire process as four parts of translation. I used it to gather info about each part to discover performance per part for better localization of problem (if it exists) for further possible investigation.

So, QueryBuilder service has number of methods. We will user four of them. 1) TranslateQuery() - gets IQueryable<T> and translates it to SQL DOM expressions; 2) CompileQuery() - gets result of TranslateQuery() and compiles them to pre-text state, containing Nodes, most of them containing final text of different parts of result command text, but some of them serve the purpose of being able to make final adjustments to command text if needed; 3) CreateRequest() - basically unites compilation results and bindings, provided by translation, into single structure for the last step; 4) CreateCommand() - creates DbCommand, fills in DbParameters and makes final preparation of the Nodes I mentioned before. This is a final step; result command is ready to be executed.

To have final results more precise, I moved results of previous steps (if any exists) out of benchmark scope. For instance, benchmark of CreateCommand() step looks like this

[MemoryDiagnoser]
[Orderer(BenchmarkDotNet.Order.SummaryOrderPolicy.Declared)]
[RankColumn]
public class QueryBuilderStepFourBenchmark : TranslationBenchmarkBase
{
  private QueryBuilder queryBuilder;
  private QueryTranslationResult translationResult;
  private SqlCompilationResult sqlCompilationResult;
  private QueryRequest request;

  public override void BeforeEveryIteration()
  {
    base.BeforeEveryIteration();
    queryBuilder = session.Services.GetService<QueryBuilder>();
    translationResult = queryBuilder.TranslateQuery(cachedQuery); //Step #1
    sqlCompilationResult = queryBuilder.CompileQuery(translationResult.Query); // Step #2
    request = queryRunner.CreateRequest(queryBuilder, sqlCompilationResult, translationResult.ParameterBindings); // Step #3
  }

  public override void AfterEveryIteration()
  {
    queryBuilder = null;
    request = null;
    sqlCompilationResult = null;
    translationResult = null;
    base.AfterEveryIteration();
  }

  [Benchmark]
  public void CreateQueryCommand()
  {
    _ = queryRunner.CreateQueryCommand(queryBuilder, request); // Step #4, which we actually benchmark
  }

  public QueryBuilderStepFourBenchmark() : base()
  {
  }
}

I believe such approach is right for getting correct picture of specific step.

I performed benchmarks for v6.0.12, v7.0.0, v7.0.1, v7.0.2, v7.0.3, v7.0.4 and v7.1.1 across different frameworks and grouped results by framework and ordered by DO version.

alex-kulakov commented 2 months ago

Translate() method results

DO version .NET version Method Mean Error StdDev Rank Allocated
6.0.12 NET 4.7.1 Translate 2.840 ms 0.0546 ms 0.0584 ms 1 586.79 KB
6.0.12 NET 4.7.2 Translate 2.771 ms 0.0251 ms 0.0196 ms 1 586.79 KB
6.0.12 NET 4.8 Translate 2.840 ms 0.0544 ms 0.0509 ms 1 586.79 KB
6.0.12 NET Core 2.1 Translate 3.075 ms 0.0607 ms 0.0568 ms 1 959.99 KB
------------ ------------- ---------- ---------- ----------- ----------- ------ -----------
6.0.12 NET Core 3.1 Translate 3.836 ms 0.2027 ms 0.5815 ms 1 938.87 KB
7.0.0 NET Core 3.1 Translate 3.748 ms 0.1801 ms 0.5079 ms 1 927.49 KB
7.0.1 NET Core 3.1 Translate 3.552 ms 0.2150 ms 0.6204 ms 1 912.91 KB
7.0.2 NET Core 3.1 Translate 3.663 ms 0.1618 ms 0.4642 ms 1 920.1 KB
7.0.3 NET Core 3.1 Translate 3.685 ms 0.1851 ms 0.5311 ms 1 921.91 KB
7.0.4 NET Core 3.1 Translate 3.697 ms 0.1881 ms 0.5396 ms 1 922.77 KB
------------ ------------- ---------- ---------- ----------- ----------- ------ -----------
6.0.12 NET5 Translate 2.986 ms 0.1799 ms 0.5276 ms 1 925.95 KB
7.0.0 NET5 Translate 2.903 ms 0.1614 ms 0.4734 ms 1 915.78 KB
7.0.1 NET5 Translate 2.942 ms 0.1705 ms 0.5001 ms 1 901.47 KB
7.0.2 NET5 Translate 3.276 ms 0.1313 ms 0.3829 ms 1 907.23 KB
7.0.3 NET5 Translate 2.962 ms 0.1657 ms 0.4885 ms 1 907.34 KB
7.0.4 NET5 Translate 3.579 ms 0.0612 ms 0.0478 ms 1 910.38 KB
7.1.1 NET5 Translate 2.913 ms 0.1666 ms 0.4887 ms 1 851.67 KB
------------ ------------- ---------- ---------- ----------- ----------- ------ -----------
7.0.0 NET6 Translate 3.285 ms 0.1166 ms 0.3420 ms 1 916.5 KB
7.0.1 NET6 Translate 3.214 ms 0.1332 ms 0.3887 ms 1 901.46 KB
7.0.2 NET6 Translate 3.343 ms 0.1148 ms 0.3367 ms 1 909.5 KB
7.0.3 NET6 Translate 3.159 ms 0.1215 ms 0.3545 ms 1 909.5 KB
7.0.4 NET6 Translate 3.223 ms 0.1160 ms 0.3401 ms 1 911.81 KB
7.1.1 NET6 Translate 3.242 ms 0.1111 ms 0.3042 ms 1 850.29 KB
------------ ------------- ---------- ---------- ----------- ----------- ------ -----------
7.0.0 NET7 Translate 3.807 ms 0.1889 ms 0.5510 ms 1 891.55 KB
7.0.1 NET7 Translate 3.896 ms 0.1716 ms 0.4840 ms 1 876.38 KB
7.0.2 NET7 Translate 4.403 ms 0.1470 ms 0.4265 ms 1 885.72 KB
7.0.3 NET7 Translate 4.082 ms 0.1984 ms 0.5788 ms 1 886.33 KB
7.0.4 NET7 Translate 4.045 ms 0.1650 ms 0.4786 ms 1 887.15 KB
7.1.1 NET7 Translate 4.362 ms 0.1272 ms 0.3691 ms 1 830.15 KB
------------ ------------- ---------- ---------- ----------- ----------- ------ -----------
7.0.0 NET8 Translate 5.493 ms 0.1482 ms 0.4181 ms 1 890.65 KB
7.0.1 NET8 Translate 5.967 ms 0.1640 ms 0.4810 ms 1 873.07 KB
7.0.2 NET8 Translate 5.937 ms 0.1968 ms 0.5772 ms 1 881.73 KB
7.0.3 NET8 Translate 5.769 ms 0.1615 ms 0.4634 ms 1 880.79 KB
7.0.4 NET8 Translate 5.891 ms 0.1674 ms 0.4937 ms 1 882.16 KB
7.1.1 NET8 Translate 5.902 ms 0.1843 ms 0.5376 ms 1 825.9 KB

I can clearly see how newer versions of .NET kill performance. After performance drop in .NET Core 2.1 and 3.1, it gets closer to .Net Framework 4.x levels in NET5, NET6 is still OK and then it drops significantly.

CompileQuery() method results

DO version .NET version Method Mean Error StdDev Rank Allocated
6.0.12 NET 4.7.1 CompileTranslated 338.8 μs 6.73 μs 10.86 μs 1 172.37 KB
6.0.12 NET 4.7.2 CompileTranslated 338.7 μs 6.77 μs 14.29 μs 1 172.37 KB
6.0.12 NET 4.8 CompileTranslated 335.6 μs 6.72 μs 11.41 μs 1 172.37 KB
6.0.12 NET Core 2.1 CompileTranslated 376.9 μs 7.36 μs 11.02 μs 1 240.73 KB
------------ ------------- ------------------ --------- --------- --------- ----- ----------
6.0.12 NET Core 3.1 CompileTranslated 528.5 μs 10.55 μs 25.68 μs 1 235.86 KB
7.0.0 NET Core 3.1 CompileTranslated 477.6 μs 23.53 μs 65.97 μs 1 235.32 KB
7.0.1 NET Core 3.1 CompileTranslated 461.4 μs 13.93 μs 39.97 μs 1 235.32 KB
7.0.2 NET Core 3.1 CompileTranslated 444.8 μs 13.28 μs 37.67 μs 1 235.23 KB
7.0.3 NET Core 3.1 CompileTranslated 380.9 μs 26.29 μs 75.85 μs 1 235.23 KB
7.0.4 NET Core 3.1 CompileTranslated 376.0 μs 23.00 μs 65.25 μs 1 235.23 KB
------------ ------------- ------------------ --------- --------- --------- ----- ----------
6.0.12 NET5 CompileTranslated 415.1 μs 8.02 μs 13.17 μs 1 235.89 KB
7.0.0 NET5 CompileTranslated 350.9 μs 26.42 μs 77.47 μs 1 235.35 KB
7.0.1 NET5 CompileTranslated 374.3 μs 27.07 μs 79.38 μs 1 235.35 KB
7.0.2 NET5 CompileTranslated 358.2 μs 22.84 μs 65.89 μs 1 235.27 KB
7.0.3 NET5 CompileTranslated 303.6 μs 12.42 μs 34.00 μs 1 235.27 KB
7.0.4 NET5 CompileTranslated 312.8 μs 13.57 μs 37.59 μs 1 235.27 KB
7.1.1 NET5 CompileTranslated 343.5 μs 35.87 μs 105.2 μs 1 82.55 KB
------------ ------------- ------------------ --------- --------- --------- ----- ----------
7.0.0 NET6 CompileTranslated 398.1 μs 7.91 μs 17.36 μs 1 237.02 KB
7.0.1 NET6 CompileTranslated 404.9 μs 10.15 μs 29.29 μs 1 236.94 KB
7.0.2 NET6 CompileTranslated 402.5 μs 8.01 μs 17.92 μs 1 236.85 KB
7.0.3 NET6 CompileTranslated 327.0 μs 16.11 μs 47.26 μs 1 236.85 KB
7.0.4 NET6 CompileTranslated 332.1 μs 16.55 μs 47.76 μs 1 236.85 KB
7.1.1 NET6 CompileTranslated 274.8 μs 13.91 μs 40.35 μs 1 82.84 KB
------------ ------------- ------------------ --------- --------- --------- ----- ----------
7.0.0 NET7 CompileTranslated 378.7 μs 27.07 μs 77.68 μs 1 236.8 KB
7.0.1 NET7 CompileTranslated 369.2 μs 24.72 μs 71.34 μs 1 236.7 KB
7.0.2 NET7 CompileTranslated 499.4 μs 18.65 μs 53.21 μs 1 236.62 KB
7.0.3 NET7 CompileTranslated 488.8 μs 20.41 μs 58.23 μs 1 236.62 KB
7.0.4 NET7 CompileTranslated 503.7 μs 14.10 μs 38.84 μs 1 236.62 KB
7.1.1 NET7 CompileTranslated 430.3 μs 11.20 μs 31.97 μs 1 82.55 KB
------------ ------------- ------------------ --------- --------- --------- ----- ----------
7.0.0 NET8 CompileTranslated 818.8 μs 16.26 μs 38.34 μs 1 236.34 KB
7.0.1 NET8 CompileTranslated 710.9 μs 14.14 μs 33.32 μs 1 236.22 KB
7.0.2 NET8 CompileTranslated 800.9 μs 21.88 μs 62.08 μs 1 236.14 KB
7.0.3 NET8 CompileTranslated 786.9 μs 39.16 μs 115.5 μs 1 236.14 KB
7.0.4 NET8 CompileTranslated 649.7 μs 12.85 μs 30.04 μs 1 236.14 KB
7.1.1 NET8 CompileTranslated 580.6 μs 11.49 μs 27.97 μs 1 81.38 KB

Same story. Performance recovers in NET 5 and NET 6 and then deeps down in NET 7 and NET 6.

CreateRequest() results

DO version .NET version Method Mean Error StdDev Rank Allocated
6.0.12 NET 4.7.1 CreateRequest 4.699 μs 0.5321 μs 1.518 μs 1 16 KB
6.0.12 NET 4.7.2 CreateRequest 4.732 μs 0.6092 μs 1.777 μs 1 16 KB
6.0.12 NET 4.8 CreateRequest 4.665 μs 0.5419 μs 1.572 μs 1 16 KB
6.0.12 NET Core 2.1 CreateRequest 8.597 μs 0.7603 μs 2.194 μs 1 560 B
------------ ------------- -------------- --------- ---------- --------- ----- ----------
6.0.12 NET Core 3.1 CreateRequest 10.86 μs 1.595 μs 4.576 μs 1 560 B
7.0.0 NET Core 3.1 CreateRequest 12.74 μs 2.165 μs 6.212 μs 1 560 B
7.0.1 NET Core 3.1 CreateRequest 12.98 μs 1.590 μs 4.510 μs 1 560 B
7.0.2 NET Core 3.1 CreateRequest 11.12 μs 1.283 μs 3.598 μs 1 560 B
7.0.3 NET Core 3.1 CreateRequest 11.92 μs 1.469 μs 4.145 μs 1 560 B
7.0.4 NET Core 3.1 CreateRequest 12.57 μs 1.775 μs 5.065 μs 1 560 B
------------ ------------- -------------- --------- ---------- --------- ----- ----------
6.0.12 NET5 CreateRequest 8.471 μs 0.7881 μs 2.286 μs 1 568 B
7.0.0 NET5 CreateRequest 8.214 μs 0.8363 μs 2.386 μs 1 568 B
7.0.1 NET5 CreateRequest 8.793 μs 0.9189 μs 2.651 μs 1 568 B
7.0.2 NET5 CreateRequest 9.151 μs 1.014 μs 2.925 μs 1 568 B
7.0.3 NET5 CreateRequest 9.264 μs 1.017 μs 2.935 μs 1 568 B
7.0.4 NET5 CreateRequest 9.141 μs 0.9475 μs 2.734 μs 1 568 B
7.1.1 NET5 CreateRequest 9.426 μs 0.9184 μs 2.650 μs 1 1.75 KB
------------ ------------- -------------- --------- ---------- --------- ----- ----------
7.0.0 NET6 CreateRequest 8.925 μs 0.8865 μs 2.500 μs 1 2.22 KB
7.0.1 NET6 CreateRequest 9.330 μs 0.8908 μs 2.570 μs 1 2.14 KB
7.0.2 NET6 CreateRequest 9.514 μs 0.9771 μs 2.835 μs 1 2.14 KB
7.0.3 NET6 CreateRequest 9.285 μs 0.8970 μs 2.574 μs 1 2.14 KB
7.0.4 NET6 CreateRequest 8.885 μs 0.8147 μs 2.311 μs 1 2.14 KB
7.1.1 NET6 CreateRequest 10.61 μs 0.847 μs 2.459 μs 1 3.34 KB
------------ ------------- -------------- --------- ---------- --------- ----- ----------
7.0.0 NET7 CreateRequest 11.25 μs 0.828 μs 2.363 μs 1 2.01 KB
7.0.1 NET7 CreateRequest 11.34 μs 0.861 μs 2.444 μs 1 1.91 KB
7.0.2 NET7 CreateRequest 11.79 μs 1.082 μs 3.104 μs 1 1.91 KB
7.0.3 NET7 CreateRequest 11.66 μs 1.016 μs 2.933 μs 1 1.91 KB
7.0.4 NET7 CreateRequest 11.90 μs 0.942 μs 2.701 μs 1 1.91 KB
7.1.1 NET7 CreateRequest 11.87 μs 0.756 μs 2.182 μs 1 3.04 KB
------------ ------------- -------------- --------- ---------- --------- ----- ----------
7.0.0 NET8 CreateRequest 10.57 μs 1.004 μs 2.912 μs 1 2.03 KB
7.0.1 NET8 CreateRequest 11.73 μs 0.998 μs 2.912 μs 1 1.91 KB
7.0.2 NET8 CreateRequest 12.51 μs 1.071 μs 3.106 μs 1 1.91 KB
7.0.3 NET8 CreateRequest 12.15 μs 0.896 μs 2.542 μs 1 1.91 KB
7.0.4 NET8 CreateRequest 11.91 μs 1.061 μs 3.043 μs 1 1.91 KB
7.1.1 NET8 CreateRequest 12.41 μs 0.957 μs 2.761 μs 1 3.04 KB

Story repeats. I already know how to slightly improve this region but since this part is basically several objects' instantiations and copying items of collection, we can do almost nothing for this part of translation. I did some research, maybe 1-2 nanoseconds and some memory efficiency will be gained. I already committed the changes in my private copy of the repo.

CreateCommand() results

DO version .NET version Method Mean Error StdDev Rank Allocated
6.0.12 NET 4.7.1 CreateQueryCommand 35.98 μs 1.814 μs 5.205 μs 1 109.1 KB
6.0.12 NET 4.7.2 CreateQueryCommand 34.36 μs 1.468 μs 4.187 μs 1 109.1 KB
6.0.12 NET 4.8 CreateQueryCommand 34.78 μs 1.564 μs 4.412 μs 1 109.1 KB
6.0.12 NET Core 2.1 CreateQueryCommand 33.18 μs 1.890 μs 5.542 μs 1 90.62 KB
------------ ------------- ------------------- --------- --------- --------- ----- ----------
6.0.12 NET Core 3.1 CreateQueryCommand 60.14 μs 8.987 μs 25.49 μs 1 90.42 KB
7.0.0 NET Core 3.1 CreateQueryCommand 58.01 μs 7.598 μs 21.18 μs 1 91.67 KB
7.0.1 NET Core 3.1 CreateQueryCommand 63.95 μs 9.164 μs 25.70 μs 1 91.67 KB
7.0.2 NET Core 3.1 CreateQueryCommand 61.33 μs 7.666 μs 21.37 μs 1 91.58 KB
7.0.3 NET Core 3.1 CreateQueryCommand 51.88 μs 2.965 μs 7.759 μs 1 91.58 KB
7.0.4 NET Core 3.1 CreateQueryCommand 56.64 μs 5.635 μs 15.33 μs 1 91.58 KB
------------ ------------- ------------------- --------- --------- --------- ----- ----------
6.0.12 NET5 CreateQueryCommand 44.99 μs 2.438 μs 6.916 μs 1 90.45 KB
7.0.0 NET5 CreateQueryCommand 49.75 μs 3.703 μs 10.56 μs 1 91.7 KB
7.0.1 NET5 CreateQueryCommand 51.17 μs 3.998 μs 11.28 μs 1 91.7 KB
7.0.2 NET5 CreateQueryCommand 53.17 μs 4.103 μs 11.77 μs 1 91.61 KB
7.0.3 NET5 CreateQueryCommand 49.89 μs 3.014 μs 8.598 μs 1 91.61 KB
7.0.4 NET5 CreateQueryCommand 52.43 μs 3.292 μs 9.392 μs 1 91.61 KB
7.1.1 NET5 CreateQueryCommand 68.96 μs 4.334 μs 12.44 μs 1 114.94 KB
------------ ------------- ------------------- --------- --------- --------- ----- ----------
7.0.0 NET6 CreateQueryCommand 44.02 μs 2.832 μs 7.895 μs 1 93.37 KB
7.0.1 NET6 CreateQueryCommand 50.77 μs 3.196 μs 9.170 μs 1 93.29 KB
7.0.2 NET6 CreateQueryCommand 47.19 μs 3.221 μs 9.139 μs 1 93.2 KB
7.0.3 NET6 CreateQueryCommand 46.43 μs 2.506 μs 6.987 μs 1 93.2 KB
7.0.4 NET6 CreateQueryCommand 48.09 μs 3.438 μs 9.585 μs 1 93.2 KB
7.1.1 NET6 CreateQueryCommand 70.61 μs 4.321 μs 12.26 μs 1 116.52 KB
------------ ------------- ------------------- --------- --------- --------- ----- ----------
7.0.0 NET7 CreateQueryCommand 56.09 μs 3.093 μs 8.723 μs 1 93.16 KB
7.0.1 NET7 CreateQueryCommand 62.40 μs 3.515 μs 9.856 μs 1 93.05 KB
7.0.2 NET7 CreateQueryCommand 61.46 μs 3.923 μs 11.13 μs 1 92.96 KB
7.0.3 NET7 CreateQueryCommand 62.14 μs 3.766 μs 10.62 μs 1 92.96 KB
7.0.4 NET7 CreateQueryCommand 63.50 μs 3.667 μs 10.16 μs 1 92.96 KB
7.1.1 NET7 CreateQueryCommand 93.73 μs 5.221 μs 14.98 μs 1 116.23 KB
------------ ------------- ------------------- --------- --------- --------- ----- ----------
7.0.0 NET8 CreateQueryCommand 64.78 μs 4.068 μs 11.67 μs 1 93.18 KB
7.0.1 NET8 CreateQueryCommand 66.94 μs 3.396 μs 9.467 μs 1 93.05 KB
7.0.2 NET8 CreateQueryCommand 66.06 μs 3.708 μs 10.34 μs 1 92.59 KB
7.0.3 NET8 CreateQueryCommand 66.74 μs 3.193 μs 9.314 μs 1 92.96 KB
7.0.4 NET8 CreateQueryCommand 66.62 μs 3.565 μs 10.17 μs 1 92.96 KB
7.1.1 NET8 CreateQueryCommand 102.9 μs 4.44 μs 12.68 μs 1 116.23 KB

Here, unfortunately I see significant drop in version 7.1.1. but framework also contribute to losses. In 7.1.1 we changed SQL DOM expression translation to Nodes, which might have shifted some compute time to final preparation, but I'm not sure. Currently I'm working with this part, trying to find points for improvement.

I left the big parts to be last, because they are much complicated and will take a lot of time to find ways to improve. I will try to compensate losses caused by .NET 7 and .NET 8, .NET5 and NET6 will also be improved.

alex-kulakov commented 2 months ago

@gkverneland, @niikoo, @ketiovv take a look. I'd like to know your thoughts.

gkverneland commented 2 months ago

@alex-kulakov Thanks a lot for your investigations so far. It is sad to see that for Xtensive.Orm, all tests show that performance is way worse with .Net 8 than before.

I see that version 6.x.x is not included in your tests for .Net 7 and .Net 8, have you tested that combination? It would be very interesting to have tests where the Xtensive.Orm code is as similar as possible. We have been able to run version 6.x.x with .Net 7 and 8, and can give you the fork if you are interested.

Btw, have you seen this issue about performance for expression compilation? Is it relevant for this project? https://github.com/dotnet/runtime/issues/75891 https://github.com/dotnet/runtime/issues/76058

niikoo commented 2 months ago

@gkverneland, @niikoo, @ketiovv take a look. I'd like to know your thoughts.

Hello!

Thanks for taking the time to reply to us.

Do you have a link to the benchmark project that you've used here: https://github.com/DataObjects-NET/dataobjects-net/issues/358#issuecomment-2056662526

alex-kulakov commented 2 months ago

@gkverneland

I see that version 6.x.x is not included in your tests for .Net 7 and .Net 8, have you tested that combination?

Yes, It is not included because there are some breaking changes in .net that prevented usage of already built packages. The benchmark project and Domain build successfully but queries don't work. To be honest, I don't want to resolve all the issues connected with migration once again just for benchmarks, I just assumed that 7.0.0 is good enough to substitute 6.0.12. If you have a version of 6.0.12 with changes that can be used in .net 5, 6 and so on I would get such version in form of packages. I believe it will be fastest way. Put some suffix in Version.props file before build, for instance <DoVersionSuffix>net5-compatible</DoVersionSuffix>. You can reach me directly via alexey.kulakov@dataobjects.net

I'll gladly include it to benchmarks and post results here.

Btw, have you seen this issue about performance for expression compilation? Is it relevant for this project?

Not yet, but I will check it out. Thank you for pointing me to these two issues. Something outside DataObjects.Net ruins the performance for sure, and since translation uses reflection a lot there must be degradation of performance somewhere there.

alex-kulakov commented 2 months ago

@niikoo,

Do you have a link to the benchmark project that you've used here:

Not yet, it is not a problem to post. I just use the projects to monitor how my changes affect performance, so it has some garbage right now. If you are interested, I'll clean it up and post as repo as before, it is not a problem, I have nothing to hide 😉

alex-kulakov commented 2 months ago

BTW guys, do you use "cached queries" feature of DataObjects.Net? I'm talking about session.Query.Execute() group of methods. Though, not all the queries can be transformed into cacheable form, many of them can be, which allows to skip the most time-consuming part of query translation after first run of the query, so you might be interested in such option.

alex-kulakov commented 2 months ago

@niikoo,

The repo is here https://github.com/alex-kulakov/query-translation-perf-check

alex-kulakov commented 1 month ago

Hello guys, @gkverneland, @niikoo, @ketiovv

I have revised code and found some points for improvement, they are posted as the PR - https://github.com/DataObjects-NET/dataobjects-net/pull/392,

They are small, no drastic difference but still, also not all of them are connected with the test query in the query-translation-perf-check repository.

During my investigation I captured this TranslateQueryCPUUsageByFunctions

The functions are sorted by CPU Self time, which shows the most inefficient methods on the top. As you can see most percentage of time spent in external or native code. The items that take 3.57% of time are very small, for instance ExtendedExpression constructor is a constructor that assigns few properties.

It doesn't mean that our code is the most efficient, I believe there are still points for improvement like in any app/framework/etc, but it shows that outside code impacts performance seriously.

SergeiPavlov commented 1 month ago

Actually we can reduce the amount of time spent in External/Native code by improving callers of that code.

For example: I found O(n²) - complexity algorithms where O(n) can be used: https://github.com/servicetitan/dataobjects-net/pull/208

alex-kulakov commented 1 month ago

@SergeiPavlov Nice.

niikoo commented 2 weeks ago

Actually we can reduce the amount of time spent in External/Native code by improving callers of that code.

For example: I found O(n²) - complexity algorithms where O(n) can be used: servicetitan#208

I did some testing, building the servicetitan version of DO.net, and setting COMPlus_EnableWriteXorExecute=0 in the environment variables. It resulted in significant performance improvements. Could you @alex-kulakov look into this and see if you can cherry-pick the changes made by @SergeiPavlov?

alex-kulakov commented 1 week ago

@niikoo,

I glanced at the changes Sergei did and proposed, the changes can appear only in master (which is 7.2 in develop). Currently released versions will not receive them.

Outside of this discussion Sergei proposed me covariant returns for cloning of certain classes (e.g SqlNode derived ones), that I successfully applied to 7.1 branch. The changes can't be applied to version older than 7.1 due to target-framework-to-language-version binding.

I will also read about the option you mentioned to better understand the outcome of it. Thank you for this information.