Closed bgrainger closed 3 years ago
lauxjpn
commented
3 years ago This adds a "direct" implementation of the MySQL protocol that bypasses all of ADO.NET (inspired by some of the discussion on dotnet/datalab#6) to see what's possible with a dedicated "direct" MySQL API (which many other languages/frameworks expose).
@bgrainger Nice! I had this next on my list. Crossing it off now.
bgrainger
commented
3 years ago I now have three separate commits to benchmark independently (for aspcore-mw-mysql):
aspcore-mw-mysql with initial "direct" implementationCLIENT_OPTIONAL_RESULTSET_METADATASELECT queries
roji
commented
3 years ago @bgrainger I've done a bit of infra work and should be able to launch runs for you pretty easily now.
However, can you please tag the commits you want me to run? I'm not able to specify a raw SHA1 with the tooling at the moment (https://github.com/dotnet/crank/issues/311). I also highly recommend that we do a run on the commit before the changes each time, to exclude unrelated perf changes from creeping in (so you'd need to tag that too).
So basically the easiest would be to get a list of tags for running.
/cc @sebastienros @ajcvickers
bgrainger
commented
3 years ago Tags to run in https://github.com/bgrainger/FrameworkBenchmarks:
| tag | test | description |
|---|---|---|
| ado-my-baseline | aspcore-mw-ado-my |
Current MySqlConnector code in TFB (to get a baseline) |
| ado-my-no-prepare | aspcore-mw-ado-my |
Update MySqlConnector; remove MySqlCommand.Prepare |
| mysql-baseline | aspcore-mw-mysql |
Add "direct" MySQL implementation |
| mysql-optional-metadata | aspcore-mw-mysql |
Use CLIENT_OPTIONAL_RESULTSET_METADATA |
| mysql-pipeline | aspcore-mw-mysql |
Pipeline MySQL requests |
roji
commented
3 years ago Here are the results. For aspcore-mw-ado-my, I ran against the URL /fortunes/raw, for aspcore-mw-mysql, against /fortunes/mysql.
| tag | scenario | RPS | description |
|---|---|---|---|
| ado-my-baseline | aspcore-mw-ado-my |
184,181 | Current MySqlConnector code in TFB (to get a baseline) |
| ado-my-no-prepare | aspcore-mw-ado-my |
177,838 | Update MySqlConnector; remove MySqlCommand.Prepare |
| mysql-baseline | aspcore-mw-mysql |
239,639 | Add "direct" MySQL implementation |
| mysql-optional-metadata | aspcore-mw-mysql |
241,393 | Use CLIENT_OPTIONAL_RESULTSET_METADATA |
| mysql-pipeline | aspcore-mw-mysql |
240,899 | Pipeline MySQL requests |
BTW on the aspcore-mw-ado-my runs, CPU on the database server was typically 80-85%. On the aspcore-mw-mysql it's 99%, so you seem to be doing something right :) Note that with PG I've never seen DB CPU go higher than 70%, if memory serves.
PS As we're just starting out with these runs, please be suspicious - it's very easy to accidentally get the command-line wrong or mess up in some other way. Don't hesitate to ask me to re-verify a specific run or introduce sanity checks to make sure things are right, etc.
lauxjpn
commented
3 years ago @roji A couple of questions:
bgrainger
commented
3 years ago That would put aspcore-mw-mysql in 6th place for all MySQL frameworks (for fortunes). Given that aspcore-ado-pg is1.6× faster than aspcore-mw-ado-pg (400,987 vs 244,817), I'm now tempted to fork the PlatformBenchmarks using MySqlDirect and see what the performance is like.
roji
commented
3 years ago What is the .NET sdk/runtime used for these benchmarks (or how is it controlled; making sure, that theses results are comparable to the TFB ones)?
The benchmark application is a docker image, so which .NET is used depends on how that image is configured/built. At least in .NET, TechEmpower generally uses released components, so this should be 5.0.x.
Is there a way to look at the raw benchmark/verification output (because I might use them for additional stats when optimizing)?
I can post everything that comes out of the run, sure.
Is there an URL we can call to see the benchmark result, so you don't have to manually copy / paste them every time?
Unfortunately not... It's all completely internal and there's no outside access... But I can copy-paste out.
roji
commented
3 years ago BTW no need to tag commit hashes any more (thanks @sebastienros).
I'm now tempted to fork the PlatformBenchmarks using MySqlDirect and see what the performance is like.
Why not give it a try :) I'll be happy to run it for you. Am also curious to know where exactly the big changes are with the regular MySqlConnector!
bgrainger
commented
3 years ago Am also curious to know where exactly the big changes are with the regular MySqlConnector!
The major change is that it doesn't use MySqlConnector at all! 😀
MySqlSession is a low-level wrapper for the MySQL protocol that uses Pipelines.Sockets.Unofficial to send/receive data.
MySqlDb uses that low-level API to implement the IDb interface, plus adds a very simplistic connection pool. (And it's really not that much more code than RawDb.cs.)
Some MySqlConnector overhead that's omitted: parsing server handshakes to support backends other than MySQL 8.0; complicated state machines to adapt the MySQL protocol to MySqlDataReader API; creating any ADO.NET objetcs; support for any data types other than INT or VARCHAR; passing data around as objects instead of ReadOnlySpan<byte>.
lauxjpn
commented
3 years ago I would be very interested to see this run against the other benchmark types as well (not just fortunes).
roji
commented
3 years ago That sounds great :) Should we just call it MySqlConnector.Core? :rofl:
So... (As you probably know) I'm interested by how much ADO.NET adds a necessary overhead, i.e. would an ADO.NET shim over this reduce perf and by how much. For example, I'm interested in how MySqlDataReader forces you to do complicated (and bad-for-perf) state machines to adapt the MySQL protocol. But of course, that's not an urgent question or anything, just something I'm interested in.
I can launch runs against other benchmarks too - just give me commits and URLs.
(BTW server handshakes are something that happens only when opening physical connection, no? So with pooling shouldn't be important?)
bgrainger
commented
3 years ago (BTW server handshakes are something that happens only when opening physical connection, no? So with pooling shouldn't be important?)
Correct; the per-connection cost should be extremely low (amortised over all the requests), so it was a bit of a red herring to mention it.
parsing server handshakes to support backends other than MySQL 8.0
But there is if/else code all through MySqlConnector to test if a particular protocol feature is enabled (the biggest one is "deprecate EOF") based on the results of parsing that server handshake. In this driver, I currently ignore all that as I assume MySQL 8.x.
bgrainger
commented
3 years ago @roji I have a "Platform" benchmark (aspcore-mysql) to run at https://github.com/bgrainger/FrameworkBenchmarks/commit/4e6fab769dabce3eab38aadb3348ad88c0e2740c.
Not quite sure if I did this right:
dotnet run --config Z:\projects\AspNetBenchmarks\scenarios\te.benchmarks.yml --scenario fortunes_mysql --profile aspnet-citrine-lin --application.source.repository https://github.com/bgrainger/FrameworkBenchmarks.git --application.source.branchOrCommit #4e6fab769dabce3eab38aadb3348ad88c0e2740c --application.source.dockerFile frameworks/CSharp/aspnetcore/aspcore-mysql.dockerfile --application.source.dockerImageName aspcore-mysql --variable path=/fortunes -i 3
roji
commented
3 years ago @bgrainger there are some technical issues running the above (probably related to the fact that it's platform), I'll be looking into it over the weekend... After we get up and running it should be very easy/fast to send me cmdlines and get the results.
bgrainger
commented
3 years ago @roji I have another submission, for both Benchmarks (aspcore-mw-mysql) and PlatformBenchmarks (aspcore-mysql): https://github.com/bgrainger/FrameworkBenchmarks/commit/97f150543d97a7bd31a3d2f17694a291246d06df. This uses prepared statements for fortunes.
Benchmarks: dotnet run --config Z:\projects\AspNetBenchmarks\scenarios\te.benchmarks.yml --scenario fortunes_mysql --profile aspnet-citrine-lin --application.source.repository https://github.com/bgrainger/FrameworkBenchmarks.git --application.source.branchOrCommit #97f150543d97a7bd31a3d2f17694a291246d06df --application.source.dockerFile frameworks/CSharp/aspnetcore/aspcore-mw-mysql.dockerfile --application.source.dockerImageName aspcore-mw-mysql --variable path=/fortunes/mysql -i 3
PlatformBenchmarks: won't try to guess the full command-line but also --application.source.repository https://github.com/bgrainger/FrameworkBenchmarks.git --application.source.branchOrCommit #97f150543d97a7bd31a3d2f17694a291246d06df
(BTW, would still like to see the PlatformBenchmarks for 4e6fab769dabce3eab38aadb3348ad88c0e2740c as requested in https://github.com/aspnet/Benchmarks/issues/1676#issuecomment-840884063 to compare before/after prepared statements.)
bgrainger
commented
3 years ago mysql-optional-metadata aspcore-mw-mysql 241,393 Use CLIENT_OPTIONAL_RESULTSET_METADATA mysql-pipeline aspcore-mw-mysql 240,899 Pipeline MySQL requests
I just realised that pipelining was only used for "multiple queries", so it would have had no effect on "fortunes" (this benchmark). Thus, this must just be experimental variation (of 0.2%).
bgrainger
commented
3 years ago Another test (for both Benchmarks aspcore-mw-mysql and PlatformBenchmarks aspcore-mysql): https://github.com/bgrainger/FrameworkBenchmarks/commit/7f236842d71208b17a296952a621f1f89a272539.
roji
commented
3 years ago OK, got everything working.
Here are the cmdlines for the platform and non-platform benchmarks:
dotnet run --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/te.benchmarks.yml --scenario fortunes_mysql_aspnet --profile aspnet-citrine-lin --application.source.repository https://github.com/bgrainger/FrameworkBenchmarks.git --application.source.branchOrCommit mysql-baseline#7f236842d71208b17a296952a621f1f89a272539 --application.source.dockerFile frameworks/CSharp/aspnetcore/aspcore-mw-mysql.dockerfile --application.source.dockerImageName aspcore-mw-mysql --variable path=/fortunes/mysql -i 3dotnet run --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/te.benchmarks.yml --scenario fortunes_mysql_aspnet --profile aspnet-citrine-lin --application.source.repository https://github.com/bgrainger/FrameworkBenchmarks.git --application.source.branchOrCommit mysql-direct#7f236842d71208b17a296952a621f1f89a272539 --application.source.dockerFile frameworks/CSharp/aspnetcore/aspcore-mysql.dockerfile --application.source.dockerImageName aspcore-mysql --variable path=/fortunes -i 3Note that you can specify a commit hash, but you must also specify a branch from where it can be reached. The rest probably won't need to change.
| commit | scenario | RPS |
|---|---|---|
| 4e6fab769dabce3eab38aadb3348ad88c0e2740c | aspcore-mysql | 242,071 |
| 97f150543d97a7bd31a3d2f17694a291246d06df | aspcore-mysql | 255,606 |
| 7f236842d71208b17a296952a621f1f89a272539 | aspcore-mysql | 255,933 |
| 97f150543d97a7bd31a3d2f17694a291246d06df | aspcore-mw-mysql | 251,565 |
| 7f236842d71208b17a296952a621f1f89a272539 | aspcore-mw-mysql | 251,812 |
Interesting that platform and non-platform are so close... Note that I haven't looked into the Docker images themselves to see what things look like.
lauxjpn
commented
3 years ago If this would make it into a publicly available (and realistically usable) library without much additional overhead, it would currently be the top ASP.NET Core middleware fortunes benchmark period.
On a second thought, the ASP.NET Core platform benchmarks use very specialized classes, like BenchmarkApplication and AsciiString (and other), to squeeze the last bit of performance out of their custom web request processing code, so it does not seem to be a requirement for the platform tests to only use generally available types (like string) for common operations. Therefore, assuming no double standard, the database access code should not be constraint in that regard as well.
So as far as I can see, we could actually already run the platform benchmarks with this code.
Of course this would not be a fit for the realistic MW and MVC benchmarks, unless it is publicly available (and realistically usable) through a library (as seems to be a requirement for those benchmarks, which also makes a lot of sense).
roji
commented
3 years ago Yeah, the interesting thing is if this level of perf can be maintained while making it into a more-or-less fully-featured driver (e.g. support for transactions, cancellation, whatever). And as I keep harping on, what the perf hit of an ADO.NET shim would be like :wink:
On a second thought, the ASP.NET Core platform benchmarks use very specialized classes, like BenchmarkApplication and AsciiString (and other), to squeeze the last bit of performance out of their custom web request processing code, so it does not seem to be a requirement for the platform tests to only use generally available types (like string) for common operations. Therefore, assuming no double standard, the database access code should not be constraint in that regard as well.
I'm not sure I agree there. Once again, it's very different to write benchmark code that's very minimal and optimized, and to use a web framework or database driver that have been deliberately stripped (or written from the ground up) to not support mainstream "required" features (see discussion in https://github.com/TechEmpower/FrameworkBenchmarks/pull/6566#issuecomment-840461435). At least, as far as I'm aware, the top runners use database drivers which are realistically-usable, "standard" drivers in their language (we may want to verify that). Otherwise everyone would just fling together the minimal code needed to transmit the binary wire representation for TechEmpower Fortunes to their database - that doesn't seem very valuable.
lauxjpn
commented
3 years ago Yeah, the interesting thing is if this level of perf can be maintained while making it into a more-or-less fully-featured driver (e.g. support for transactions, cancellation, whatever). And as I keep harping on, what the perf hit of an ADO.NET shim would be like
Yeah, that is really the hard part to make this publicly available. Though I guess restricting it in some ways might be good, because moving from a general DAL model to a many individual ones is also kind of a troubling development. So keeping it slim might be preferable design goal.
At least, as far as I'm aware, the top runners use database drivers which are realistically-usable, "standard" drivers in their language (we may want to verify that).
So, I checked this before I was writing the post above. The only official thing about it I found once again in the tooltips of the filters:
We classify frameworks as follows:
- Full-stack, meaning a framework that provides wide feature coverage including server-side templates, database connectivity, form processing, and so on.
- Micro, meaning a framework that provides request routing and some simple plumbing.
- Platform, meaning a raw server (not actually a framework at all). Good luck! You're going to need it.
So the Full-stack classification explicitly says, that database connectivity should be part of the framework.
For Micro, its probably not in there anymore (unless it falls in the "simple pumpbing" category).
However, in the Platform classification, database connectivity is definitely not part of the package. You just get a server, and that is it. How you make the benchmarks work, is all up to you (and your luck).
Otherwise everyone would just fling together the minimal code needed to transmit the binary wire representation for TechEmpower Fortunes to their database - that doesn't seem very valuable.
I think this is very valuable. Because we learn the overhead of the ADO.NET provider (or the equivalent for other frameworks), which we would not know otherwise. It is pretty much the maximum in database access performance you can get from an application. We even indirectly learn, how effective the different database systems are in comparison to each other.
We established in https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/issues/1408#issuecomment-840611734, that measuring the difference of the middleware and MVC approaches in contrast to the raw server approach is important.
This "direct" approach does essentially the same on the database level, as the already existing Platform classification benchmarks on the web request level (with the same benefits when it comes to comparing the benchmarks).
In the end, it is the same tradeoff on the database access layer, as on the web request process level. Trading reasonable/sane comfort for custom/insane code and raw performance under the umbrella of the Platform classification.
In fact, I am convinced that this provides so much high quality information that is not available otherwise, that I don't even think it is a question of whether benchmarks like that should be added to the TFB, but just how this should be done:
Conceptionally, the Platform classification is accurate and fits very well.
I think it could be debated, whether the Raw ORM category fits this direct database approach. But even here, I don't really see the benefit of including a specific "raw-er" category, as we do not differentiate by capabilities within the other ORM categories and Raw fits pretty well. The name would contain something to differentiate the benchmark from the ADO.NET ones anyway (e.g. aspcore-direct-my).
From my point of view, the only real discussion here is, whether this can be categorized as realistic, or needs to be categorized as stripped. We talked about this scenario (with ASP.NET Core instead of the DAL) recently in https://github.com/TechEmpower/FrameworkBenchmarks/issues/6585. So arguments or decisions made here about the DAL of the Platform classification benchmarks, should also have an influence on the Platform classifications in general (basically the web request processing part).
I guess my main point here is:
The fact that nobody else has used this approach so far should not matter for whether a new benchmark is viable or not. Instead, us potentially pushing this approach should lead to other framework benchmarks adding a direct approach as well, which will not just make this benchmark comparable to those other frameworks/databases, but also their framework specific default implementation overhead measurable within their own framework boundary.
I think benchmarks like this one provide very valuable information, and will lead to more optimized/performant database providers (not just using this "direct" approach, but really in general), because when the gap is big between the "direct" approach and the default database provider, there will be a natural urge to close it as much as reasonably possible, which is really a gain for everybody.
And since at this point, database systems have a huge impact on RPS, improving database access performance in general will increase throughput, and as far as I understand, this is one of the main goals of the TFB, I don't see how benchmarks like this one are not beneficial to the TFB.
roji
commented
3 years ago Otherwise everyone would just fling together the minimal code needed to transmit the binary wire representation for TechEmpower Fortunes to their database - that doesn't seem very valuable.
I think this is very valuable. Because we learn the overhead of the ADO.NET provider (or the equivalent for other frameworks), which we would not know otherwise. It is pretty much the maximum in database access performance you can get from an application. We even indirectly learn, how effective the different database systems are in comparison to each other.
I think there's some conflation here... It's one thing to write a full-featured, non-ADO.NET driver that supports the basic/standard functionality that's expected from a database driver. I think that's valuable, and indeed could help understand if ADO.NET itself imposes a perf penalty as an abstraction (e.g. by writing a well-optimized ADO.NET shim on top of it). It's quite another thing to slap together a thing that only supports some very basic scenarios - fitted to make TE Fortunes pass - and call that a driver. Aside from maybe comparing MySQL vs. PG (which is explicitly a non-goal of TE), I don't see what anyone learns from it.
(note that I'm not saying anything at all about the current state of MySqlConnector.Core!)
To make this very concrete, let's say you implement a fast, low-level driver that doesn't support cancellation, transactions, or even parameterization. Chances are it's going to be pretty fast, because you've decided not to implement a lot of standard DB functionality. This doesn't tell us anything about ADO.NET as an abstraction, since you can write an ADO.NET driver that doesn't support the above. Maybe, once you add these essential things, you'll be more or less back where you started.
So I'm not arguing about the API - ADO.NET or not. All I'm saying, is that for this to be useful in benchmarking and comparisons, it needs to reach some sort of minimal level of functionality as a database driver, so that we have a rough apples-to-apples comparison.
bgrainger
commented
3 years ago At least, as far as I'm aware, the top runners use database drivers which are realistically-usable, "standard" drivers in their language
This appears to be the case to me. The top MySQL entrants (which are all PHP) seem to use the PHP MySQL PDO API (which maybe wraps the official C client? not sure).
I'm generally with @roji here that (to use an extreme example) a raw TCP sockets implementation that sends the minimal necessary HTTP & MySQL packets that just happens to be written in C# is not a "useful" benchmark to include in TFB. (We may want to run it privately to evaluate the costs of various pieces of the framework, but don't need to upstream it to TFB.)
lauxjpn
commented
3 years ago I think you guys are missing the following points here:
Nobody is expecting users to implement their own DAL for a realistic approach, in the same way that nobody expect users to implement their own web request processing pipeline.
Or to rephrase this, are you not interested to know the performance of the underlying database system and the overhead of the ADO.NET provider? Because you get this information with a benchmark like this one (and I am not even aware of a credible source, that would provide you with similar information, so this is pretty unique out there).
And while it is true, that comparing different database systems to each other is not one of the TFB's primary goals (though in practice, it is definitely used for this), it is to increase awareness for performance issues and to move framework and database developers to remove those if possible. And that is the natural result of benchmarks like this one.
If you see what can theoretically be achieved and what the current state of things is, you will look at ways to improve them, as we currently do by thinking about ways to improve MySqlConnector, e.g. through an auto preparation feature.
Finally, this benchmark gives a good measurement for database level improvements between runs, in contrast to DAL level improvements or web framework improvements. This is not just interesting information in general, but should also further stimulate database developers to improve their systems, since their work can be measured (mostly) independent of the DALs.
roji
commented
3 years ago The value in a benchmark like the one I propose does not primarily come from the benchmark itself. It comes from comparing it to the existing platform benchmarks to get the overhead of the ADO.NET provider. That is true, whether the DAL is based on a fully functional library or not.
If everyone else is using incomplete or stripped-down drivers created purely to show better perf numbers, then maybe. I don't think that's the case; if other platforms use real-world drivers, than we learn nothing from the comparison with them (not apples-to-apples).
I don't see how a compelling argument against this can be made, when I don't even insist that this has to be implemented as a realistic approach (though the reasoning not putting it there would need to be discussed).
Oh, I definitely feel strongly that we can't call anything "realistic" that runs on an incomplete DB driver. Re doing "stripped" scenarios... There's a huge amount of scenarios we could potentially benchmark and publish; the question is what we can learn, and I'm just not seeing that. Otherwise we're just working on something with little value, and creating more results which confuse/obscure more than they reveal. But I don't have super-strong feelings here - as long as there's no interference with the mainstream benchmarks, you can add more if you feel that's important.
Or to rephrase this, are you not interested to know the performance of the underlying database system
Aside from it being a non-goal of the TE benchmarks to measure DB performance (as here, they are simply ill-suited to compare raw database performance; they are full-stack benchmarks which include many moving parts that would make this imprecise in various ways. If I wanted to compare PG and MySQL, I'd use a an extremely minimal client program (definitely no web, 2 machines instead of 3), to isolate the DB as much as possible.
Such tools and benchmarks undoubtedly exist out there.
[...] and the overhead of the ADO.NET provider?
I'm not really interested in knowing the overhead of features I consider necessary, like cancellation, transactions or parameterization, because I consider them standard (again, regardless of the ADO.NET abstraction). I don't consider a DB driver complete without certain things, so why would I be interested in how much perf those things take?
In case it's not clear, I love what @bgrainger is doing - and that's why I'm investing my time in running experiments on our perf lab. I just don't see the point of including an early prototype in the official benchmarks - here - so that they appear in the ranks and give everyone the impression that there's a stable, feature-complete driver they should be using.
I'll take a step back here, since we've been discussing a lot. My main priority here is to keep us from regressing any of our mainstream/official benchmarks; so I indeed want to keep a close eye on changes there. Anything else is really less important for me. I'd rather we avoided producing too much noise through scenarios whose exact meaning and "comparability" with other scenario is unclear; but if you feel strongly that various stripped experiments should be added, and the TechEmpower people think that's valuable, then by all means go ahead.
lauxjpn
commented
3 years ago The value in a benchmark like the one I propose does not primarily come from the benchmark itself. It comes from comparing it to the existing platform benchmarks to get the overhead of the ADO.NET provider. That is true, whether the DAL is based on a fully functional library or not.
If everyone else is using incomplete or stripped-down drivers created purely to show better perf numbers, then maybe. I don't think that's the case; if other platforms use real-world drivers, than we learn nothing from the comparison with them (not apples-to-apples).
We are in full agreement there. But that is not my argument. My argument is, that we learn important information by comparing the aspcore-ado-my benchmark to the aspcore-direct-my benchmark.
the question is what we can learn, and I'm just not seeing that.
From my previous post:
I think this is very valuable. Because we learn the overhead of the ADO.NET provider (or the equivalent for other frameworks), which we would not know otherwise. It is pretty much the maximum in database access performance you can get from an application. We even indirectly learn, how effective the different database systems are in comparison to each other.
Also, keep in mind that the benchmarks are not just about learning (assuming this is one of their goals). They are also explicitly about encouraging performance improvements for all of the involved components. I addressed this above:
I think benchmarks like this one provide very valuable information, and will lead to more optimized/performant database providers (not just using this "direct" approach, but really in general), because when the gap is big between the "direct" approach and the default database provider, there will be a natural urge to close it as much as reasonably possible, which is really a gain for everybody.
And while it is true, that comparing different database systems to each other is not one of the TFB's primary goals (though in practice, it is definitely used for this), it is to increase awareness for performance issues and to move framework and database developers to remove those if possible. And that is the natural result of benchmarks like this one.
If you see what can theoretically be achieved and what the current state of things is, you will look at ways to improve them, as we currently do by thinking about ways to improve MySqlConnector, e.g. through an auto preparation feature.
Aside from it being a non-goal of the TE benchmarks to measure DB performance (as here, they are simply ill-suited to compare raw database performance; they are full-stack benchmarks which include many moving parts that would make this imprecise in various ways. If I wanted to compare PG and MySQL, I'd use a an extremely minimal client program (definitely no web, 2 machines instead of 3), to isolate the DB as much as possible.
I think that is a fair point. However, adding some kind of acceptable baseline will greatly help in measuring improvements to the DAL layer in general. It should also lead to more performance improvements in general (see previous point).
Also, keep in mind that this statement is a counter argument to the one you gave in https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/issues/1408#issuecomment-840118136 (mainly the first two paragraphs).
Such tools and benchmarks undoubtedly exist out there.
I am honestly very interested in those. Something of quality, that continuously measures database performance for realistic scenarios in high concurrency situations, and is open source.
So if you have something concrete here, please drop a link, as this will greatly help us with our general optimization improvements.
I just don't see the point of including an early prototype in the official benchmarks - here - so that they appear in the ranks and give everyone the impression that there's a stable, feature-complete driver they should be using.
So when this is part of the stripped approach, it is not displayed by default, and should not mislead users (also, its description should ideally state its intention).
[...] and the overhead of the ADO.NET provider?
I'm not really interested in knowing the overhead of features I consider necessary, like cancellation, transactions or parameterization, because I consider them standard (again, regardless of the ADO.NET abstraction). [...]
And I think this is probably the main area where our personal interests diverge, since I am quite interested in measuring those in a highly concurrent environment (which I cannot do well locally).
Because if we can indirectly calculate, how much time is spend on average in a particular provider (e.g. a specific ADO.NET provider) in particular scenarios, we can optimize the default provider much more effectively, because we know what is just "database server noise" and what is DAL level overhead (kind of).
I'll take a step back here, since we've been discussing a lot. My main priority here is to keep us from regressing any of our mainstream/official benchmarks; so I indeed want to keep a close eye on changes there. Anything else is really less important for me. I'd rather we avoided producing too much noise through scenarios whose exact meaning and "comparability" with other scenario is unclear [...]
Yeah, I did not really intend to go that much down the rabbit hole. I guess it was not completely unexpected by looking at how it went the last time I posted something of that nature. Though the last one (even though at the time I was just publishing some findings) could have had potential consequences to other tests if actually implemented. This one here really hasn't.
My concrete intention here for us for now was, that we use the experimental work of @bgrainger in a positive way to improve performance for MySqlConnector and thereby for Pomelo. Adding it as a stripped test seems like a simple and effective way of doing so, without having to increase your workload for running benchmarks for us when we make changes in that area, so we know what is database time and what is DAL time (kind of).
[...] but if you feel strongly that various stripped experiments should be added, and the TechEmpower people think that's valuable, then by all means go ahead.
I think there would be no harm in doing this and we could always remove it later, if it proves to be of no help. Adding/removing benchmarks to the TFB seems to be a common practice. And measuring high concurrency scenarios locally is tricky.
bgrainger
commented
3 years ago I think you guys are missing the following points here: It comes from comparing it to the existing platform benchmarks to get the overhead of the ADO.NET provider. ... My argument is, that we learn important information by comparing the
aspcore-ado-mybenchmark to theaspcore-direct-mybenchmark.
That's what I said:
We may want to run it privately to evaluate the costs of various pieces of the framework,
But I'm still thinking:
but don't need to upstream it to TFB.
I'm getting faster turnaround on benchmark results here, without generating unnecessary work for the TFB maintainers to keep merging WIP prototype code.
lauxjpn
commented
3 years ago @bgrainger If you think its not worth it, we are not doing it.
roji
commented
3 years ago Am closing this this there's nothing specifically actionable, but we can keep posting run requests and results here. Another way is to ping me on PRs on MySqlConnector... Basically whatever works.
Please run
aspcore-mw-ado-myandaspcore-mw-mysqlfrom the tip of https://github.com/bgrainger/FrameworkBenchmarks/tree/mysql-direct and report the benchmark results.This adds a "direct" implementation of the MySQL protocol that bypasses all of ADO.NET (inspired by some of the discussion on https://github.com/dotnet/datalab/issues/6) to see what's possible with a dedicated "direct" MySQL API (which many other languages/frameworks expose).
If this is promising, further work could include:
CLIENT_OPTIONAL_RESULTSET_METADATA: https://github.com/aspnet/DataAccessPerformance/issues/34Long-term future work could include packaging this API up into an (experimental?) NuGet package and submitting this as another entrant in TFB (since .NET is at risk of falling out of the top ten for MySQL).