TechEmpower / FrameworkBenchmarks

Source for the TechEmpower Framework Benchmarks project
https://www.techempower.com/benchmarks/
Other
7.56k stars 1.94k forks source link

Add SqlServer on Linux DB? #3786

Open benaadams opened 6 years ago

benaadams commented 6 years ago

Could a SqlServer DB be added?

Based on docker FROM microsoft/mssql-server-linux:latest

bhauer commented 6 years ago

Just out of curiosity, what is licensing like for SQL Server on Linux?

The free Express edition would not be the right fit since Express is limited to 4 CPU cores. I suspect we'd want to use the Developer edition.

benaadams commented 6 years ago

Same as Windows I believe; SQL Server 2017 brought parity between the editions in base db features/programmability (whereas previously whether you could use memory optimised, column store or data-compression was based on the edition - so Express was more a SQL Server Lite)

At SQL Server 2017, Express is now fully featured from a base db perspective; with editions upping core, memory, db size limits as well as adding "bigger database" features: scaling (failovers, clustering etc), analytics, auditing etc - so using Developer edition (which is Enterprise), should still be a valid representation of Express.

Going from the pricing page; Developer and Express are both free; with the caveat that Developer can only be used for build, test and demonstration - but not production.

So I think it would be correct to use Developer edition for the benchmark tests.

bhauer commented 6 years ago

@benaadams We were just discussing this internally. We believe that Citrine qualifies as a "test" environment rather than a "production" environment for the purposes of licensing. Is it possible to get anyone at Microsoft who knows the SQL Server license model well to confirm this for us? We just don't want any bad surprises.

bhauer commented 6 years ago

We will also need to contend with clause 6 from the license terms:

  1. BENCHMARK TESTING. You must obtain Microsoft's prior written approval to disclose to a third party the results of any benchmark test of the software.
benaadams commented 6 years ago

We will also need to contend with clause 6 from the license terms:

BENCHMARK TESTING. You must obtain Microsoft's prior written approval to disclose to a third party the results of any benchmark test of the software.

Oh wow, that's definitely an issue... @DamianEdwards any idea how to proceed?

Also earlier one on licencing:

We believe that Citrine qualifies as a "test" environment rather than a "production" environment for the purposes of licensing. Is it possible to get anyone at Microsoft who knows the SQL Server license model well to confirm this for us? We just don't want any bad surprises.

bhauer commented 6 years ago

Does anyone watching this conversation know who we could contact on the SQL Server team to get the necessary waivers or permission to proceed?

jamiere-msft commented 6 years ago

Hi. I can handle getting the data reviewed and provide written permission post-review. I do this for all of our TPC benchmark results as well as our Data Warehouse Fast Track program.

michaelhixson commented 6 years ago

@jamiere-msft We'd like to publish results continuously without anyone needing to review each individual set of results. Currently, we publish results automatically at https://tfb-status.techempower.com/ and manually at https://www.techempower.com/benchmarks/. The amount of SQL Server-related data in our results would increase over time, because any contributor to this project would be able to write a test that pairs SQL Server with some web framework of their choosing.

Is this model of benchmarking compatible with SQL Server's licensing?

jamiere-msft commented 6 years ago

Thanks Michael. I need to run this by a few people here to see if we could do a global permission for data coming from this workload.

mi-hol commented 6 years ago

@jamiere-msft did you have any luck with this question yet? Corporate decisions are often influenced by creating potential bad headlines as a test case ;-)

jamiere-msft commented 6 years ago

@mi-hol Yes and no. I reached out to @benaadams to see if he could share with us some preliminary data points to help our argument. Once we get a data point or two we can continue the internal discussions.

mi-hol commented 5 years ago

@jamiere-msft @benaadams I wonder if there might be news to share?

Wraith2 commented 5 years ago

These benches would be handy for perf comparisons on work i've done. Would they have to use linux builds? the linux perf of SqlClient lags behind native

salarzobir commented 5 years ago

@benaadams any news about SqlServer?

msmith-techempower commented 5 years ago

@Wraith2 Currently, because of a bug in Docker4Win, the toolset is limited to Linux-only. This will, presumably, be corrected at some point.

bruno-garcia commented 5 years ago

I'd be very curious to see how it performs with MS SQL Server. Anything the community can do to help here?

Wraith2 commented 5 years ago

no, waiting on Microsoft.

Wraith2 commented 4 years ago

Still waiting on the MS side of things, an update would be useful given the performance work that i've put into Microsoft.Data.SqlClient

Wraith2 commented 4 years ago

We're coming up to the 2 years anniversary of this thread and we're still waiting on movement from the Microsoft side on the licensing question. Given that the TE benchmarks are being used as optimization targets for much of aspnet core it'd be really good to get some benchmarks for the DB in there as well.

Wraith2 commented 3 years ago

Another 6 months. Given how often techempower was mentioned a .netconf it's still a big omission that sql server isn't present in the benches. It gives the impression of a lack of confidence in the product.

Wraith2 commented 3 years ago

Another round missed, https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune Will it make 3 years without anything happening? Looks likely.

Wraith2 commented 3 years ago

3 Years old today.

Wraith2 commented 2 years ago

4 Years old today! 🎉

Still waiting on Microsoft.

Wraith2 commented 1 year ago

5 Years old today!

Still waiting on Microsoft. It is possible that the request has been lost internally. Does anyone know who to contact who is currently in a position to do something about this?

Wraith2 commented 3 months ago

6 Years old today.

Still waiting on Microsoft.