search

ashfurrow / xcode-hardware-performance

Results from running Xcode on a non-trivial open source project using various Macs
Other
1.13k stars 115 forks source link

Add MacBook Pro 15 Retina, Mid 2015 #116

Closed azonov closed 4 years ago

azonov commented 4 years ago

Very Strange behaviour, 1) I can't understand how is it possible that MBP 2015 faster MBP 2016 2) Depends on trolling, did 7 clean builds, so with intel power gadget, after the second one it throttle (iMac 27 do not throttle yesterday at all) 3) Incremental build really fast in 11.3.1 regarding MacBook Pro 16 benchmarks

My results of clean->build 37, 42, 49, 43, 50, 45, 51

azonov commented 4 years ago

The reason to upgrade from my MBP 15 to iMac is throttle, and at large project results are more significant: Clean build 195s - iMac 412s - MBP 2015 370s - MBP 2017 (colleague)

Incremental build 27s - iMac 55s - MBP 2015

So, thank you for the project, it helps me to make right choice!!

P.S. What about fastlane script, that build project without indexing and publishing results?

azonov commented 4 years ago

Thanks for the result! Using Fastlane is a cool idea, for the reasons you point too. While the results would be more consistent, they'd be less representative of real-world use. Any thoughts on how to balance these? So far, I've leaned towards real-world use.

https://github.com/ashfurrow/xcode-hardware-performance/pull/117 Example of the real-world use, compile and run on simulator empty test

ashfurrow commented 4 years ago

Ah, interesting! The thing that I liked about using the Eidolon repo was that it's a full app that's actually used in prod. I think adding a test lane to the Eidolon Fastfile with these changes might give us the best of both worlds – what do you think? I can take a stab at that this weekend.

azonov commented 4 years ago

Sounds cool, still, it's better to make it somehow more complex to build, for the real throttling processor test. It's just an idea of the real comparing case (an example MBP and Mac mini with the same performance could have different results with throttling and with not)

ashfurrow commented 4 years ago

Hey! So I tried getting this working this weekend but ran into an issue: it's difficult to get a measurement of the time it takes to build the app and launch it in the simulator because we need to measure the time we start running xcodebuild to the time the app launches. I'm thinking we could maybe use an environment variable to inject in the current working directory, and then the first thing the app can do when it launches is write out a file with the current timestamp in that directory.

It's kind of complicated, would be difficult, and I'm not sure it'd totally work.

I see #117 showed running the unit tests instead, and I think this makes sense. It is a relative measurement, after all, and this approach would make the measurement more precise. However, I'm worried that it would loose the real-world relevance of build-and-run. Lots of test suites take forever, and build-and-run time is a viscerally more important measurement to iOS developers. So I'm on the fence.

I think that the numbers have gotten low enough (as hardware and software have improved) that this repo could really benefit from some more precision. However, I'm having trouble balancing the desire for precision and the desire for measuring an important use case. I'd love to know what anyone thinks, here.