Adding Startup Time Analysis Capability

mackoj commented 6 months ago

Hello @marinofelipe,

I'm reaching out to propose an enhancement that I believe could open the door to a plethora of new valuable metrics.

I aim to enable users to extract the startup time in order to compare with or without the SDK. The proposed functionality would involve executing the application both with and without the SDK multiple times, capturing the startup times for each iteration. Subsequently, the average startup time for each scenario would be calculated, allowing users to compare the performance impact of integrating the SDK.

The output could be presented as either the percentage increase in startup time or the absolute time difference. I am open to discussing the most appropriate format based on the project's requirements and user preferences.

Additionally, implementing this feature would pave the way for extracting more data from MetricKit in the future.

But implementing this will require MeasurementApp to run on device.

Ressources

I would appreciate your thoughts and feedback.

The proof of concept have generated this.

{
  "applicationLaunchMetrics" : {
    "average" : 1867.1428571428571
  }
}

Thank you

marinofelipe commented 6 months ago

Hi @mackoj , first of all, I'm sorry for the late reply, the last weeks were quite busy for me.

Thanks for the suggestion, it's an interesting idea and MetrickKit really gives tons of metrics we could explore.

General thoughts

Pre-main

Depends on the linking type, dynamic linking will add up to the start up, while static linking happens mostly in build time and adds up to build times
For example, in the company I work for, we were able to drastically reduce the start up time by "switching" to static linking
SPM defaults to static linking, especially for targets, while libraries can be exposed as dynamic typed

Post-main

Many times, an SDK will add up to the startup time only when it's "initiated" via code, which may differ depending on the capabilities available
For example, I remember Firebase being one of the biggest offenders, and that was triggered by a FirebaseApp.init (or so) call

Complexities

Like you said, more accurate on devices, and generally with release optimizations on

Questions

Could you share more about the proof of concept? What linking did you use? Did you just link the package or also "initialized it"?
Are you thinking of covering post-main as well? If so, how?
One thing I think we have to consider is how valuable it would be versus just using Instruments, or MetricKit in your own app So far, I think it might still be valuable to get a quick idea of the potential impact, especially on pre-main and for those relying on dynamic linking

marinofelipe commented 6 months ago

Some other thoughts

Pre-main can also be affected by other things, like static initialisation, Objective-C +load, etc.. but from my experience those are not so relevant nowadays with most of the packages in pure Swift
While MetricKit is great, I believe it might not be the best fit for a static analyzes like this project does, meaning, most of the metrics available, like memory usage would be more accurate if evaluated against the users app binary

Curious to hear your thoughts :), even thought there are things to consider, we may still be able to find a valuable path

mackoj commented 5 months ago

Hi @marinofelipe sorry for the delayed response.

Could you share more about the proof of concept? What linking did you use? Did you just link the package or also "initialized it"?

At the moment, it's quite basic. I've added a basic MetricKit implementation in the MeasurementApp that writes the data to a specific path. I haven't initialized it; I just want to see how it performs from a linking point of view. I would love to have startup impact information, similar to what's discussed in this article: https://www.emergetools.com/blog/posts/comparing-top-analytics-sdks-for-ios.

Are you thinking of covering post-main as well? If so, how?

If MetricKit can support it I will have this data too. I will need to look more in depth what is available.

One thing I think we have to consider is how valuable it would be versus just using Instruments, or MetricKit in your own app

So far, I believe it could still be valuable to quickly gauge the potential impact, especially on pre-main and for comparing dynamic linking versus static or mergeable libraries(https://www.emergetools.com/blog/posts/make-your-ios-app-smaller-with-dynamic-frameworks / https://www.humancode.us/2024/01/02/all-about-mergeable-libraries.html).

For me, the goal is to compare SDKs before implementation, and being able to track the evolution of the SDKs I ship based on these metrics can be really helpful.

Pre-main can also be affected by other things, like static initialisation, Objective-C +load, etc.. but from my experience those are not so relevant nowadays with most of the packages in pure Swift

True

While MetricKit is great, I believe it might not be the best fit for a static analyzes like this project does, meaning, most of the metrics available, like memory usage would be more accurate if evaluated against the users app binary

I don't think all MetricKit data is relevant or even accessible for a tool like swift-package-info. However, some of them can be really helpful and maybe leveraged in a more static fashion. I do agree that it doesn't align with this project's direction, but I don't think we can't leverage it to our advantage.

marinofelipe commented 3 months ago

Hi @mackoj , I'm sorry for taking so long to answer but I didn't manage to focus on Open Source the past couple weeks.

At the moment, it's quite basic. I've added a basic MetricKit implementation in the MeasurementApp that writes the data to a specific path. I haven't initialized it; I just want to see how it performs from a linking point of view. I would love to have startup impact information, similar to what's discussed in this article: https://www.emergetools.com/blog/posts/comparing-top-analytics-sdks-for-ios.

Nice, let me know how far you've got. It definitely make sense from a linking point of view. It's very cool what EmergeTools does, I believe they ask for an ipa and then do their analyzes on top, which would be then very accurate post-main too.

If MetricKit can support it I will have this data too. I will need to look more in depth what is available.

From what I recall, MetricKit has the total launch time duration, see here, which is a a very good starting point.

For a detailed pre and post-main I found the App Launch Instruments a great option, but for our use case it would further complicate things and significantly extend the swift-package-info runtime.

So far, I believe it could still be valuable to quickly gauge the potential impact, especially on pre-main and for comparing dynamic linking versus static or mergeable libraries(https://www.emergetools.com/blog/posts/make-your-ios-app-smaller-with-dynamic-frameworks / https://www.humancode.us/2024/01/02/all-about-mergeable-libraries.html).

For me, the goal is to compare SDKs before implementation, and being able to track the evolution of the SDKs I ship based on these metrics can be really helpful.

I don't think all MetricKit data is relevant or even accessible for a tool like swift-package-info. However, some of them can be really helpful and maybe leveraged in a more static fashion. I do agree that it doesn't align with this project's direction, but I don't think we can't leverage it to our advantage.

Agreed. Let's see what we can have, a simple static analyzes of linking is already a great feature 😄

Again, in case you made progress, let me know how it went. I've done one thing or two around launch times and measuring app performance at work, especially when we further scaled and modularized our app, so I'd be happy to collaborate with you on this one.

marinofelipe / swift-package-info