Memory Usage explosion in parse command in version 0.2.37

MobileNativeFoundation / XCLogParser

Tool to parse Xcode and xcodebuild logs stored in the xcactivitylog format

Apache License 2.0

1.75k stars 125 forks source link

Memory Usage explosion in parse command in version 0.2.37 #196

Open hernancorigliano opened 11 months ago

hernancorigliano commented 11 months ago

Hi, I'd like to report an issue after upgrading from 0.2.36 to 0.2.37.

The parse command started having an issue in which it starts taking up massive amounts of system memory (RAM) indefinitely until the OS collapses.

The issue is not happening for previous versions where the parse command finishes instantly.

I have tried using the mentioned flags --omit_notes --omit_warnings to simplify the parsing of a large log, but this won't stop the issue.

Steps to reproduce:

Build your app in Xcode 15.0.1
Run parse command as xclogparser parse --project MyProject --reporter flatJson --output log.json
The xclogparser process will slowly start taking up all available system memory. (Check the growing amount in Activity Monitor Memory tab)
(Manually kill the process before the OS crashes)

Context:

MacOS: Sonoma 14.1.2
Device: MacbookPro M1
Xcode: 15.0.1
XCLogParser installation: 0.2.37 (Homebrew)

Parabak commented 11 months ago

seyfeddin commented 11 months ago

BalestraPatrick commented 11 months ago

Hello, unfortunately I don't have time to debug exactly what's wrong, but it must be one of the recently added contributions between the last two versions: https://github.com/MobileNativeFoundation/XCLogParser/compare/v0.2.36...v0.2.37

I have a feeling it must be one of these 3 changes:

If you can locally try to revert each of those commits and see which one fixes it, that would be helpful.

Parabak commented 11 months ago

I can confirm that rolling back to version 0.2.36 worked for me

bcylin commented 11 months ago

Tested locally. Reverting #182 avoids the massive RAM usage. But the underlying issue seems to be the BuildStep struct that has too many subSteps.

With SwiftCompile as a detail type, it includes each file's compilation, such as:

SwiftCompile normal arm64 /opt/homebrew/var/agent/builds/ip-123-45-67-890-internal/project/pipeline/.../Sources/.../file.swift (in target '...' from project '...') ... etc

I tried it in a project with ~7,000 Swift files, it adds more than 400,000 sub steps into the parsed result, nested in 5 levels down.

In both use cases of --reporter json and --reporter flatJson, the struct becomes too large that JSONEncoder can't handle. The memory usage shoots up on line 42:

https://github.com/MobileNativeFoundation/XCLogParser/blob/cd41cfc2f910b19b163919c5ab2afc399d7ad7e1/Sources/XCLogParser/reporter/JsonReporter.swift#L39-L44

An alternative workaround is to use --reporter summaryJson as it drops all the sub steps in the SummaryJsonReporter.

BalestraPatrick commented 10 months ago

Reverting in https://github.com/MobileNativeFoundation/XCLogParser/pull/199 and then will cut a new release. Thanks for reporting!

InspektorKek commented 9 months ago

Same problem with JSON, but not only on last version. Actually on each version I face that issue(

Context: MacOS: Sonoma 14.2.1 Device: MacbookPro M3 Pro Xcode: 15.1 XCLogParser installation: rake build v 0.2.36

daltonclaybrook commented 4 months ago

Any update on this? It's unfortunate that SwiftCompile is not supported.