Leverage Spotlight to quickly find dSym avoiding hard-coded search paths

[bblacey]

Invaluable tool and love the fact that you have added support for Sample reports too. I have a suggestion to improve the workflow even more. For non-App Store apps, it is common that a crash report will be attached to a bug reporting or customer care system (as in the case for SuperDuper!) that the developer will download to ~/Downloads for analysis. Currently the dSym search is limited to the location of the crash report file itself or a hard-coded Xcode archive path. In my particular use case, neither works because I download the crash reports to ~/Downloads and my Xcode archives are located in ~/Projects/Archives.

That said, Spotlight indexes all Xcode .dSym files on the system providing an extremely fast way to locate a dSym file for a given uuid. For example, mdfind "com_apple_xcode_dsym_uuids == 50D5553D-FA9A-2541-272F-7C6F43916630" will quickly return the dSym location for 50D5553D-FA9A-2541-272F-7C6F43916630. You can even find all dSym files on the system using mdfind "com_apple_xcode_dsym_uuids == *" as an intelligent enumeration set. I'de suggest that the file search should be extended to intelligently short-circuit the dSym search by using Spotlight first or at least by adding it between the peer-to-crash-report check and the hard-coded Xcode archive enumeration search (this Xcode Archive path should also be a preference).

Thanks for taking the time to develop such an invaluable tool and making it available to the broader community.

[inket]

Thank you for the kind words! The previous version used spotlight to find DSYM files but I had to remove that since it wasn't working with hardened runtime... I think. Hardened runtime is required for notarization.

I'll have to double-check again, but I do distinctly remember Spotlight not giving any results.

[bblacey]

Cool. If MacSymbolicator were running Sandbox'd, then that might require a more complex implementation (i.e. require a helper tool) to perform the search but it seems that it isn't sand-boxed. I know that the hardened runtime will allow the use of Process to run an mdfind command task (MacSymbolicator already uses Process to run dwarfdump) but I think the hardened runtime also allows the use of NSMetaDataQuery, for example:

let uuid = "50D5553D-FA9A-2541-272F-7C6F43916630"

let query = NSMetadataQuery()
query.searchScopes = [NSHomeDirectory()]
query.predicate = NSPredicate(format: "com_apple_xcode_dsym_uuids == %@", uuid)

Anyway, thanks for considering this extension.

[inket]

I'm adding Spotlight search back right now and I think the reason I removed it in the first place is that mdfind just cannot find anything under ~/Library/Developer/ on Catalina. (and therefore it cannot find anything under ~/Library/Developer/Xcode/Archives)

I'm curious, does this command output something for you?

mdfind -onlyin ~/Library/Developer/ "*"

I just want to double check that; I will be adding Spotlight back regardless of the result since there are legitimate uses of it besides that folder.

[bblacey]

Excellent! I am running Catalina but in my case, Xcode is configured to save archives in ~/Projects/_Staging instead of ~/Library/Developer so SpotLight works as expected.

That said, mdfind 'com_apple_xcode_dsym_uuids=*' actually finds two .dSYM archives in ~/Library/Developer so SpotLight seems to search there.

$  mdfind 'com_apple_xcode_dsym_uuids=*
/Users/blacey/Library/Application Support/Autodesk/webdeploy/shared/AcCoreView/N009.D.254/MAC64_MAS/AcCoreView/libAcIPC.dylib.dSYM
/Users/blacey/Library/Application Support/Autodesk/webdeploy/shared/AcCoreView/N009.D.254/MAC64_MAS/AcCoreView/libAcCoreViewProxy.dylib.dSYM

And searching for dSYMs in ~/Library/Developer by name returns the same result.

$ mdfind -onlyin ~/Library "kMDItemFSName == *.dSYM"
/Users/blacey/Library/Application Support/Autodesk/webdeploy/shared/AcCoreView/N009.D.254/MAC64_MAS/AcCoreView/libAcCoreViewProxy.dylib.dSYM
/Users/blacey/Library/Application Support/Autodesk/webdeploy/shared/AcCoreView/N009.D.254/MAC64_MAS/AcCoreView/libAcIPC.dylib.dSYM

Finally, here is an example of SpotLight looking for a UUID on my system.

$ mdfind 'com_apple_xcode_dsym_uuids=606A81A6-74AD-34A9-8FF0-4B0008A955A2'
/Users/blacey/Projects/_Staging_Area/Archives/2019-11-28/SuperDuper-3.3.1 v121.xcarchive

On your system, what happens if you re-import the dSYMs in ~/Library/Developer?

mdimport -r /Applications/Xcode.app/Contents/Library/Spotlight/uuid.mdimporter ~/Library/Developer/*

[inket]

On your system, what happens if you re-import the dSYMs in ~/Library/Developer?

I always get no results for mdfind -onlyin ~/Library/Developer "*" (notice it's not just dSYMs, but any files) which makes me think that something is blocking Spotlight from indexing ~/Library/Developer. ~/Library doesn't have that issue.

Anyway, the fix will be in the app soon!

[bblacey]

What happens if you do mdfind -onlyin ~/Library/Developer "kMDItemFSName = *"?

[inket]

No results

[bblacey]

That's strange. mdfind finds 355K files in ~/Library/Developer on my system.

$ mdfind -onlyin ~/Library/Developer "kMDItemFSName = *" | wc -l
  354915

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.15.4
BuildVersion:   19E224g

[inket]

Ok, that's good to know! It must be an isolated issue on my system. Thanks!!

[inket]

New version was just released - Can you check that it works well?

[bblacey]

Cool! Late here so I will check it out tomorrow. Thanks!

[bblacey]

Ok, some good news and not so good news. The not-so good news is that the first time I drag a crash report onto MacSymbolicator, it doesn't not find the dSYM however, the good news is, if I drag a second crash report onto MacSymbolicator, it finds the .dSYM.

Scenario:

1. Launch MacSymbolicator
2. Drag A.crash file onto panel
   • <A.dSYM NOT located>
3. Drag B.crash file onto panel
   • <B.dSYM located>
4. Drag A.crash file onto panel
   • <A.dsym located>

So, you're very close. Just need to resolve the upon-launch state issue related to first drop/.dSYM search.

Thanks!

[inket]

🤔 are you able to reproduce it?

[bblacey]

Absolutely. In fact, the app gets in a state such that 4-5 doesn't always work. So there is some sort of intermittent issue with searching by SpotLight.

[inket]

Can you build from master and see if using NSMetadataQuery instead of mdfind fixes that issue? https://github.com/inket/MacSymbolicator/commit/5f8a6de5fdcd298d67fc49b3d77e23bc4b8feb2c

I have trouble reproducing these issues so your help is appreciated 🙇 We're all busy with life so please take your time.

[bblacey]

Chuckle, I tried to build / debug this morning but ran into dependency issues here - https://github.com/Swift-Squirrel/Squirrel/issues/39. I also thought it might be that the UI isn't updating but I confirmed that isn't the case because the Symbolicate button doesn't do anything when it can't find the .dSYM (you might consider de-activating/greying out the button until a dSYM is available). I know the feeling where you can't repro an issue that a user has so let me see what I can do about building from master with a bit more time. Thanks again for re-implementing SpotLight.

[inket]

I just realized that the mentioned Squirrel project is not the one that was included in the app 😬 Anyhow, Squirrel was removed in the latest version so the project should be easier to setup.

I will close this issue since it's been a couple of years, but if you do see any problems with the app please feel free to create a new issue!