search

seanhenry / SwiftMockGeneratorForXcode

An Xcode extension (plugin) to generate Swift test doubles automatically.
MIT License
748 stars 47 forks source link

Long Spy Generation Time (Both on 11 and 11.1 GM) #29

Closed joshwoods closed 4 years ago

joshwoods commented 4 years ago

We are noticing some long load times on our end when generating spies.

For example:

public protocol LocationProviding {

    /// Call back delegate to receive location update events
    var delegate: LocationProviderDelegate? { get set }

    /// The most current location of the user
    var location: Location? { get }

    var locationIsEnabled: Bool { get }

    var shouldSendToSettings: Bool { get }

    /// Start tracking a user's location
    func start()

    /// Stop tracking a user's location
    func stop()

    func getPlacemarks(for location: CLLocation, completion: @escaping ((Result<[Placemark], Error>) -> Void))

    func authorizationStatus() -> CLAuthorizationStatus

    func requestLocationPermissions()

    func requestLocation()
}

We have a protocol (listed above), this took ~2 minutes to generate whereas before it was pretty instantaneous.

To be fair, LocationProviding is from a different target in our project. I've tried changing out @testable import and just plain ole import of that module in the mock file and neither of those seemed to help. In fact, I think that I had to specifically change the project path to the absolute root path of the project, rather than where the the file actually is (which would encompass both the target LocationProvider is in as well as the target the mock is in).

Any ideas?

Also...would really love to know if you have any reading material how to get involved with these types of improvements, could totally get some additional eyes on helping keep! :)

alexcurylo commented 4 years ago

Don't think it has anything to do with the target, I'm seeing a couple orders of magnitude time increase too on identical protocols with App Store Xcode 11. Happy to help if there's any useful diagnostics I can perform!

seanhenry commented 4 years ago

Hi @joshwoods and @alexcurylo ,

Thanks for raising this issue. I've been able to reproduce it, but not able to find a solution so far. The problem is that source kit seems to get itself into a weird state. This is what Activity Monitor says:

Screenshot 2019-10-01 at 09 33 36

I'm looking into the issue so I'll hopefully get an update out soon.

Sorry for the inconvenience!

seanhenry commented 4 years ago

I've figured out the issue. SourceKit needs a module cache parameter to find the indexed files now - before this was automatic. You can download the latest update here

P.S. If you are interested in getting involved, I'm working on making this 100% open source again to enable people to contribute. I'll post again when that's ready.

Enjoy!

wes-nz commented 4 years ago

Thanks for the quick update @seanhenry!

Unfortunately it hasn't changed the processing time for me. I made sure to check the presence of /Users/Username/Library/Developer/Xcode/DerivedData/ModuleCache.noindex.

I've created a brand new project and tried with @joshwoods example, it took 2 minutes.

If you have any other ideas, I am happy to try 🙏. Thanks a lot for your work.

seanhenry commented 4 years ago

@waleeg - I've just tried the same and it works just fine for me. I've also tried it on large projects and the performance seems fine.

Hopefully we can get this working for you. Could you check the following:

seanhenry commented 4 years ago

@joshwoods does the update work for you?

Kevinvandenhoek commented 4 years ago

@seanhenry I'm also having this issue with XCode Version 11.0 (11A420a).

xcrun --show-sdk-path --sdk macosx responds with /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk

pluginkit -m -p com.apple.dt.Xcode.extension.source-editor -A -D -vvv shows im running version codes.seanhenry.MockGeneratorApp.XcodePlugin(0.22)

The generate spy command seems to take an infinite amount of time for me.

joshwoods commented 4 years ago

@seanhenry I went through and installed the new update and the path is showing correctly, however, it is giving me the "could not find a protocol". I suspect this maybe has to do with the protocol existing in another target as when I go the manual route (moving the parent folder of the entire project), it works, however, still as slow as before. 🤔

wes-nz commented 4 years ago

Have you restarted Xcode before trying the update?

Yep

Do you have automatically detect project path selected in the companion app? And does it show the correct directory when you have your project open in Xcode?

Yep

What version of macOS are you on?

Mojave 10.14.6 (18G95), Xcode 11.0 (11A420a)

What is the response when you run: xcrun --show-sdk-path --sdk macosx?

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk

What is the response when you run: pluginkit -m -p com.apple.dt.Xcode.extension.source-editor -A -D -vvv

+    codes.seanhenry.MockGeneratorApp.XcodePlugin(0.22)
                Path = /Applications/Swift Mock Generator for Xcode.app/Contents/PlugIns/Mock Generator.appex
                UUID = 94B3E12A-6296-4C29-9D47-66E5853B2983
           Timestamp = 2019-10-01 22:02:47 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Swift Mock Generator for Xcode.app
        Display Name = Mock Generator
          Short Name = Mock Generator
         Parent Name = Swift Mock Generator for Xcode

+    com.apple.dt.XCDocumenter.XCDocumenterExtension(1.0)
                Path = /Applications/Xcode.app/Contents/PlugIns/XCDocumenterExtension.appex
                UUID = 317718B9-18BA-4379-AAE2-607786F66506
           Timestamp = 2019-09-24 04:20:49 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Xcode.app
        Display Name = XCDocumenterExtension
          Short Name = XCDocumenterExtension
         Parent Name = Xcode

Thanks

alexcurylo commented 4 years ago

Still seeing it here too after fully cleaning and restarting, stock releases for everything.

Mojave 10.14.6 (18G95), Xcode 11.0 (11A420a) Cache exists at /Users/alex/Library/Developer/Xcode/DerivedData/ModuleCache.noindex SDK /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk Plugins

+    com.apple.dt.XCDocumenter.XCDocumenterExtension(1.0)
                Path = /Applications/Xcode.app/Contents/PlugIns/XCDocumenterExtension.appex
                UUID = 48788E93-D1FF-42FE-9202-7A3696B81240
           Timestamp = 2019-09-20 23:15:54 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Xcode.app
        Display Name = XCDocumenterExtension
          Short Name = XCDocumenterExtension
         Parent Name = Xcode

+    name.lazarenko.daniel.xcsort.xcsortext(1.0)
                Path = /Applications/Xcode Extensions/xcsort.app/Contents/PlugIns/xcsortext.appex
                UUID = 6B58BF5F-F88D-42E9-9A2F-896DA6CF8B08
           Timestamp = 2019-09-04 12:38:59 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Xcode Extensions/xcsort.app
        Display Name = xcsort
          Short Name = xcsort
         Parent Name = xcsort

+    codes.seanhenry.MockGeneratorApp.XcodePlugin(0.22)
                Path = /Applications/Xcode Extensions/Swift Mock Generator for Xcode.app/Contents/PlugIns/Mock Generator.appex
                UUID = D41345C0-D154-4962-B395-8D6131304F0B
           Timestamp = 2019-10-01 21:33:39 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Xcode Extensions/Swift Mock Generator for Xcode.app
        Display Name = Mock Generator
          Short Name = Mock Generator
         Parent Name = Swift Mock Generator for Xcode

+    io.github.norio-nomura.SwiftLintForXcode.SwiftLint(0.1)
                Path = /Applications/Xcode Extensions/SwiftLintForXcode.app/Contents/PlugIns/SwiftLint.appex
                UUID = 0BD5B48E-19C1-4778-A29A-0CD33787CAB8
           Timestamp = 2019-09-04 12:38:53 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Xcode Extensions/SwiftLintForXcode.app
        Display Name = SwiftLint
          Short Name = SwiftLint
         Parent Name = SwiftLintForXcode

+    io.quicktype.quicktype-xcode(8.2.22)
                Path = /Applications/Xcode Extensions/Paste JSON as Code • quicktype.app/Contents/PlugIns/quicktype-xcode.appex
                UUID = 9283576D-DC1D-4B94-829E-EBC5F1B0B933
           Timestamp = 2019-09-04 12:38:54 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Xcode Extensions/Paste JSON as Code • quicktype.app
        Display Name = quicktype
          Short Name = Paste JSON as
         Parent Name = Paste JSON as Code • quicktype

-    com.Swiftify.Xcode.Extension(5.1)
                Path = /Applications/Xcode Extensions/Swiftify for Xcode.app/Contents/PlugIns/Swiftify.appex
                UUID = 56E2872C-3BCA-4EB4-8844-3C70C28C54D9
           Timestamp = 2019-09-13 11:39:22 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Xcode Extensions/Swiftify for Xcode.app
        Display Name = Swiftify
          Short Name = Swiftify
         Parent Name = Swiftify for Xcode
seanhenry commented 4 years ago

I may (or may not) have fixed the issue... I'm not sure why it was working on my machine but I was finally able to reproduce the issue again. I have uploaded a new version but I'm going to hold off from making it an official release until it works for you guys so please do let me know when you've tried it and if it is working or not.

If it still isn't working, I've enabled logging of SourceKit requests so if you attach the log we might be able to understand the problem better.

To get the log, open the Console.app, select your machine and look for Mock Generator or Swift Mock Generator for Xcode. You can also filter by Errors and Faults to narrow down the list.

Screenshot 2019-10-06 at 23 45 38
wes-nz commented 4 years ago

No luck for me, I've checked everything again.

You can find a dump of my logs here. Logs filtered for errors only here.

Thanks!

seanhenry commented 4 years ago

Thanks - could you attach the first 10 lines or so of the request - in your case it was written to: /var/folders/tz/9lwdf4n55wndz_zhcy63qvh80000gn/T/SourceKitLogging-1570412278.log

Could you also check your xcode-select -p and make sure it points to Xcode 11 or 11.1?

wes-nz commented 4 years ago 
request: "-j12",
"-sdk",
"/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk",
"-module-cache-path",
"/Users/W/Library/Developer/Xcode/DerivedData/ModuleCache.noindex",
"-index-store-path",
"/Users/W/Library/Developer/Xcode/DerivedData/MyApp-cfaypvvkfnqmosglaakrwlmaroym/Index/DataStore",
"/Users/W/Sites/MyApp/MyApp/MyAppDataKitTests/MyAppDataKitTests.swift",

xcode-select -p is pointing to the correct Xcode. 🙏

alexcurylo commented 4 years ago

No joy here either and my request looks about the same:

request: "-j8",
"-sdk",
"/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk",
"-module-cache-path",
"/Users/alex/Library/Developer/Xcode/DerivedData/ModuleCache.noindex",
"-index-store-path",
"/Users/alex/Library/Developer/Xcode/DerivedData/MTP-dpcoaujwhiyqhyfvlbotfsxubsgq/Index/DataStore",
"/Volumes/Coding/Trollwerks/projects-active/MTP/github~mtp/MTPUITests/RankingsUITests.swift",
... all the rest of the swift files...
joshwoods commented 4 years ago

Pretty similar on this end:

request: "-j12",
"-sdk",
"/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk",
"-module-cache-path",
"/Users/josh.woods/Library/Developer/Xcode/DerivedData/ModuleCache.noindex",
"-index-store-path",
"/Users/josh.woods/Library/Developer/Xcode/DerivedData/Xyz-addoozozgcomjsgyeryckihqekyu/Index/DataStore",
seanhenry commented 4 years ago

Thanks everyone for your continued testing efforts!

I have another build for your to try. I think the problem has been that I've been largely testing on macOS projects and I'm hoping you are all on iOS projects? In the new build you now need to select a platform from the companion app.

Try the new build out and if it doesn't work I have some further steps to try:

wes-nz commented 4 years ago

iOS: Using @joshwoods example it only took 1 second. I tried with macOS or iOS selected within the app, no difference. 🎉

macOS: No luck here. I made sure to select macOS in the app. The app crashed when the spy was being generated, it took a while to happen. https://pastebin.com/BF1AM6MX

I am wondering if the size of the project has anything to do with it... I am working most of my time on a macOS project, it's heavily relying on protocols.

Thanks for the support!

alexcurylo commented 4 years ago

😢 Doesn't seem to make a difference here, but pluginkit still says (0.23) so maybe it's just having trouble picking up the new build? I'm on my way to Iran so probably won't be able to report more for a couple weeks...

TomVanDerSpek commented 4 years ago

Unfortunately with this new build it still takes about 4 to 5 minutes to generate a spy. Even after following all 6 steps. This new build also didn't automatically detect the root folder of my project, I had to specify it in the companion app.

(Platform: iOS, Xcode 11.0, MacOS 10.14.6)

joshwoods commented 4 years ago

@seanhenry Sorry, I've been AFK for a bit on vacation so I haven't been able to respond. I tried pulling down the download you linked to above on dropbox, however, I don't see any option in that build to select a platform so I'm not entirely sure it's correct. But you would be correct, I am building for iOS!

gonzaloalu commented 4 years ago

I have the same problem. Xcode 11.0 Swift Mock Generator for Xcode 0.22

xcode-select -p
/Applications/Xcode.app/Contents/Developer
pluginkit -m -p com.apple.dt.Xcode.extension.source-editor -A -D -vvv
+    com.apple.dt.XCDocumenter.XCDocumenterExtension(1.0)
                Path = /Applications/Xcode.app/Contents/PlugIns/XCDocumenterExtension.appex
                UUID = 610F8C55-44D2-45AB-BA03-4FDC32EEDA5B
           Timestamp = 2019-10-17 19:58:59 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Xcode.app
        Display Name = XCDocumenterExtension
          Short Name = XCDocumenterExtension
         Parent Name = Xcode11_1

+    codes.seanhenry.MockGeneratorApp.XcodePlugin(0.22)
                Path = /Applications/Swift Mock Generator for Xcode.app/Contents/PlugIns/Mock Generator.appex
                UUID = 8EC74438-6C4D-4296-979F-7396354EA8D2
           Timestamp = 2019-10-17 19:50:36 +0000
                 SDK = com.apple.dt.Xcode.extension.source-editor
       Parent Bundle = /Applications/Swift Mock Generator for Xcode.app
        Display Name = Mock Generator
          Short Name = Mock Generator
         Parent Name = Swift Mock Generator for Xcode

 (2 plug-ins)
gonzaloalu commented 4 years ago

I found a workaround. If you copy and paste the protocol on Playground and generate the mock from there, it'll take only 3 seconds.

seanhenry commented 4 years ago

@gonzaloalu The performance problem we're having is that SourceKit doesn't seem to be finding the index/cache. So copying the files to the playground works because SourceKit is quick with or without an index when there are only a few files. I'm glad that's working as a workaround for now though.

@joshwoods Not sure why the build was incorrect for you but it doesn't seem to work for anyone anyway 😆

I have a new update to try. Just make sure that the companion app is pointing to your project directory - don't worry about the platform selector for this one.

TomVanDerSpek commented 4 years ago

@seanhenry, seems to be fixed in this build! Generates complex spy's in less than a second. 🎉Thanks!

joshwoods commented 4 years ago

@seanhenry so I'm definitely seeing some go a little bit faster than they were. I am sorta thinking that part of my issue now is the way this specific project is setup. For example, I setup a completely brand new project and made a protocol and a mock was created almost instantly.

However, using the LocationProviding example that I originally posted...LocationProviding is a public protocol that exists in one of our external frameworks. I tried using @testable import and import of said framework into the test file and both times it told me that it couldn't find a class or protocol for my mock. 😢 If it means anything, when I look at the logs, it doesn't look like the file (LocationProviding.swift) shows up as being imported from the framework. Is that normal?

Also...for what it's worth, I just went and tried to make a mock of this protocol:

protocol UINavigationControllerInterface: AnyObject {
    var viewControllers: [UIViewController] { get set }
    var topViewController: UIViewController? { get }
    @discardableResult func popViewController(animated: Bool) -> UIViewController?
    func pushViewController(_ viewController: UIViewController, animated: Bool)
    func present(_ viewControllerToPresent: UIViewController, animated flag: Bool, completion: (() -> Void)?)
    @discardableResult func popToRootViewController(animated: Bool) -> [UIViewController]?
}

It exists in our main target so I wouldn't expect any of the other issues that I indicated above, but it took 2.5 minutes to generate the mock.

For what it's worth, from the time that it started, I was monitoring the console for Mock Generator errors and I saw 5 logs written. Is that expected?

wes-nz commented 4 years ago

@seanhenry 💯 it's fixed and faster than ever! It's instant. Thank you very much.

@joshwoods It seems to be project dependent, I just tried your example (UINavigationControllerInterface) and it was instant.

seanhenry commented 4 years ago

@waleeg @TomVanDerSpek That's great news! I'm pleased it's working.

@joshwoods I'm pretty sure you're not using the latest version of the app. I have noticed that Xcode can get itself in a weird state and continue to reference old plugins in certain situations. The reason I think this is because there shouldn't be any logs generated in the new build, and it should also fail quite quickly now, even if it can't find your referenced protocol/class. Either that, or there is another unrelated issue to the SourceKit issue.

Can you try closing Xcode, deleting the current app, emptying the trash, restarting your machine and then copying the new app into /Applications again? I've had to do this once or twice to get Xcode back in line. If after this it still takes over 10 seconds to succeed/fail, then let me know and we'll figure this out.

However, using the LocationProviding example that I originally posted...LocationProviding is a public protocol that exists in one of our external frameworks.

Unfortunately, it won't find classes or protocols that do not reside in your project directory. There is another open issue for that: https://github.com/seanhenry/SwiftMockGeneratorForXcode/issues/19. I am actively working on this, but it's not a quick fix so I will post again in that issue when it gets resolved.

A workaround for this issue is to manually specify the path to your framework project directory in the companion app. It's not ideal, I know, but any source files found in the specified directory should get resolved. You could even arrange your two projects to be side-by-side in the file system and manually specify the parent directory - this would stop you from switching between directories in the companion app. But be careful - identically named protocols or classes are not supported and will result in one being picked at random so be sure not to include other projects in that directory.

Thanks for your patience with all these workarounds - 3rd party development tools aren't quite ready to be used in the Apple ecosystem yet!

joshwoods commented 4 years ago

well I'll be damned I think you're right! I went through the process of deleting, emptying trash, restarting and reinstalling and we're in business! thank you so much!

I even moved the project path all the way to the root (so that it'd include the base target as well as all dependencies) and that seems to work on those ones I was having issues with where it couldn't find the protocol!

Thanks so much for all of your work on this!!

seanhenry commented 4 years ago

@joshwoods No worries - I'm really pleased it's working for you!

seanhenry commented 4 years ago

I've created a new update which adds some other minor improvements to the build that I posted here, so be sure to install the new version ASAP (the version number of the new release is still 0.23).