Adding support for external source, e.g. Foundation or 3rd Party libraries

[swizzlr]

Feature request: access Foundation types, standard library types, and even third party package types within stencils. This would make it a lot easier to generate boilerplate extensions. My current workaround is copy/pasting the swift interface to a file in the folder but not adding it to Xcode (hacky but works).

Not sure how we'd go about achieving this, could be a limit of SourceKit – maybe we could replicate the copy-paste hack by specifying a target, generating the interface, writing it to a temporary file and pointing SourceKit's type parser at it (which seems to happen at the Sema phase, pre-AST).

[krzysztofzablocki]

We could introduce teaching phase to Sourcery, e.g. feed it foundation once and keep that data around, the problem with caching is invalidation, so we'd need to be careful when we require recaching those informations

[ilyapuchka]

@krzysztofzablocki do you mean running Sourcery against swift libs source code somewhere locally, dumping result into file and bundling it with a binary to load it at runtime?

[krzysztofzablocki]

that might be prefered for swift stdlib, but won't work for 3rd party so if we want to support 3rd party then we'd need to integrate more advanced caching

[swizzlr]

Ideally SourceKitten/SourceKit can access Foundation/stdlib types natively. @jpsim is that possible?

[jpsim]

Ideally SourceKitten/SourceKit can access Foundation/stdlib types natively

SourceKitten can get data relevant to Foundation/stdlib just fine. What info would you like out of them? Information regarding source parsing obviously won't be available for compiled modules, however.

[swizzlr]

@jpsim listing of all types + their publicly available methods, variables, etc. The stuff you see in the generated interface.

@krzysztofzablocki currently sourcery pulls this info from a given set of source files somehow; what sort of data is that? Perhaps @jpsim could advise on how we can retrieve similar data from the stdlib and Foundation?

[krzysztofzablocki]

we just scan source code, this isn't the issue.

The question we need to figure out first is how often we need to invalidate those informations from stdlib / foundation, because we should not be scanning this each time, the amount of code we might have in Swift / Foundation would add overhead to each run and that's not needed because it shouldn't change

[swizzlr]

I'm assuming that object code exports a listing of available types to declare its interface – it's that info I think we ought to access.

[bolismauro]

+1 for this

I'm defining some protocol conformance for some types (e.g, array) in a library. In my stencils I can't do something like if variable.type.implements.MyProtocol since, when the type is an array or any other type is not defined in the project, type is empty.

[ilyapuchka]

@krzysztofzablocki should we allow not only provide the path for source files in cli, but also provide a path to Xcode project and list of targets to scan (scanning all targets by default maybe)? I think it may be better than just scanning all source files in provided path as some of them can not be even linked to the same target or to any target at all.

[krzysztofzablocki]

I think an option to pick a workspace / proj instead of directory is something we should do in the future(post 1.0?), but that's a different concern to this issue, we should create a new enchancement task for that.

[krzysztofzablocki]

We can look at this issue after I'm finished with #97 and #96 as I think they will be very useful here

[ilyapuchka]

The issue is partially solved now with #193. To support external binaries (i.e. Foundation) we can have an option to provide a path to Sourcery cache for this binary and load it before scanning other sources. We can ship Sourcery with Foundation / stdlib cache (created from a particular Swift release).

[Joannis]

@ilyapuchka I've tried letting it scan for other sources using --sources ./Packages/MyDependency/Sources/ --sources ./Sources/ but it doesn't recognize the libraries' types. You mentioned in multiple places that it's possible to let sourcery cache these libraries and then point Sourcery on the project to also use the caches for it's dependency/ies but couldn't find anything about that in the source code. I'm running the latest version on the master branch.

Also; @krzysztofzablocki do you have any plans to release a new version of Sourcery any time soon? I'm building a neat project with @Obbut that he's mentioned before, something like a technically complex practically transparent persistence framework that is largely making use of Sourcery.

[krzysztofzablocki]

@Joannis I'll make a new release when we merge all remaining PR's

[ilyapuchka]

@Joannis I think when using CLI sources path parameter got overridden by the latest value. Try using config file. I will align implementation with documentation, sorry for confusion ^^

@krzysztofzablocki I think we better have this in the release, I'll fix it in the evening, or you can go for it if you have time, ok?

[Joannis]

@ilyapuchka I tried specifying 2 sources in the config file and it does detect the dependency now. But it still doesn't recognize the library's File struct that's being used inside the application.

EDIT: I'm trying to iterate over a Model, the model contains a set of variables such as "username" which are working fine. One of these variables is a File that's defined in a library. File is a generic struct, one generic is used for the Storage method and the other is a struct (with static variables) defining the requirements of the file, such as PNG and <1MB.

So whenever I try to access the variable.type it's not detected, so I cannot access the generics.

[ilyapuchka]

Hm, I guess it's not related to scanning sources. Can you come up with a minimal example that reproduces this failure and create an issue for that? That will really help.

[ilyapuchka]

Just checked CLI --sources parameter, works as expected, so issue is in something else.

[Joannis]

The CLI has been failing for me but the config file worked fine. With the config file I found out that sourcery doesn't recognize a struct/class with specified generics.

struct Wrapper<Type> {
  ...
}

let thingy: Wrapper works for Sourcery although the compiler doesn't compile it, as it shouldn't

let thingy: Wrapper<String> works for the compiler, but sourcery doesn't recognize this.

[ilyapuchka]

Yes, Sourcery does not resolve generic types, so type property of this variable will be nil. But you still will have typeName available.

[Joannis]

Ah, now it makes sense. I assumed the type would be Wrapper but the generics wouldn't be accessible.

[Sajjon]

👍 for this! :D

I need to access all UIKit types (UILabel, UIButton...) and get all their properties that has setters and their types. How would I go about doing so?

@ilyapuchka @krzysztofzablocki I did not understand how I should use targets, https://github.com/krzysztofzablocki/Sourcery/pull/193 , to scan UIKit types? Is it possible somehow in the current release 0.7.2?

[ilyapuchka]

@Sajjon this is not supported and I'm afraid will be not possible to do for UIKit, as we will need to scan its sources (which are objc btw, so "no" right away, Sourcery can scan only swift sources) but they are not available.

[Sajjon]

@ilyapuchka What about scanning the Swift interface for UIKit types, as it sounded that @swizzlr suggested? When CMD clicking on e.g. UITableView we see this interface:

Swift Interface of ObjC code for `UITableView`

```swift @available(iOS 2.0, *) open class UITableView : UIScrollView, NSCoding { public init(frame: CGRect, style: UITableViewStyle) // must specify style at creation. -initWithFrame: calls this with UITableViewStylePlain public init?(coder aDecoder: NSCoder) open var style: UITableViewStyle { get } weak open var dataSource: UITableViewDataSource? weak open var delegate: UITableViewDelegate? @available(iOS 10.0, *) weak open var prefetchDataSource: UITableViewDataSourcePrefetching? open var rowHeight: CGFloat // will return the default value if unset open var sectionHeaderHeight: CGFloat // will return the default value if unset open var sectionFooterHeight: CGFloat // will return the default value if unset @available(iOS 7.0, *) open var estimatedRowHeight: CGFloat // default is 0, which means there is no estimate @available(iOS 7.0, *) open var estimatedSectionHeaderHeight: CGFloat // default is 0, which means there is no estimate @available(iOS 7.0, *) open var estimatedSectionFooterHeight: CGFloat // default is 0, which means there is no estimate @available(iOS 7.0, *) open var separatorInset: UIEdgeInsets // allows customization of the frame of cell separators @available(iOS 3.2, *) open var backgroundView: UIView? // the background view will be automatically resized to track the size of the table view. this will be placed as a subview of the table view behind all cells and headers/footers. default may be non-nil for some devices. // Data open func reloadData() // reloads everything from scratch. redisplays visible rows. because we only keep info about visible rows, this is cheap. will adjust offset if table shrinks @available(iOS 3.0, *) open func reloadSectionIndexTitles() // reloads the index bar. // Info open var numberOfSections: Int { get } open func numberOfRows(inSection section: Int) -> Int open func rect(forSection section: Int) -> CGRect // includes header, footer and all rows open func rectForHeader(inSection section: Int) -> CGRect open func rectForFooter(inSection section: Int) -> CGRect open func rectForRow(at indexPath: IndexPath) -> CGRect open func indexPathForRow(at point: CGPoint) -> IndexPath? // returns nil if point is outside of any row in the table open func indexPath(for cell: UITableViewCell) -> IndexPath? // returns nil if cell is not visible open func indexPathsForRows(in rect: CGRect) -> [IndexPath]? // returns nil if rect not valid open func cellForRow(at indexPath: IndexPath) -> UITableViewCell? // returns nil if cell is not visible or index path is out of range open var visibleCells: [UITableViewCell] { get } open var indexPathsForVisibleRows: [IndexPath]? { get } @available(iOS 6.0, *) open func headerView(forSection section: Int) -> UITableViewHeaderFooterView? @available(iOS 6.0, *) open func footerView(forSection section: Int) -> UITableViewHeaderFooterView? open func scrollToRow(at indexPath: IndexPath, at scrollPosition: UITableViewScrollPosition, animated: Bool) open func scrollToNearestSelectedRow(at scrollPosition: UITableViewScrollPosition, animated: Bool) // Row insertion/deletion/reloading. open func beginUpdates() // allow multiple insert/delete of rows and sections to be animated simultaneously. Nestable open func endUpdates() // only call insert/delete/reload calls or change the editing state inside an update block. otherwise things like row count, etc. may be invalid. open func insertSections(_ sections: IndexSet, with animation: UITableViewRowAnimation) open func deleteSections(_ sections: IndexSet, with animation: UITableViewRowAnimation) @available(iOS 3.0, *) open func reloadSections(_ sections: IndexSet, with animation: UITableViewRowAnimation) @available(iOS 5.0, *) open func moveSection(_ section: Int, toSection newSection: Int) open func insertRows(at indexPaths: [IndexPath], with animation: UITableViewRowAnimation) open func deleteRows(at indexPaths: [IndexPath], with animation: UITableViewRowAnimation) @available(iOS 3.0, *) open func reloadRows(at indexPaths: [IndexPath], with animation: UITableViewRowAnimation) @available(iOS 5.0, *) open func moveRow(at indexPath: IndexPath, to newIndexPath: IndexPath) // Editing. When set, rows show insert/delete/reorder controls based on data source queries open var isEditing: Bool // default is NO. setting is not animated. open func setEditing(_ editing: Bool, animated: Bool) @available(iOS 3.0, *) open var allowsSelection: Bool // default is YES. Controls whether rows can be selected when not in editing mode open var allowsSelectionDuringEditing: Bool // default is NO. Controls whether rows can be selected when in editing mode @available(iOS 5.0, *) open var allowsMultipleSelection: Bool // default is NO. Controls whether multiple rows can be selected simultaneously @available(iOS 5.0, *) open var allowsMultipleSelectionDuringEditing: Bool // default is NO. Controls whether multiple rows can be selected simultaneously in editing mode // Selection open var indexPathForSelectedRow: IndexPath? { get } // returns nil or index path representing section and row of selection. @available(iOS 5.0, *) open var indexPathsForSelectedRows: [IndexPath]? { get } // returns nil or a set of index paths representing the sections and rows of the selection. // Selects and deselects rows. These methods will not call the delegate methods (-tableView:willSelectRowAtIndexPath: or tableView:didSelectRowAtIndexPath:), nor will it send out a notification. open func selectRow(at indexPath: IndexPath?, animated: Bool, scrollPosition: UITableViewScrollPosition) open func deselectRow(at indexPath: IndexPath, animated: Bool) // Appearance open var sectionIndexMinimumDisplayRowCount: Int // show special section index list on right when row count reaches this value. default is 0 @available(iOS 6.0, *) open var sectionIndexColor: UIColor? // color used for text of the section index @available(iOS 7.0, *) open var sectionIndexBackgroundColor: UIColor? // the background color of the section index while not being touched @available(iOS 6.0, *) open var sectionIndexTrackingBackgroundColor: UIColor? // the background color of the section index while it is being touched open var separatorStyle: UITableViewCellSeparatorStyle // default is UITableViewCellSeparatorStyleSingleLine open var separatorColor: UIColor? // default is the standard separator gray @available(iOS 8.0, *) @NSCopying open var separatorEffect: UIVisualEffect? // effect to apply to table separators @available(iOS 9.0, *) open var cellLayoutMarginsFollowReadableWidth: Bool // if cell margins are derived from the width of the readableContentGuide. open var tableHeaderView: UIView? // accessory view for above row content. default is nil. not to be confused with section header open var tableFooterView: UIView? // accessory view below content. default is nil. not to be confused with section footer open func dequeueReusableCell(withIdentifier identifier: String) -> UITableViewCell? // Used by the delegate to acquire an already allocated cell, in lieu of allocating a new one. @available(iOS 6.0, *) open func dequeueReusableCell(withIdentifier identifier: String, for indexPath: IndexPath) -> UITableViewCell // newer dequeue method guarantees a cell is returned and resized properly, assuming identifier is registered @available(iOS 6.0, *) open func dequeueReusableHeaderFooterView(withIdentifier identifier: String) -> UITableViewHeaderFooterView? // like dequeueReusableCellWithIdentifier:, but for headers/footers // Beginning in iOS 6, clients can register a nib or class for each cell. // If all reuse identifiers are registered, use the newer -dequeueReusableCellWithIdentifier:forIndexPath: to guarantee that a cell instance is returned. // Instances returned from the new dequeue method will also be properly sized when they are returned. @available(iOS 5.0, *) open func register(_ nib: UINib?, forCellReuseIdentifier identifier: String) @available(iOS 6.0, *) open func register(_ cellClass: Swift.AnyClass?, forCellReuseIdentifier identifier: String) @available(iOS 6.0, *) open func register(_ nib: UINib?, forHeaderFooterViewReuseIdentifier identifier: String) @available(iOS 6.0, *) open func register(_ aClass: Swift.AnyClass?, forHeaderFooterViewReuseIdentifier identifier: String) // Focus @available(iOS 9.0, *) open var remembersLastFocusedIndexPath: Bool // defaults to NO. If YES, when focusing on a table view the last focused index path is focused automatically. If the table view has never been focused, then the preferred focused index path is used. } ```

Is this Swift Interface generated by Xcode only when I CMD click? Surely it must be possible to find this interface somewhere else? :) Maybe Sourcery can access this Interface somehow? :)

[ilyapuchka]

I'm not sure how to access those headers out of Xcode.

[jpsim]

Please don't parse the autogenerated Swift interfaces out of Xcode. It's error prone and you'd lose out on a lot of semantic information.

Instead, you could follow an approach similar to the one I described in https://github.com/jpsim/SourceKitten/issues/405 to get the structural info for all the public declarations in the same structural/typed format as you do today with structures from SourceKitten parsed from users' source code.

[ilyapuchka]

That's an awesome tip, thanks @jpsim !

[Sajjon]

@jpsim Hey! Thanks for input, sorry but I did not fully understand, how can the issue #405, which is closed help me to retrieve information about properties for all UIKit types, such as UILabel, UIButton etc? Can you shed some light on how I would do it?

Any help is greatly appreciated ❤️

[jpsim]

Have you read the whole thread? Let me know if there are any parts that you'd like clarified.

Note that jpsim/SourceKitten#405 is an issue on the SourceKitten issue tracker, not Sourcery.

[Sajjon]

@jpsim Yes I have read the thread, it sound like I should use a command xcrun swift-ide-test, which I fail to run. And running said (unavailable) command on some Xcode Project? But as I said this is unclear to me.

I would like access the interfaces (properties) of UIView classes (UIButton, UILabel etc..) :)

[jpsim]

it sound like I should use a command xcrun swift-ide-test

No, I've highlighted programmatic ways in which to use SourceKittenFramework, from the command line too: https://github.com/jpsim/SourceKitten/issues/405#issuecomment-316490274

$ pwd
/Users/jp/Downloads
$ curl -O -L https://github.com/jpsim/SourceKitten/releases/download/0.18.0/SourceKittenFramework.framework.zip
$ unzip SourceKittenFramework.framework.zip
$ cat req.yml
key.request: source.request.editor.open.interface
key.name: "5F63C5B8-6D92-44FF-8012-DCA7D787D243"
key.compilerargs:
  - "-target"
  - "x86_64-apple-macosx10.12"
  - "-sdk"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk"
  - "-I"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/local/include"
  - "-F"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/PrivateFrameworks"
  - "-F"
  - "/Users/jp/Downloads/Carthage/Build/Mac"
  - ""
key.modulename: "SourceKittenFramework"
key.toolchains:
  - "com.apple.dt.toolchain.XcodeDefault"
key.synthesizedextensions: 1
$ sourcekitten request --yaml req.yml | jq -r '.["key.sourcetext"]'

[sidmani]

I had this issue too, and I solved it with a shell script that saves the generated interfaces as intermediary files and then gets sourcery to generate the code based on those. See https://github.com/sidmani/Chain, specifically generate.sh; if anyone has advice on how to make it less hacked-together, I'd love to hear it.

[ilyapuchka]

I don't think we need it for 1.0 version, but will be definitely interesting to investigate it. Though there is solution for stated problem, integrating it in Sourcery seems to be hacky and easy to break with each new Xcode version as it requires some hardcoding.

[AndriiPetrovDev]

Guys, just add folder with your Source code to the script for eample if you need some type from RealmSwift

shellScript = "sourcery --sources YourFolder/ --sources ../Pods/RealmSwift --templates ../Sourcery/Templates --output YourFolder/\n";

[rehsals]

I've made a PR https://github.com/krzysztofzablocki/Sourcery/pull/1035 That's how we intend to solve this with our project. See PR description for details Maybe we could liven this discussion up as the problem is rather common I guess

[art-divin]

I am closing this issue due to inactivity, thank you for your PR @rehsals 🤝

If needed, please re-open 👍🏻