Closed drewag closed 4 years ago
You're right. However, as far as I know there's no way for a package author to indicate in the manifest that the package supports Linux.
I'd really love to add this feature, but I don't know how to figure out whether a package supports Linux (other than compiling it).
Compile it! :-)
A reasonable heuristic should be this:
- grep for import uikit,cocoa, other mac only stuff
That wouldn't work for some of my repos because I sometimes #if def them out.
I assume it will come to the SPM spec at some point, but perhaps for now you can define your own spec? (especially since repositories are manually submitted anyway).
Another option would be to look for the "Platforms" tag in the README that a lot of repositories use.
Also, FYI, compiling on Linux isn't enough to indicate it truly supports linux. Unfortunately there are still some Linux runtime issues. I wouldn't really trust a repo unless it explicitly says it supports Linux.
How does it matter if you ifdef them out? that doesn’t affect grep and is the point of checking for os(linux). successful compilation is a near 100% check that it’ll work. But sure, running the tests is a no brainer too ;-)
do you have a package at hand which fails my heuristics?
@helje5 Yup, Swiftlier because of FileSystem+iOSDirectories. It contains an import UIKit
.
Hm, fair enough. So it ignores packages which are Linux compatible but have Apple extensions.
So extra check: This one has a .travis.yml w/ an os: linux
, that is a perfect indicator :-)
Same with a Linux job in azure-pipelines.yml :)
I'm interested in adding this feature but detecting it makes me feel funny, and not in a good way.
It looks like Linux support was originally proposed but didn't make it to evolution which is a shame. It does mention support in the future, but the future can be a long time away.
I've started putting together a spec for a definitive check for how to detect it, but I must admit I hate everything about using detection methods like this. 😂 They are prone to failure, and harder than they should be to implement.
If I do go down this path, I think the strategy should be to look for positive indicators before declaring Linux support rather than assuming support and then finding negative indicators... For example, grepping for import UIKit would be a really common negative indicator, where actually that import could be fine inside an #ifdef
.
Also tracking this here: https://trello.com/c/aWEdW8ey/15-detect-libraries-that-support-linux
If I do go down this path, I think the strategy should be to look for positive indicators before declaring Linux support rather than assuming support and then finding negative indicators...
Since you can't be 100% certain a package is compatible with a specific platform without compiling it and running all code paths, another option would be to assign a score and use a few of the aforementioned heuristics with positive and negative values to come to a conclusion on how likely it is that Linux is supported.
Honestly, I think the more I've thought about this today the more in favour I am of making this a completely opt-in thing by the package author.
Something like placing a .supports_linux (not necessarily that, but some file placed somewhere) file in the repository root. That way, it's the authors responsibility, and decision on whether to declare support for linux, as it is with their decision to state support for a specific version of iOS, macOS, etc...
I really wish the Package.swift was just a JSON or YML file so it could be extended, but it's not.
I have actually been thinking about making a spec for an additional metadata file which could go in the repository that would potentially include categorisation, a better description, keywords, etc... If I do this, having that be where the Linux support flag lives would be good. Need to think about this more though, and the library needs to get some adoption before people would consider adding a metadata file specifically for it.
I've been giving this more thought and have opened up a quick survey on the extra metadata idea. https://iosdevweekly.com/issues/416#start
I realize this is slightly off-topic for this thread, but you mentioned how you've been considering making a spec for your own metadata file (and I took your short survey) so I'm trying to make a persuasive argument for why it would be a great idea ;)
There have been pitches/discussions about extending Package.swift to include metadata, but they haven't gone anywhere. The main issue is that there isn't any clear consensus about where the metadata should reside, but there is widespread support of putting it somewhere so that we're at least moving towards the end goal of an official SwiftPM Package Manager.
I think this is a great first step though! It was stated in those forum discussions that if someone just went out and made their own Metadata format and if it gained widespread usage that it could end up being a candidate for the official metadata file/format in SwiftPM. Perhaps we could discuss and come up with a metadata format here, then if people began adopting it we could later on propose it (or something similar to it after we learn from any mistakes we make) to the swift evolution and see where it goes.
I think it's perfectly on topic for this thread, the idea of that metadata file came partially out of this conversation and it would potentially fix this problem.
There have been pitches/discussions about extending Package.swift to include metadata, but they haven't gone anywhere. The main issue is that there isn't any clear consensus about where the metadata should reside, but there is widespread support of putting it somewhere so that we're at least moving towards the end goal of an official SwiftPM Package Manager.
This is the biggest thing for me. The people who have responded to the survey so far who have said this is a bad idea are consistent in saying that it should be done through evolution. I agree with them, that would be best, but it's going to take a long time and I feel we need this now. I also feel this is a way we can quickly see what works and what demand there is for different metadata.
I genuinely don't want to subvert the evolution process. I want this to feed back into it, while at the same time providing immediate value for the library.
I really wish the Package.swift was just a JSON or YML file so it could be extended, but it's not.
Not sure I can follow that, you can simple do a:
let SwiftPMLibraryInfo = [
"supportsLinux": true
]
It'll be ignored by SPM just like other JSON/YML tools ignore extra stuff.
It'll be ignored by SPM just like other JSON/YML tools ignore extra stuff.
That's true, but as far as I know it would also be ignored by swift package dump-package
which is what I'm running to extract the metadata right now. I don't really want to parse the Swift in that file manually.
Found the forum thread I mentioned: https://forums.swift.org/t/pitch-adding-metadata-to-spm-package/25720
BTW (srz if I'm annoying 😎): I still think that just compiling the package in a Linux docker container is the best way to test for Linux compat. Yes, I know there is still a possibility that the package won't actually run on Linux due to Linux Foundation missing certain things, but that really is a bug in Foundation!
The first run would take a while to build through 1k packages, but few of them are updated every day. A 3€/month Scaleway/DO instance should be more than sufficient to run such a daily job. Go for it! :-)
You could also just assume that the ones which use travis/circle/other CIs to build on linux will support linux. But then you'd have to look for those CI files and parse them...
I like the idea of a “PackageInfo.json” in the root. To make it localizable, it should be allowed to specify language specific overrides in: “PackageInfo.en.json” and so on (any keys in there set/override keys in the main PackageInfo.json
As much as I like the idea of a spec for metadata, I agree a lot with @helje5. Even if a package says it supports Linux, it doesn't have to. Not saying that package authors are malicious, but I've run into this myself many times. Without actually trying to build (and/or run) on Linux, you won't be aware of many potential pitfalls. Make a small change, push the change and everything breaks on another platform I don't test regularly.
I went ahead and tried building all 988 packages listed (as of 63463c8a4d5f3325135a9208847284cc4e20159a) against the current official Swift docker container version (Swift v5.0.2), the following is a list of packages for which swift build
returns 0 as the exit code. Use it if you like, I just wanted to see the outcome as I was curious. This does not mean that everything else runs on some subset of Darwin platforms, I saw quite a few could not find source files for target(s); use the 'path' property in the Swift 4 manifest to set a custom target path
errors for example, apparently many packages just added a manifest without ever trying it out 🙈
It's 354/988 packages, ~35%.
I’d be curious how many also have a non-zero number of tests and have a successful “swift test” on Linux
BTW (srz if I'm annoying 😎)
Not at all! But I'm not sure this is something I want to do, even if it would be cheap/easy. I set out to build a package search tool rather than a CI system. I'm definitely in favour of having a better handle on linux compatibility, but it's not really why I built the SwiftPM library.
I'm happy to trust when people declare compatibility with iOS, macOS, tvOS and watchOS through metadata in the Package.swift and I see Linux in the same way.
I’d be curious how many also have a non-zero number of tests and have a successful “swift test” on Linux
Again this would be fascinating data, and not just on Linux but on the other platforms too. I'm going to be really cautious before trying to do this though. Metadata feels like the right approach to me for this project, for now.
I saw quite a few could not find source files for target(s); use the 'path' property in the Swift 4 manifest to set a custom target path errors for example, apparently many packages just added a manifest without ever trying it out 🙈
Yes, and this is after an initial purge of several different types of failures. My initial scrape was cut down by 50% just by applying a few simple checks 1, 2, 3.
I removed those because they should never have been there but I don't want to be a gatekeeper. I only want to limit the index to libraries that support the SPM, and that are written in Swift 4+ (but only because the tools will no longer work with Swift 3 and below). With those very basic rules, I think I'm happy to have all other packages in the index.
My hope is that the search ranking will bubble the good results to the top, and that that'll be enough. I know this is probably naive but it's my current plan.
As far as I can tell, there is no way to filter or even see if a package has Linux support. This is a major use case for me as most of what I do is cross-platform and it is much harder to find packages that are also cross-platform.