search

apple / swift-protobuf

Plugin and runtime library for using protobuf with Swift
Apache License 2.0
4.57k stars 450 forks source link

Linking against static library that uses SwiftProtobuf causes linker errors #1101

Open BalestraPatrick opened 3 years ago

BalestraPatrick commented 3 years ago

Hello! I'm seeing a pretty weird linker error related to SwiftProtobuf symbols. I'm going to describe the situation as much as possible.

  1. We build a static library that contains some SwiftProtobuf generated code. This library is built for library distribution, since we want to support multiple Xcode and Swift versions.

  2. When we integrate this library in a consumer, that builda SwiftProtobuf from source (we had problems with distributing SwiftProtobuf as a binary built with library distribution in the past since it would cause some weird Objective-C runtime data corruption), the compilation works fine.

  3. When the app gets to linking stage, it throws the following errors because some symbols included in the prebuilt static library aren't found in SwiftProtobuf. We are using the same version everywhere, so I'm pretty sure it's not a version mismatch kind of error.

    Undefined symbols for architecture x86_64:
"_$s13SwiftProtobuf19_ProtoNameProvidingP17_protobuf_nameMapAA01_dH0VvgZTq", referenced from:
"_$s13SwiftProtobuf26_MessageImplementationBaseP29_protobuf_generated_isEqualTo5otherSbx_tFTq", referenced from:
"_$s13SwiftProtobuf7DecoderP14decodeMapField9fieldType5valueyAA01_bE0Vyqd__qd_0_Gm_SDy04BaseH0Qyd__AJQyd_0_GztKAA0e3KeyH0Rd__AA0e5ValueH0Rd_0_r0_lFTj", 
"_$s13SwiftProtobuf7DecoderP15nextFieldNumberSiSgyKFTj", referenced from:
"_$s13SwiftProtobuf7DecoderP24decodeSingularInt32Field5valueys0F0VSgz_tKFTj", referenced from:
"_$s13SwiftProtobuf7DecoderP24decodeSingularInt64Field5valueys0F0VSgz_tKFTj", referenced from:
"_$s13SwiftProtobuf7DecoderP25decodeSingularStringField5valueySSSgz_tKFTj", referenced from:
"_$s13SwiftProtobuf7MessageP05protoC4NameSSvgZTq", referenced from:
"_$s13SwiftProtobuf7MessageP06decodeC07decoderyqd__z_tKAA7DecoderRd__lFTq", referenced from:
"_$s13SwiftProtobuf7MessageP13isInitializedSbvgTq", referenced from:
"_$s13SwiftProtobuf7MessageP13unknownFieldsAA14UnknownStorageVvMTq", referenced from:
"_$s13SwiftProtobuf7MessageP13unknownFieldsAA14UnknownStorageVvgTq", referenced from:
"_$s13SwiftProtobuf7MessageP13unknownFieldsAA14UnknownStorageVvsTq", referenced from:
"_$s13SwiftProtobuf7MessageP4hash4intoys6HasherVz_tFTq", referenced from:
"_$s13SwiftProtobuf7MessageP8traverse7visitoryqd__z_tKAA7VisitorRd__lFTq", referenced from:
"_$s13SwiftProtobuf7MessageP9isEqualTo7messageSbAaB_p_tFTq", referenced from:
"_$s13SwiftProtobuf7MessagePxycfCTq", referenced from:
"_$s13SwiftProtobuf7VisitorP13visitMapField9fieldType5value0G6NumberyAA01_bE0Vyqd__qd_0_Gm_SDy04BaseH0Qyd__AKQyd_0_GSitKAA0e3KeyH0Rd__AA0e5ValueH0Rd_0_r0_lFTj", "_$s13SwiftProtobuf7VisitorP23visitSingularInt32Field5value11fieldNumberys0F0V_SitKFTj", referenced from:
"_$s13SwiftProtobuf7VisitorP23visitSingularInt64Field5value11fieldNumberys0F0V_SitKFTj", referenced from:
"_$s13SwiftProtobuf7VisitorP24visitSingularStringField5value11fieldNumberySS_SitKFTj", referenced from:
"_$s13SwiftProtobuf8_NameMapV0C11DescriptionO4sameyAEs12StaticStringV_tcAEmFWC", referenced from:
"_$s13SwiftProtobuf8_NameMapV0C11DescriptionO8standardyAEs12StaticStringV_tcAEmFWC", referenced from:
ld: symbol(s) not found for architecture x86_64

    (I redacted the from references since they contain internal code, but these symbols are referenced from the static library built with library distribution turned on)

  4. SwiftProtobuf is built as a static library in our project, and I noticed that building it with SwiftProtobuf with library distribution in our project would fix the issue, but given the other issue raised above, we can't do that.

Do you have any idea how what the problem could be here? I'm pretty sure someone has used SwiftProtobuf in a Swift SDK before, so this problem should have surfaced. Why would building some Swift proto generated code for library distribution include some symbols that are not present when linking it to a SwiftProtobuf which is not built for library distribution.

Thanks for the help!

thomasvl commented 3 years ago

At first glance, this doesn't really seem specific to Swift Protobuf and seems more like it is an issue mixing compile modes, so you might need to reach out on the Swift forums or some place else where folks might have more knowledge how how the different flags you might be using change how the compiler behaves.

Lukasa commented 3 years ago

With the library you’re building for library distribution, are you using @_implementationOnly imports of Swift Protobuf? If you aren’t, the bundled version of Protobuf may be exposed in your module’s API and so you have a multiple dependency issue.

BalestraPatrick commented 3 years ago

@Lukasa The generated code with protoc-gen-swift contains a import SwiftProtobuf, so effectively this is included in the only module that I built. I tried to hack the generated code to change this to an @_implementationOnly import SwiftProtobuf, but since the public API of my library also uses the generated type, it's not allowed.

BalestraPatrick commented 3 years ago

Even if that would work, how can someone generate protos that contain a @_implementationOnly import SwiftProtobuf instead of a import SwiftProtobuf?

Lukasa commented 3 years ago

You cannot easily do this, and this limitation is ultimately derived from the fact that Swift Protobuf doesn’t really support being used in this way. In principle with sufficient design work it would be possible for Protobuf’s code generation to create types that would support this mode of operation, but it would require a separate mode for invoking the protoc plugin as it’s fundamentally incompatible with the current mode of operation.

What are you trying to achieve by using library evolution mode?

BalestraPatrick commented 3 years ago

I'm trying to ship one binary Swift framework that uses SwiftProtobuf. I think this should be a pretty common use case. It's hard for me to believe that there's isn't a single SDK out there that uses SwiftProtobuf and is built for distribution as a XCFramework?

Lukasa commented 3 years ago

There might be, but those distributions would need to hide the presence of SwiftProtobuf. Swift Protobuf does not support itself being built in library evolution mode, so it cannot be present in the public ABI of any framework that is built in library evolution mode. That means it can only ever be imported @_implementationOnly, and its types can never be public.

tbkka commented 3 years ago

I'd love to hear more details about the "weird Objective-C runtime data corruption" you saw. Can you help us reproduce that?

mkotsiandris commented 3 years ago

I just faced a similar problem. Our structure is the following:

Undefined symbol: protocol descriptor for SwiftProtobuf.ExtensionMap
Undefined symbol: SwiftProtobuf.BinaryDecodingOptions.init() -> SwiftProtobuf.BinaryDecodingOptions
Undefined symbol: (extension in SwiftProtobuf):SwiftProtobuf.Message._merge(rawBuffer: Swift.UnsafeRawBufferPointer, extensions: SwiftProtobuf.ExtensionMap?, partial: Swift.Bool, options: SwiftProtobuf.BinaryDecodingOptions) throws -> ()
Undefined symbol: (extension in SwiftProtobuf):SwiftProtobuf.Message.serializedData(partial: Swift.Bool) throws -> Foundation.Data

Following the discussion above I switched Library A to expose a dynamic product instead of a static one and that seemed to resolve these issues. It's not clear to me though why this is happening and if there is something wrong on they way we are linking the different projects.

Lukasa commented 3 years ago

@mkotsiandris How are you linking frameworks A and B?

acecilia commented 3 years ago

I am experiencing also multiple linking issues with the latest version of Firebase 8.1.0 that added SwiftProtobuf as a dependency. The SwiftProtobuf version is 1.17.0:

❌ Undefined symbols for architecture x86_64
❌   "method descriptor for SwiftProtobuf.Message.init() -> A", referenced from:
...

I managed to reproduce it in a sample project. The sample project integrates SwiftProtobuf in different ways, some of them work well some of them dont. The dependency tree is App -> Framework (static) -> FirebaseMLModelDownloader (static) -> SwiftProtobuf.

To run the sample project install xcodegen, xcbeautify and carthage. Then do:

git clone https://github.com/acecilia/SwiftProtobufLinkageIssue.git
cd SwiftProtobufLinkageIssue
make carthage_bootstrap # To build SwiftProtobuf as xcframeworks. This will take a couple of minutes

I tested multiple integration alternatives for SwiftProtobuf:

TLDR:

Why would building some Swift proto generated code for library distribution include some symbols that are not present when linking it to a SwiftProtobuf which is not built for library distribution.

So far I could not figure out the root cause of the linking issues, but it is clear that they only happen when BUILD_LIBRARY_FOR_DISTRIBUTION = NO

paynerc commented 2 years ago

There might be, but those distributions would need to hide the presence of SwiftProtobuf. Swift Protobuf does not support itself being built in library evolution mode, so it cannot be present in the public ABI of any framework that is built in library evolution mode. That means it can only ever be imported @_implementationOnly, and its types can never be public.

I face a similar issue. I develop an SDK that at the present time uses Protobuf as an internal implementation detail. We ship this framework both as a dynamic .xcframework as well as a static .xcframework for our consumers that then build their own SDK which encapsulates ours. Using the @_implementationOnly import approach has worked fine and we happily have been moving along.

I now face the situation where I need to make one of the Protobuf generated objects from this SDK public and accessible to the consumers of our SDK. Flipping the first bit pubic, of course begins the cascade where the the @_implemenationOnly imports fail. Reverting those then causes the warning:

Module 'SwiftProtobuf' was not compiled with library evolution support; using it means binary compatibility for 'MySDK' can't be guaranteed

I guess the main question I am asking here is... what is the reason that

Swift Protobuf does not support itself being built in library evolution mode

Is there anything that can be done to make that support possible?

Lukasa commented 2 years ago

Is there anything that can be done to make that support possible?

Ultimately this is just a project policy decision. The project can choose to define a stable ABI if it wishes to. In the short term this would require a pretty extensive audit of the ABI, to confirm that everything that is currently exposed should be. It also has performance implications that might be important as well, so the project should probably be investing in that effort too.

pwittchen commented 1 year ago

Hi,

I'm facing the same problem. Can you please provide any suggestions, how can I correctly add Swift Protobuf dependency as XCFramework to the project without having errors?

pwittchen commented 1 year ago

@Lukasa as you told in #1348

Swift Protobuf doesn't provide a stable ABI, so generally speaking you cannot offer it to clients in the form of an XCFramework. You would need to conceal your use of Swift Protobuf, importing it as @_implementationOnly.

so changing import keyword to @_implementationOnly in my project for Swift Protobuf only should solve the problem and clients would be able to add my library provided as XCFramework which uses Swift Protobuf without the need to add Swift Protobuf as a dependency in their app? It this correct?

Lukasa commented 1 year ago

Yes, that will work. Note that you'll be unable to use any protobuf message types in your API.

pwittchen commented 1 year ago

I'm using protobuf messages inside the library, so it's not exposed to the client via external API. Ok, thanks! I'll try it and let you know about the results.

pwittchen commented 1 year ago

@Lukasa I changed import Swift Protobuf to @_implementationOnly import SwiftProtobuf in my code, compiled my project to the XCFramework and client claims that his application does not see SwiftProtobuf dependency.

Lukasa commented 1 year ago

That sounds like everything is working correctly.

pwittchen commented 1 year ago

@Lukasa

In this case app is compiled correctly, but during starting it in the iOS simulator, the following errors are thrown:

dyld[50051]: Library not loaded: @rpath/SwiftProtobuf.framework/SwiftProtobuf

  Referenced from: /Users/.../Library/Developer/Xcode/DerivedData/app-fjxtxghupzrzaqgmblomcfbryxpp/Build/Products/UAT-iphonesimulator/iOS_SDK.framework/iOS_SDK

  Reason: tried: '/Users/.../Library/Developer/Xcode/DerivedData/app-fjxtxghupzrzaqgmblomcfbryxpp/Build/Products/UAT-iphonesimulator/SwiftProtobuf.framework/SwiftProtobuf' (no such file), '/Users/.../Library/Developer/Xcode/DerivedData/Millennium-fjxtxghupzrzaqgmblomcfbryxpp/Build/Products/UAT-watchsimulator/SwiftProtobuf.framework/SwiftProtobuf' (no such file),

What should I do to make it working correctly?

Lukasa commented 1 year ago

This appears to be an issue with the way the app is being built or with the xcframework. I don't know how you're packaging it or how you've built your XCFramework, but questions about how to get xcframeworks working in simulators are probably best handled on the Apple Developer Forums.

pwittchen commented 1 year ago

I don't know how you're packaging it or how you've built your XCFramework

I'm building my XCFramework with the following script below. It's pretty basic. Is there anything wrong with that?

Moreover, I have more dependencies to my XCFramework and I provided them to the client via other XCFrameworks (SWCompression, RxCocoa, RxSwift) and there are problems only with SwiftProtobuf. Maybe I need to add something to my script to make it work with SwiftProtobuf?

#!/usr/bin/env bash

start_time=$(date +%s)

echo "▸ CLEAN"

rm -rf build/ || true
xcodebuild clean

echo "▸ BUILD"

xcodebuild build \
    -workspace iOS-SDK.xcworkspace \
    -scheme iOS-SDK \
    -sdk iphoneos \
    -configuration Release \
    SKIP_INSTALL=NO \

echo "▸ ARCHIVE: iOS Device"

xcodebuild archive \
    -workspace iOS-SDK.xcworkspace \
    -scheme iOS-SDK \
    -configuration Release \
    -destination 'generic/platform=iOS' \
    -archivePath './build/iOS-SDK.framework-iphoneos.xcarchive' \
    SKIP_INSTALL=NO \
    BUILD_LIBRARIES_FOR_DISTRIBUTION=YES

echo "▸ ARCHIVE: iOS Simulator"

xcodebuild archive \
    -workspace iOS-SDK.xcworkspace \
    -scheme iOS-SDK \
    -configuration Release \
    -destination 'generic/platform=iOS Simulator' \
    -archivePath './build/iOS-SDK.framework-iphonesimulator.xcarchive' \
    SKIP_INSTALL=NO \
    BUILD_LIBRARIES_FOR_DISTRIBUTION=YES

echo "▸ CREATE XCFRAMEWORK"

xcodebuild \
    -create-xcframework \
    -framework './build/iOS-SDK.framework-iphonesimulator.xcarchive/Products/Library/Frameworks/iOS_SDK.framework' \
    -framework './build/iOS-SDK.framework-iphoneos.xcarchive/Products/Library/Frameworks/iOS_SDK.framework' \
    -output './build/iOS-SDK.xcframework'

echo "▸ CLEAN XCARCHIVES"

rm -rf build/*.xcarchive

echo "▸ COPY PODSPEC"

cp iOS-SDK.podspec.integration build/iOS-SDK.podspec

if [ -d "build/iOS-SDK.xcframework" ] ; then
  echo "▸ SUCCESS"
else
  echo "▸ FAILURE !!!"
  exit 0
fi

end_time=$(date +%s)
elapsed=$(( end_time - start_time ))

echo "▸ DONE in $elapsed s"
Lukasa commented 1 year ago

Where is the requirement to load SwiftProtobuf coming from? I'd normally expect that the above would not produce a SwiftProtobuf dylib directly, if you're using Swift Package Manager, as typically SwiftPM statically links dependencies.

pwittchen commented 1 year ago

Unfortunately, I don't have the access to the final app code. I'm just providing my XCFramework. Client sent me errors pasted above and said that he compiled app successfully and got these errors when app was starting in the iOS simulator. I know that they don't use any iOS package managers like Swift PM or CocoaPods due to some corporate/security regulations and they're adding all dependencies manually as XCFrameworks in the XCode.

Lukasa commented 1 year ago

Ok, so they'll need a detailed build log to work out what is trying to load SwiftProtobuf.framework.

pwittchen commented 1 year ago

Ok, I got build logs. When they don't attach SwiftProtobuf as XCFramework and they attach my library only as XCFramework, then nothing is trying to load SwiftProtobuf. There's no SwiftProtobuf in the build log, but build finishes with success.

When app is starting, the following error is thrown in the runtime:

Symbol not found: _$s13SwiftProtobuf19_ProtoNameProvidingP17_protobuf_nameMapAA01_dH0VvgZTq

  Referenced from: /Users/.../Library/Developer/Xcode/DerivedData/app-fjxtxghupzrzaqgmblomcfbryxpp/Build/Products/UAT-iphonesimulator/iOS_SDK.framework/iOS_SDK

  Expected in: /Users/.../Library/Developer/Xcode/DerivedData/app-fjxtxghupzrzaqgmblomcfbryxpp/Build/Products/UAT-iphonesimulator/SwiftProtobuf.framework/SwiftProtobuf

In the log above iOS_SDK is the name of the custom library (XCFramework).

Can you give any suggestions, how to make it work without Cocoa Pods or Swift PM?

Lukasa commented 1 year ago

So this seems to imply that your custom library is expecting a SwiftProtobuf.framework. Where is it getting that expectation from? SwiftPM would normally statically link SwiftProtobuf into your SDK.

pwittchen commented 1 year ago

I'm not using Swift PM. I'm using Cocoa Pods in my project and I'm adding SwiftProtobuf as a CocoaPod dependency. In my code, the only place where I import SwiftProtobuf are files representing protobuf messages with *.pb.swift postfix, which were automatically generated by the protoc tool. I have 8 files like that in the project. In the each file I changed import SwiftProtobuf to @_implementationOnly import SwiftProtobuf manually. I also had to add import SwiftProtobuf in my tests to compile them, but it should not influence output of the XCFramework, because it's the test code - not the library code. In my code, I'm using struct (actually two structs) from one of these files, because I want to send HTTP request with proto message to the server, so I guess, generated file, which contains this struct requires SwiftProtobuf for work.

My Podfile looks like that:

platform :ios, '11.4'

use_frameworks!

target 'iOS-SDK' do
  pod 'SwiftProtobuf', '~> 1.20.3'
  pod 'SWCompression', '~> 4.8.3'
  pod 'RxSwift', '6.5.0'
  pod 'RxCocoa', '6.5.0'
end

target 'iOS-SDKTests' do
  pod 'Nimble', '10.0.0'
  pod 'CryptoSwift', '~> 1.6.0'
end

post_install do |installer|
  installer.pods_project.targets.each do |target|
      target.build_configurations.each do |config|
          config.build_settings.delete 'IPHONEOS_DEPLOYMENT_TARGET'
          config.build_settings['BUILD_LIBRARY_FOR_DISTRIBUTION'] = 'YES'
      end
  end
end
Lukasa commented 1 year ago

Unfortunately I'm not really able to help here: I don't know how to build an xcframework that bundles multiple frameworks. @_implementationOnly doesn't force Cocoapods to statically link Swift Protobuf, so you'll still need to distribute a dylib.

pwittchen commented 1 year ago

By dylib you mean SwiftProtobuf as XCFramework or something else? Can you please tell me, how can I correctly distribute this dylib?

Lukasa commented 1 year ago

I can't, because I don't know. All I can tell you is what I can see: your framework iOS_SDK.framework is expecting to find SwiftProtobuf.framework. This is because it is linked against SwiftProtobuf.framework. That forces you to distribute that. You may be able to distribute it via xcframework, but I don't know that that will work. Ideally, you'd statically link SwiftProtobuf with your SDK so that it was entirely invisible, but I don't know how to do that with Cocoapods.

allevato commented 1 year ago

Unfortunately I'm not really able to help here: I don't know how to build an xcframework that bundles multiple frameworks. @_implementationOnly doesn't force Cocoapods to statically link Swift Protobuf, so you'll still need to distribute a dylib.

I don't believe this is possible; since the compiler's search path logic requires exactly one module per framework, I don't think the .xcframework functionality in SwiftPM/Xcode that unpacks them supports anything else.

To folks distributing SwiftProtobuf as an @_implementationOnly import in a static library, I think the best recommendation now is to use the module alias feature in Swift 5.7 to rename the symbols in SwiftProtobuf to something else. As long as you never vend protos as part of the public API (which you can't be, if you're using @_implementationOnly import), you should be able to link it into the same static library as the rest of your SDK, users of your SDK won't be required to import it, and if the SDK user wants to use SwiftProtobuf, they're freed from the constraint of having to use the same exact version the SDK author used.

The downside is that there's a binary size increase if someone is using your SDK and SwiftProtobuf because they'll need to link both, but there's not really another option if you want your SDK to be completely self-contained.

I don't know Cocoapods well enough to get an idea of the extra work required there. But if you're just using Cocoapods to distribute the static library xcframework artifacts, it shouldn't require any additional effort; all the changes would be to which flags you pass to swiftc to alias the modules.

pwittchen commented 1 year ago

Thanks for the response @allevato. Can you give some hints how can I apply module alias in CocoaPods based project? Examples from the link are dedicated to Swift PM as far as I see. I tried to search a solution for using module alias in Cocoa Pods, but could not find anything... I'm using Cocoa Pods mostly for managing project dependencies (pods) and generating xcworkspace for XCode for development.

allevato commented 1 year ago

I'm not sure how Cocoapods would handle it, exactly. I'm assuming there's a mechanism that lets you pass arbitrary command line flags to swiftc so that you could write -module-alias SwiftProtobuf=MySDKSwiftProtobuf. But this also needs to be done when compiling SwiftProtobuf, not just your SDK, do you may need to fork the SwiftProtobuf pods to add those flags there as well. Unless there's a global flag you can pass to Cocoapods that says "apply these flags to all Swift compilations across all dependencies".

Sorry I can't offer more specific guidance; I have next to zero experience with Cocoapods.