search

swiftlang / swift-package-manager

The Package Manager for the Swift Programming Language
Apache License 2.0
9.67k stars 1.32k forks source link

Two-phase build needed to avoid link errors #7251

Open dabrahams opened 6 months ago

dabrahams commented 6 months ago

Description

Check out the spm-bug-two-builds-needed tag of https://github.com/dabrahams/hylo-spec-parser and do swift build --build-tests. It will fail with a link error:

swift build --explicit-target-dependency-import-check=error --build-tests
warning: 'hylo-spec-parser': found 1 file(s) which are unhandled; explicitly declare them as resources or exclude from the target
    /Users/dave/src/hylo-spec-parser/Sources/ParseGen/EBNFParser.citron
Building for debugging...
[0/2] Compiling CitronCLibrary main_.c
[2/4] Emitting module citron
[3/4] Compiling citron main.swift
[3/4] Linking citron
Build complete! (1.12s)
[1/1] Compiling plugin CitronParserGenerator
Building for debugging...
[1/3] Generating EBNFParser.swift from EBNFParser.citron
[3/13] Compiling LoftDataStructures_BitVector BitVector.swift
[4/13] Emitting module LoftDataStructures_BitVector
[5/13] Compiling LoftDataStructures_Zip2Collection Zip2Collection.swift
[6/13] Emitting module LoftDataStructures_Zip2Collection
[7/17] Emitting module CitronParserModule
[8/17] Compiling CitronParserModule CitronParser.swift
[9/17] Compiling CitronLexerModule SourceRegion.swift
[10/17] Emitting module Utils
[11/17] Compiling Utils Incidental.swift
[12/17] Compiling Utils Algorithms.swift
[13/17] Compiling Utils String+Extensions.swift
[14/17] Compiling CitronLexerModule CitronLexer.swift
[15/17] Compiling CitronLexerModule Scanner.swift
[16/17] Emitting module CitronLexerModule
[17/25] Compiling ParseGen BNF.swift
[18/25] Compiling ParseGen EBNFScanner.swift
[19/25] Compiling ParseGen EBNF.swift
[20/25] Compiling ParseGen Diagnostic.swift
[21/25] Compiling ParseGen Analysis.swift
[22/25] Compiling ParseGen EBNFGrammar.swift
[23/25] Compiling ParseGen EBNFParser.swift
[24/25] Emitting module ParseGen
[25/30] Compiling ParseGenTests TokenPatternTests.swift
[26/30] Compiling ParseGenTests BNFConversionTests.swift
[27/30] Emitting module ParseGenTests
[28/30] Compiling ParseGenTests EBNFScannerTests.swift
[29/30] Compiling ParseGenTests EBNFParseResultTests.swift
ld: Undefined symbols:
  nominal type descriptor for ParseGen.EBNFParser.CitronTokenCode, referenced from:
      _symbolic _____ 8ParseGen10EBNFParserC15CitronTokenCodeO in EBNF.swift.o
      _symbolic SJ______t 8ParseGen10EBNFParserC15CitronTokenCodeO in EBNFScanner.swift.o
      _symbolic ______SSt 8ParseGen10EBNFParserC15CitronTokenCodeO in EBNFScannerTests.swift.o
      _symbolic Say______SStG 8ParseGen10EBNFParserC15CitronTokenCodeO in EBNFScannerTests.swift.o
      _symbolic _____ySay_____GSay______SStGG s12Zip2SequenceV 8ParseGen4EBNFO5TokenV AC10EBNFParserC06CitronF4CodeO in EBNFScannerTests.swift.o
      _symbolic _____ySay_____GSay______SStG_G s12Zip2SequenceV8IteratorV 8ParseGen4EBNFO5TokenV AE10EBNFParserC06CitronG4CodeO in EBNFScannerTests.swift.o
  type metadata for ParseGen.EBNFParser.CitronTokenCode, referenced from:
      static ParseGen.EBNF.Token.__derived_struct_equals(ParseGen.EBNF.Token, ParseGen.EBNF.Token) -> Swift.Bool in EBNF.swift.o
      lazy protocol witness table accessor for type ParseGen.EBNFParser.CitronTokenCode and conformance ParseGen.EBNFParser.CitronTokenCode : Swift.RawRepresentable in ParseGen in EBNF.swift.o
      ParseGen.EBNF.Symbol.init(ParseGen.EBNF.Token) -> ParseGen.EBNF.Symbol in EBNF.swift.o
      ParseGen.EBNF.Symbol.init(ParseGen.EBNF.Token) -> ParseGen.EBNF.Symbol in EBNF.swift.o
      ParseGen.EBNF.Token.description.getter : Swift.String in EBNF.swift.o
      one-time initialization function for tokenID in EBNFScanner.swift.o
      static ParseGen.EBNF.tokens(in: Swift.Substring, onLine: Swift.Int, fromFile: Swift.String) -> [ParseGen.EBNF.Token] in EBNFScanner.swift.o
      ...
  protocol conformance descriptor for ParseGen.EBNFParser.CitronTokenCode : Swift.Equatable in ParseGen, referenced from:
      lazy protocol witness table accessor for type ParseGen.EBNFParser.CitronTokenCode and conformance ParseGen.EBNFParser.CitronTokenCode : Swift.Equatable in ParseGen in EBNFScannerTests.swift.o
  protocol conformance descriptor for ParseGen.EBNFParser.CitronTokenCode : Swift.RawRepresentable in ParseGen, referenced from:
      lazy protocol witness table accessor for type ParseGen.EBNFParser.CitronTokenCode and conformance ParseGen.EBNFParser.CitronTokenCode : Swift.RawRepresentable in ParseGen in EBNF.swift.o
  protocol conformance descriptor for ParseGen.EBNFParser : CitronParserModule.CitronParser in ParseGen, referenced from:
      lazy protocol witness table accessor for type ParseGen.EBNFParser and conformance ParseGen.EBNFParser : CitronParserModule.CitronParser in ParseGen in EBNFParseResultTests.swift.o
  ParseGen.EBNFParser.__allocating_init() -> ParseGen.EBNFParser, referenced from:
      ParseGenTests.ebnf((text: Swift.Substring, sourceFilePath: Swift.String, startLine: Swift.Int)) throws -> [ParseGen.EBNF.Definition] in EBNFParseResultTests.swift.o
  type metadata accessor for ParseGen.EBNFParser, referenced from:
      ParseGenTests.ebnf((text: Swift.Substring, sourceFilePath: Swift.String, startLine: Swift.Int)) throws -> [ParseGen.EBNF.Definition] in EBNFParseResultTests.swift.o
      lazy protocol witness table accessor for type ParseGen.EBNFParser and conformance ParseGen.EBNFParser : CitronParserModule.CitronParser in ParseGen in EBNFParseResultTests.swift.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[29/30] Linking HyloSpecParserPackageTests

The missing symbols are in EBNFParser.swift which it says it generated above:

[1/3] Generating EBNFParser.swift from EBNFParser.citron

If you simply issue the same command again it will succeed. It will also succeed if you start with a plain swift build and then issue swift build --build-tests.

Expected behavior

Tests build successfully from the get-go.

Actual behavior

No response

Steps to reproduce

No response

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.9.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.87.3 Apple Swift version 5.9.2 (swiftlang-5.9.2.2.56 clang-1500.1.0.2.5) Target: arm64-apple-macosx14.0 Darwin DaveA-MBP14-5.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64

neonichu commented 6 months ago

My theory for this would be that we're not linking EBNFParser.o to ParseGenTests, not entirely sure why this sometimes works and sometimes doesn't though.

dabrahams commented 6 months ago

Isn't that .o file supposed to be incorporated into ParseGen and thus not linked directly into ParseGenTests, because ParseGen is linked to ParseGenTests?

neonichu commented 6 months ago

It would be good if things worked that way, but today SwiftPM actually aggregates all the .o files from all dependencies (except dynamic ones, of course) and links them directly to the final product.

My guess to what happens here is that under certain conditions, the .o files from plugin-generated files aren't part of the link step.

dabrahams commented 6 months ago

OK, well those conditions ought to be pretty easy to track down from this example, as it's quite small. If you need it smaller, it would probably be "easy" to minimize, though I'd rather not spend the hour or two on that if I don't have to.

neonichu commented 6 months ago

I think the example we have should be sufficient, thanks!