SPM build manifest does not exist… or does not match expected format

[dabrahams]

I'm using SourceKitten indirectly through the Jazzy doc generator. The YAML file does in fact exist and is attached (compressed)

Using config file /Users/dave/src/val/.jazzy.yml
Running swift build
SPM build manifest does not exist at `/Users/dave/src/val/.build/debug.yaml` or does not match expected format.
Error: Failed to generate documentation
/Users/dave/.gem/ruby/2.6.0/gems/jazzy-0.14.3/lib/jazzy/executable.rb:39:in `execute_command': /Users/dave/.gem/ruby/2.6.0/gems/jazzy-0.14.3/bin/sourcekitten ["doc", "--spm", "--module-name", "Val", "--", "-c", "debug", "-v"] (RuntimeError)

Running swift build

SPM build manifest does not exist at `/Users/dave/src/val/.build/debug.yaml` or does not match expected format.

Error: Failed to generate documentation
    from /Users/dave/.gem/ruby/2.6.0/gems/jazzy-0.14.3/lib/jazzy/sourcekitten.rb:269:in `run_sourcekitten'
    from /Users/dave/.gem/ruby/2.6.0/gems/jazzy-0.14.3/lib/jazzy/doc_builder.rb:82:in `block in build'
    from /Users/dave/.gem/ruby/2.6.0/gems/jazzy-0.14.3/lib/jazzy/doc_builder.rb:80:in `chdir'
    from /Users/dave/.gem/ruby/2.6.0/gems/jazzy-0.14.3/lib/jazzy/doc_builder.rb:80:in `build'
    from /Users/dave/.gem/ruby/2.6.0/gems/jazzy-0.14.3/bin/jazzy:16:in `<top (required)>'
    from /usr/local/bin/jazzy:23:in `load'
    from /usr/local/bin/jazzy:23:in `<main>'

[dabrahams]

That was with jazzy version: 0.14.3 but I just reproduced it with the tip of main. @compnerd any idea about this?

[compnerd]

@dabrahams unfortunately, SourceKitten doesn't (officially) support Windows yet. I do have it working on Windows, but need to get some more patches merged which are waiting on @jpsim to help with. In the mean time, I would recommend that you experiment with https://github.com/thebrowsercompany/SourceKitten/tree/compnerd/windows (the file handling in SourceKitten was making assumptions that don't hold).

[dabrahams]

You mean, even though I'm running on a Mac, your fork might solve the problem?

[compnerd]

Oh! I missed the fact it was on macOS!

[compnerd]

@dabrahams what version of the toolchain are you using? There was a change recently in SPM that would could cause conflicts as the contents of the manifest has changed.

[dabrahams]

@compnerd Thanks; I only asked you because I saw a recent commit of yours in this repo and though you might have a clue.

[dabrahams]

@dabrahams what version of the toolchain are you using?

Whatever comes with 14.3.1

[johnfairh]

Quick workaround: temporarily replace line 1364 of that debug.yaml with "Generating XCTestCases":. (And probably replace the module name in the command with ValModule).

The error is because Yams is refusing to load the file; it turns out SPM (or maybe something called TestGeneratorPlugin?) is creating technically invalid yaml.

TIL that yaml specifies the maximum key length is 1024 characters, and Yams implements this strictly.

SPM (or maybe this plugin? at any rate SPM permits it) creates a yaml key in this case by concatenating all of the test source file names. This is fundamentally doomed and hits the limit at line 1364 of your debug.yaml starting "Generating XCTestcases for [\"While\".....

Pragmatically I wonder if the long-term solution here is to allow Yams to be more relaxed about this constraint: I doubt SPM will be particularly enthusiastic about policing their key lengths and the behaviour is already shipped.

Yams error

1364:2463: error: scanner: while scanning a simple key in line 1364, column 3 could not find expected ':': "Generating XCTestCases for [\"While\", \"GenericParameters\", \"FunctionTrivial\", \"TraitExtension\", \"MetatypeType\", \"Existential\", \"BindingDecl\", \"IntegerLiteralExpr\", \"ProjectionEscape\", \"StructuralConformance\", \"DefaultArgument\", \"ConditionalStmt\", \"Tuple\", \"AssociatedTypeLookup\", \"CallOperator\", \"FunctionIdentity\", \"CallGenericFunction\", \"ConditionalStmt\", \"MemberwiseInit\", \"ImplicitMutableCapture\", \"BuiltinFunction\", \"NumericLiteral\", \"ParameterEscape\", \"BindingEscape\", \"Import\", \"Property\", \"StringLiteralExpr\", \"Conformance\", \"MatchExpr\", \"UnterminatedString\", \"UnionType\", \"ConditionalExpr\", \"UnterminatedNestedComment\", \"BindingTypeInferenceWithHints\", \"TraitDecl\", \"GenericTypeAlias\", \"Init\", \"UnterminatedComment\", \"ExtensionOfImport\", \"DuplicateFunctionParameter\", \"UnionTypeGeneric\", \"InoutExpr\", \"DeclModifiers\", \"MatchExpr\", \"CallInit\", \"Overloading\", \"DoWhile\", \"CallGenericUnnamedSubscript\", \"ExistentialGenericAPI\", \"MethodBundleDecl\", \"Constructor\", \"Semicolons\", \"ConformanceGeneric\", \"Return\", \"ConditionalExpr\", \"CyclicRefinements\", \"UnusedResult\", \"CallMethodBundle\", \"Pragma\", \"FloatLiteralExpr\", \"LocalBindings\", \"BodylessDecl\", \"ImplicitQualification\", \"CastUp\", \"CallNonCallable\", \"StructDecl\", \"CallMemberwiseInit\", \"Ownership\", \"CastDown\", \"GenericArguments\", \"Loops\", \"ImplicitReturn\", \"Deinit\", \"Assignment\", \"Methods\", \"DI\", \"ConditionalExtension\", \"ConformanceWithAssociatedType\", \"ExistentialTraitAPI\", \"CallFunction\", \"BuiltinExtension\", \"MethodDecl\", \"ImplicitImmutableCapture\", \"Ownership2\", \"IllegalExtension\", \"BindingTypeInference\", \"TupleMembers\", \"NameLookupGeneric2\", \"Factorial\", \"Destructuring\", \"Shadowing\", \"Cast\", \"Unused\", \"CallStaticFunction\", \"UndefinedOperator\", \"LambdaTypeInference\", \"UnresolvedUnnamedSubscript\", \"NameLookup\", \"Assignment\", \"NameLookupGeneric\", \"NameLookupLabels\", \"ParameterSet\", \"DefaultArgument\", \"DefaultAssociatedType\", \"StoredPropertyDecl\", \"DuplicateOperatorDecl\", \"ArgumentLists\", \"ParameterDecl\", \"ExplicitCapture\", \"NestedGenerics\", \"DuplicateCapture\", \"SubscriptDecl\", \"Skolemization\"] into /Users/dave/src/val/.build/plugins/outputs/val/ValTests/TestGeneratorPlugin/ValFileTests.swift-4dcbc18baf77385a008774c6a6977d516e2c0fa600a31a93445ee708165cdd98": ^

[dabrahams]

Wow, thanks for the info. I don't know how to intervene in Jazzy's process to rewrite that file though. I think I probably ought to be using Swift-DocC anyway.

[dabrahams]

Wow, I really don't like what Swift-DocC generates. So I'm back to looking for a solution to this.

[johnfairh]

I can't think of a simple solution.

To pursue the workaround above your process would be something like swift build <your args>; <kludge the yaml file>; jazzy <stuff to provide same 'your args' to spm> -- it looks like SPM won't recreate the file, though I don't know if this is guaranteed.

You could track down the TestGeneratorPlugin owner/code and ask if they recognise the key text and can trim it. (drastically, even knock out that part of your project out of the Package.swift for the purposes of building docs?)

You could try jazzy in symbolgraph mode, but you will get better results with the SPM version if you can get it going.

Opened apple/swift-package-manager#6790 to query the yaml.

[dabrahams]

It turns out I'm in a perfect position to trim the text from TextGeneratorPlugin; perhaps I'll try that.

[dabrahams]

I did, and got Jazzy to work for the most part. Please feel free to close this issue unless it is marking something you still want to address (like maybe resiliency to non-conformant YAML).

[johnfairh]

Great. I'm going to keep this around while SwiftPM decide whether they have a bug or not.