Migrate Dokka's analysis to the K2 compiler API

[IgnatBeresnev]

Background

Dokka depends on the Kotlin compiler to analyze source code and KDoc comments.

Kotlin's K2 compiler has been in the works for some time, and it is going stable in Kotlin 2.0.

That means that the K1 compiler API that Dokka uses will be phased out over time, and Dokka should migrate its analysis to use the new K2 compiler API.

Steps

There are 4 major steps to make this migration happen:

1. Extract K1 analysis from dokka-base (✅)

The current K1 analysis is deeply ingrained in dokka-base, with internal compiler types being exposed in public API and used by downstream plugins. This makes the migration to K2 very painful both for maintainers and for downstream users.

K1 analysis should be extracted from dokka-base, and a common analysis-api abstraction should be introduced, making it possible for K1 and K2 implementations to coexist at the same time and be used interchangeably.

• [x] #3034

Update: This step has been done, see #3099 for more details.

2. Implement a prototype to identify blockers (✅)

K2 already provides some semi-stable API that Dokka can use. We should migrate as much of the analysis as possible to identify the missing API and other blocking issues that would need to be addressed by the compiler team.

• [x] #3094

3. Stabilize JVM-only projects (In Progress)

Kotlin/JVM projects should be easier to support and debug on, so this is the first step in migrating users to the new K2 analysis.

Once generating documentation for Kotlin/JVM projects is stable enough, we will enable K2 analysis by default for such projects.

• [ ] https://github.com/Kotlin/dokka/issues/3140

4. Stabilize KMP projects (semi-blocked)

Once Kotlin/JVM projects are stabilized, the only thing left will be to support Kotlin Multiplatform projects. This is a separate step as it will likely require more time to test it and support less trivial parts of KMP, such as native libraries.

• [ ] https://github.com/Kotlin/dokka/issues/3137

While some KMP projects do already work, there are many corner cases and issues that need to be fixed, both on Dokka's side and in Analysis API.

Blocking issues:

• KT-62696 - Analysis API: KMP support for Standalone Mode

Technical debt

A collection of issues and TODOs that are not necessarily part of any particular step, but that need to be addressed either during the migration or after it:

• https://github.com/Kotlin/dokka/issues/3142
• https://github.com/Kotlin/dokka/issues/3141
• Once the API is more or less stable and no significant changes are expected:
  • Revisit the architecture / do the final code review.
  • Write internal documentation and KDocs to make it easier to onboard new contributors.

Related issues

Have a look at the issues marked with the topic: K2 tag to see what's been done and what still needs to be addressed in relation to this issue.

[IgnatBeresnev]

The K2 analysis prototype has been imlpemented in #3094 :tada:

Currently, the following projects are supported:

• All Kotlin/JVM projects that only use Kotlin's Standard Library (so no Java SE API or any 3rd party libraries). On any JVM version.
• Kotlin/JVM project that use Java SE API directly (for example, import java.util.HashMap) only work if it's Java >= 9, because non-modularized Java (JCL) is not supported.

However, we are unable to proceed further and stabilize Kotlin/JVM projects until K2 introduces some additional API. A separate issue has been created to keep track of this:

• https://github.com/Kotlin/dokka/issues/3140

---

This issue's description has been updated, and additional issues have been created to reflect the current status.

[IgnatBeresnev]

Current status

The team has made significant progress in stabilizing the K2 analysis implementation. Many of our efforts also help find and fix bugs in the K2 IDE plugin, since we share a lot of the underlying analysis logic (same low-level API).

Since the last update, we've been unblocked with stabilizing Kotlin/JVM projects, have fixed the known major issues, and have completed the first iteration of testing Dokka manually on a variety of user projects (#3328).

The subtasks have been updated accordingly, they now include the newly discovered bugs and other issues.

• https://github.com/Kotlin/dokka/issues/3140
• https://github.com/Kotlin/dokka/issues/3137

We will proceed with stabilizing Kotlin/JVM projects in the near future, aiming to enable K2 for them by default in Dokka 2.0.0, possibly making it available for public testing before the release. Once Kotlin/JVM projects are stable, we'll switch to KMP projects, aiming to enable K2 for them by default in Dokka 2.1.0, but we'll do it sooner if possible (Dokka depends a lot on the upstream API issues).

---

The API that Dokka uses for K2 analysis is not yet stable, so we encounter various issues and bugs, which we report to Kotlin's YouTrack. Analysis API issues that affect Dokka in one way or another are marked with the kt-analysis-dokka tag in YouTrack.

I'll leave a list of issues below for future reference.

Bugs affecting Kotlin/JVM projects:

• KT-62695 - K2 IDE: Unresolved extension functions in KDoc
• KT-64051 - K2 IDE: Analysis API: Unresolved links to typealias in KDoc
• KT-64190 - K2 IDE: Analysis API: KDoc link leads to a function instead of interface
• KT-62880 - K2 IDE: Unresolved java annotation methods in KDoc
• KT-62405 - Analysis API: Symbols SUBSTITUTION_OVERRIDE have no overridden symbols
• KT-64067 - Analysis API: Member scope has an inherited nested class from Java

KMP-specific issues:

• KT-62696 - Analysis API: KMP support for Standalone Mode
  • (child) KT-64159 - Analysis API: JS klib do not work for Kotlin <1.9.0
  • (child) KT-64155 - Analysis API: KotlinIllegalArgumentExceptionWithAttachments: Unexpected FirClassLikeSymbol null for class org.jetbrains.kotlin.fir.symbols.impl.ConeClassLikeLookupTagImpl

API stabilization issues:

• KT-61387 - Analysis API: stabilize Analysis API
  • (child) KT-61404 - Analysis API: implement proper library publishing structure
  • (child) KT-61403 - Analysis API: Stabilize API Surface
• KT-63339 - Analysis API: Provide a way to extract KDoc for symbols
• KT-53669 - Analysis API: redesign KtSymbolOrigin to distinguish kotlin/java source/library declarations

Noticeable differences in Dokka's behaviour that likely come from the upstream Analysis API, but it's undecided if they are bugs, so no YouTrack tickets yet:

• https://github.com/Kotlin/dokka/issues/3356
• https://github.com/Kotlin/dokka/issues/3334
• https://github.com/Kotlin/dokka/issues/3250

[vmishenev]

Current status

We are fixing some issues discovered after the second iteration of testing. TBD...

User-affective issues (28):

• Java-Kotlin interop — synthetic Java properties (7 issues):

  • 3339

  • 3338

  • 3128

  • 3672

  • 3669

  • 3685

  • 3675

• KDoc spec (11 issues):
  • Design doc is done

    • 3451

    • 3179

    • 3555 ;

    • Requires some clarifications on the specification.
    • KDoc link resolving

      • 3639

      • 3586

      • 3604

      • 3632

      • 3385

      • 3356

    • Others:

      • 3348

      • 3373

• Currently blocked by Analysis API (6 issues):

  • 3681

  • 3631

  • 3702

  • 3751

  • KMP: Support intersecting source roots #3701 and #3373

• Order #3250 and #3590

  - Inconsistent resolution of KDoc links to extensions #3632

  • Link to an inherited declaration in @see block replace class container to parent one in link label #3680
• KMP: Expect/actual enums have expect modifiers but don't have actual for builtin functions #3704
• Overriden methods in enum entries are not rendered #3682

Small impact on a user (5):

• 3383

• 3699

• Plugin Kotlin-as-Java (3 issues)

Non-user-affective (5):

• KMP: Generating StdLib #3354 (blocked)

• 3141 (blocked)

• Integration testing

  • 3142

  • 3136

  • 3618