Allow for usage of `DefaultProjectParser` outside of Gradle Daemon process

BoykoAlex commented 1 year ago

What problem are you trying to solve?

Re-use DefaultProjectParser outside of Gradle daemon process. There are various markers added throughout the ASTs via the project parser. The recipes rely on these markers and therefore it is important to place exactly these markers for the OR recipes to work as expected. There is 3rd party code developing its own project parsers attempting to parse the source and place the same markers - essentially reinventing the wheel with a high possibility of introducing bugs.

Describe the solution you'd like

Modify DefaultProjectParser to work with a proxy to/wrapper of org.gradle.api.Project rather than org.gradle.api.Project directly. Project parser only requires a small subset of the data from org.gradle.api.Project hence the suggestion is aggregate this required data in an interface, lets name in ProjectInfo for now. The hope is that API client could fetch ProjectInfo from Gradle Daemon via the Gradle Tooling API by implementing a serializable ProjectInfoImpl class and then with the fetched ProjectInfo use DefaultProjectParser for parsing projects.

Have you considered any alternatives or workarounds?

Parse the project on the Gradle Daemon process side and then with Gradle Tooling API fetch the parsed project, i.e. list of source files. This requires changes across many OR projects to make AST classes implement Serializable. Firstly making all AST classes implement Serializable and work with Java serialization gets out of control (just too many classes with some requiring writeObject/readObject implemented) and secondly this kind of serialize/deserialize ASTs will take very, very long time and huge amounts of memory. This solutions seemed like a complete insanity

Try fetching org.gradle.api.Project via Gradle tooling api. Of course implementations of org.gradle.api.Project iare not serializable, however, it is possible to serialize parts of it and have a trimmed down version of org.gradle.api.Project outside of Gradle Daemon process that would work for the current implementation of DefaultProjectParser. This won't be easy to achieve given the calls such as:


        //noinspection DataFlowIssue
        if (subproject.getPlugins().hasPlugin("org.jetbrains.kotlin.gradle.dsl.KotlinMultiplatformExtension") ||
            subproject.getExtensions().findByName("kotlin") != null && subproject.getExtensions().findByName("kotlin").getClass().getCanonicalName().startsWith("org.jetbrains.kotlin.gradle.dsl.KotlinMultiplatformExtension")) {
            return parseMultiplatformKotlinProject(subproject, exclusions, alreadyParsed, projectProvenance, ctx);
        }

Instead with the `ProjectInfo` approach i would just have a boolean property `isMultiplatformKotlinProject`

Reflection API usage make it even more challenging:
```java
        Object kotlinExtension = subproject.getExtensions().getByName("kotlin");
        NamedDomainObjectContainer<KotlinSourceSet> sourceSets;
        try {
            Class<?> clazz = kotlinExtension.getClass().getClassLoader().loadClass("org.jetbrains.kotlin.gradle.dsl.KotlinMultiplatformExtension");
            //noinspection unchecked
            sourceSets = (NamedDomainObjectContainer<KotlinSourceSet>) clazz.getMethod("getSourceSets")
                    .invoke(kotlinExtension);

        } catch (Exception e) {
            logger.warn("Failed to resolve KotlinMultiplatformExtension from {}. No sources files from KotlinMultiplatformExtension will be parsed.",
                    subproject.getPath());
            return Stream.empty();
        }

Instead with the ProjectInfo approach i would have a collection of kotlin source sets on the ProjectInfo where each sourceset has a name, compile and implementation classpaths etc.

Overall this approach seems rather fragile and hard to maintain in the future. Since usage of org.gradle.api.Project is limitless on the rewrite-gradle-plugin we'd have support more and more features from org.gradle.api.Project and ensure they are serializable. At some point this just may stop working...

Are you interested in contributing this feature to OpenRewrite?

Of course! I have the POC for the ProjectInfo approach and if you are okay with the approach we can polish up the ProjectInfo API, naming etc.

timtebeek commented 1 year ago

@sambsnyd what are your thoughts on this?

BoykoAlex commented 1 year ago

@sambsnyd I can propose the API for ProjectInfo described above that worked for me which we could polish up perhaps if the proposed solution seems acceptable :-)

sambsnyd commented 1 year ago

Yeah that would bean improvement I think. Put it in rewirte-gradle-tooling-model

BoykoAlex commented 1 year ago

@sambsnyd I'd like to get some help with this :-)

I have completed majority of the work in rewrite-gradle-tooling-model

Ability to fetch GradleProjectData model via Gradle Tooling API
ProjectParser in the new module parser that parses the project using the fetched GradleProjectData

Rewrite Tooling Model PR: https://github.com/openrewrite/rewrite-gradle-tooling-model/pull/8

I have also attempted to remove project (sources) parsing logic from the rewrite-gradle-plugin and re-use the ProjectParser from above

Rewrite Gradle Plugin PR: https://github.com/openrewrite/rewrite-gradle-plugin/pull/238

However, i got stuck on the rewrite-gradle-plugin :-\ The isolated classpath loader is the issue... I always get LinkageErrors:

Caused by: java.lang.LinkageError: loader constraint violation: loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @5f74a59a wants to load interface org.openrewrite.gradle.toolingapi.parser.GradleProjectData. A different interface with the same name was previously loaded by org.openrewrite.gradle.RewriteClassLoader @7a259b89. (org.openrewrite.gradle.toolingapi.parser.GradleProjectData is in unnamed module of loader org.openrewrite.gradle.RewriteClassLoader @7a259b89, parent loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @5f74a59a)
    at org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader.findClass(VisitableURLClassLoader.java:186)
    at org.openrewrite.gradle.parser.ProjectParser.repositoryRoot(ProjectParser.java:104)
    at org.openrewrite.gradle.parser.ProjectParser.<init>(ProjectParser.java:83)
    at org.openrewrite.gradle.isolated.DefaultProjectParser.getProjectParser(DefaultProjectParser.java:509)
    at org.openrewrite.gradle.isolated.DefaultProjectParser.parse(DefaultProjectParser.java:541)
    at org.openrewrite.gradle.isolated.DefaultProjectParser.listResults(DefaultProjectParser.java:582)
    at org.openrewrite.gradle.isolated.DefaultProjectParser.dryRun(DefaultProjectParser.java:241)
    at org.openrewrite.gradle.DelegatingProjectParser.lambda$dryRun$3(DelegatingProjectParser.java:107)
    at org.openrewrite.gradle.DelegatingProjectParser.unwrapInvocationException(DelegatingProjectParser.java:147)
    at org.openrewrite.gradle.DelegatingProjectParser.dryRun(DelegatingProjectParser.java:106)
    at org.openrewrite.gradle.RewriteDryRunTask.run(RewriteDryRunTask.java:49)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
    at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:248)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:233)
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:216)
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:199)
    at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)
    at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:105)
    at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)
    at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:59)
    at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:56)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
    at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:56)
    at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)
    at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:67)
    at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:37)
    at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
    at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
    at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
    at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)
    at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:28)
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:100)
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:72)
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:50)
    at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:40)
    at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:29)
    at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:179)
    at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:70)
    at org.gradle.internal.Either$Right.fold(Either.java:175)
    at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
    at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:68)
    at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:46)
    at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
    at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
    at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
    at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
    at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:91)
    at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:55)
    at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
    at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:37)
    at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:65)
    at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:36)
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
    at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:77)
    at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:38)
    at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:94)
    at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:49)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:71)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:45)
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNonEmptySources(SkipEmptyWorkStep.java:177)
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:81)
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:53)
    at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
    at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
    at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:36)
    at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:23)
    at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:75)
    at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:41)
    at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.lambda$execute$2(ExecuteWorkBuildOperationFiringStep.java:66)
    at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:66)
    at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:38)
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:32)
    at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:293)
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:21)
    at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
    at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
    at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:47)
    at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:34)
    at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:64)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:146)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:135)
    at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
    at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:331)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:318)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.lambda$execute$0(DefaultTaskExecutionGraph.java:314)
    at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:80)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:314)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
    at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)

I can add GradleProjectData to RewriteClassLoader loadFromParent list but then I get a similar error about ProjectParser and then types in its method signatures and so on...

I'd really need some kind of guidance at this point :-)

BoykoAlex commented 1 year ago

@sambsnyd NVM :-) Fixed the isolated classpath issue with the new ProjectParser (LinkageErrors). Had to add the parser from tooling model to resolved dependencies in ResolveRewriteDependenciesTask.

All tests pass now - should be good for review very soon.

openrewrite / rewrite-gradle-plugin