Improve the performance of the server

[jdneo]

The discussion was originally raised at https://github.com/microsoft/build-server-for-gradle/pull/113#issuecomment-1852842306

What I do think is that this data might have to be cached. Have you tried the plugin on a big project? e.g. Gradle itself. It takes about a minute to respond to build/initialize (which I think should be instant and is slow because the source sets are queried in response to this request but not yet needed). Then, after a compile, the source sets are all queried again. It would be good to delay the building of the sourcesets until the client requests them and do they need to be built again after every compile?

[jdneo]

Hi @Arthurm1, let's continue the perf discussion here.

I can add some background why after a compile, the source sets are all queried again.

The reason to do that is because some gradle plugin may generate source code into a specific folder that is unknown by using the api of SourceSet. For example, the protobuf plugin will add generated sources to the input of the javaCompile task: https://github.com/google/protobuf-gradle-plugin/blob/42b7ee3e2a2ba05cc8922a9702a6d520cf2f2f12/src/main/groovy/com/google/protobuf/gradle/ProtobufPlugin.groovy#L483.

Query the sourceset again after compilation makes sure that this kind of generated code will be discovered automatically, though maybe it could be too 'heavy'.

[Arthurm1]

Does that require compilation though?

If I setup a gradle project with...

plugins {
    id 'java'
    id 'java-gradle-plugin'
    id "com.google.protobuf" version "0.9.4"
}

The ...\build-server-for-gradle\testProjects\protobuf\build\generated\source\proto\main\java directory seems to appear in gradleSourceSet#getSourceDirs. It seems to be populated by SourceSet#getJava#getSrcDirs. And doesn't seem to require a compile - unless I'm missing something. I guess there might be an issue in that it populates into sourceDirs instead of generatedSrcDirs

[Arthurm1]

One way to shift the protobuf dir from srcDirs into the generated dirs might be...

  private void moveProtobufGeneratedSourceDirs(Project project,
      Set<File> srcDirs, Set<File> generatedSrcDirs) {
    Object protobufPlugin = project.getExtensions().findByName("protobuf");
    if (protobufPlugin != null) {
       try {
        Method getGeneratedFilesBaseDirMethod = protobufPlugin.getClass().getMethod("getGeneratedFilesBaseDir");
        Object baseDirObj = getGeneratedFilesBaseDirMethod.invoke(protobufPlugin);
        if (baseDirObj instanceof String) {
          Path basePath = new File((String) baseDirObj).toPath();
          // does the source dir start with the Protobuf Base dir
          // if so, shift it from srcDirs to generatedSrcDirs
          Set<File> dirsToMove = srcDirs.stream().filter(src -> src.toPath().startsWith(basePath)).collect(Collectors.toSet());
          srcDirs.removeIf(src -> dirsToMove.contains(src));
          generatedSrcDirs.addAll(dirsToMove);
        }
      } catch (NoSuchMethodException | SecurityException | IllegalAccessException
          | IllegalArgumentException | InvocationTargetException e) {
        // ignore
      }
    }
  }

[jdneo]

Does that require compilation though?

They changed their implementation. If I remember correctly, versions below 0.9 (0.8.x) won't append the generate output folder into sources.

One way to shift the protobuf dir from srcDirs into the generated dirs might be

I've considered that as well, but finally gave up this approach. Because it tightly binds the implementation of protobuff plugin. And cannot handle other kinds of code generation plugins which use same approaches.

[mbruce]

This remains quite slow. Often taking an hour when loading a large project. Any updates?