Issues running on Windows

[rocketraman]

Ok, I don't use Windows myself (ugh), but some of my coworkers do. There are issues running this on Windows:

1. The compiler is run with an invalid path that contains a leading "/". Here is the debug output from Gradle:

11:48:22.846 [INFO] [org.gradle.process.internal.DefaultExecHandle] Starting process 'command 'C:\jdk1.8.0_191\bin\java.exe''. Working directory: H:\source\myproject\lib-proto Command: C:\jdk1.8.0_191\bin\java.exe -Dfile.encoding=windows-1252 -Duser.country=US -Duser.language=en -Duser.variant -jar /C:/Users/Me/.gradle/caches/modules-2/files-2.1/com.github.marcoferrer.krotoplus/kroto-plus-compiler/0.1.3/620b2a278b4d4beed80320bb4800339967f850d7/kroto-plus-compiler-0.1.3.jar H:\source\myproject\lib-proto/src/main/proto H:\source\myproject\lib-proto\build/extracted-include-protos/main -default-out H:\source\myproject\lib-proto\build\generated\source\proto\main\kotlin -writers 3 -StubOverloads -o|H:\source\myproject\lib-proto\build\generated\source\proto\main\kotlin|-coroutines -MockServices -o|H:\source\myproject\lib-proto\build\generated\source\proto\main\kotlin -ProtoTypeBuilder
...
11:48:22.874 [ERROR] [system.err] Error: Unable to access jarfile /C:/Users/Me/.gradle/caches/modules-2/files-2.1/com.github.marcoferrer.krotoplus/kroto-plus-compiler/0.1.3/620b2a278b4d4beed80320bb4800339967f850d7/kroto-plus-compiler-0.1.3.jar

Note the leading slash on the -jar argument.

2. I don't know if this is an issue in this plugin or in the upstream protobuf gradle plugin, but when adding the kroto code gen plugin to the protobuf plugin, the build fails with the error:

Execution failed for task ':generateProto'.
> protoc: stdout: . stderr: --kroto_out: protoc-gen-kroto: %1 is not a valid Win32 application.

Looking at the --debug logs, it looks like a --plugin parameter is addd to the call to protoc, with the following value:

 --plugin=protoc-gen-kroto=C:\Users\Me\.gradle\caches\modules-2\files-2.1\com.github.marcoferrer.krotoplus\protoc-gen-kroto-plus\0.1.3\3c41d071d04c8558822447643894490e33a857b7\protoc-gen-kroto-plus-0.1.3-jvm8.jar

but protoc is assuming the jar file is an executable it can run. I have tried associating ".jar" files with "java" (and this seems to have worked), but for some reason when running via protoc, the same "not a valid Win32 application" error still happens.

[rocketraman]

This might be related to the second item mentioned above: https://github.com/google/protobuf-gradle-plugin/issues/268

[marcoferrer]

It looks like the problem is that the script thats embedded in the jar via spring boot is not executable in windows. I dont think there is any issue with the gradle protobuf plugin.

The fix is probably going to involved distributing another artifact that is executable on windows. I dont have any way to test it out at the moment. But I think if you point the protobuf gradle plugin to an executable bat script that executes the jar via java it might work in mean time. This will give me a chance to work on the long term solution. I dont have much experience with windows batch scripts but try this if possible

@if "%DEBUG%" == "" @echo off

set CMD_LINE_ARGS=%*
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe

"%JAVA_EXE%" -jar C:\Users\Me\.gradle\caches\modules-2\files-2.1\com.github.marcoferrer.krotoplus\protoc-gen-kroto-plus\0.1.3\3c41d071d04c8558822447643894490e33a857b7\protoc-gen-kroto-plus-0.1.3-jvm8.jar %CMD_LINE_ARGS%

build.gradle

protobuf {
  protoc {
    artifact = 'com.google.protobuf:protoc:3.6.1'
  }
  plugins {
    kroto {
      path = "absolute/path/to/script.bat"
    }
  }
  ...
}

If that works then I can come up with a gradle task to hold you over in the mean time.

[rocketraman]

@marcoferrer Thanks, the batch script ~solves~ works around problem 2. However, the build gets further and still encounters problem 1. The specific error is:

18:36:04.214 [INFO] [org.gradle.process.internal.DefaultExecHandle] Successfully started process 'command 'C:\jdk1.8.0_191\bin\java.exe''
18:36:04.214 [DEBUG] [org.gradle.process.internal.ExecHandleRunner] waiting until streams are handled...
18:36:04.235 [ERROR] [system.err] Error: Unable to access jarfile /C:/Users/Me/.gradle/caches/modules-2/files-2.1/com.github.marcoferrer.krotoplus/kroto-plus-compiler/0.1.3/620b2a278b4d4beed80320bb4800339967f850d7/kroto-plus-compiler-0.1.3.jar

I think that leading slash is the issue.

[marcoferrer]

Whats confusing for me is that the snippet you posted is referencing the old stand alone compiler "kroto-plus-compiler-0.1.3.jar" and not the protoc plugin "protoc-gen-kroto-plus-0.1.3-jvm8.jar" . If your using the kroto-plus gradle plugin for 0.1.3 you can disable it. If possible can you post a snippet of your gradle config?

[rocketraman]

Aha, looks like I still had a krotoPlus { ... } block from an earlier version. I've removed it and gotten everything working properly on Linux again. Now on to the next Windows issue:

Execution failed for task ':generateProto'.
> protoc: stdout: . stderr: \source\myproject\krotoPlusConfig.json:H:\source\myproject\build/generated/source/proto/main/java/: No such file or directory

The krotoplus config is:

outputSubDir = "java"

I've used the --debug output to obtain the actual protoc command line, and the argument passed was as follows:

--kroto_out=ConfigPath=H:\source\myproject\krotoPlusConfig.json:H:\source\myproject\build/generated/source/proto/main/java

so you can see that the first : in H: is being mistaken for the separator between the config and output directory, which is why it is looking for directory \source\myproject\krotoPlusConfig.json:H:\source\myproject\build/generated/source/proto/main/java/. If I manually edit the argument to look like this:

kroto_out=ConfigPath=\source\myproject\krotoPlusConfig.json:H:\source\myproject\build/generated/source/proto/main/java

then the protoc command exits without an error. Note the second H: after the : delimiter does seem to be necessary -- oddly, removing the second H: results in another No such file or directory error.

[rocketraman]

A workaround seems to be to do this to relativize the config path before passing it to protoc:

option "ConfigPath=${file('.').toPath().toAbsolutePath().relativize(krotoConfig.toPath().toAbsolutePath()).toString()}"

Messy, but it seems to work.

UPDATE: Use file(System.getProperty("user.dir")) instead of file('.') -- see below.

[marcoferrer]

If this can hold you over in the mean time I can work on getting better windows support into an upcoming release.

[rocketraman]

If this can hold you over in the mean time I can work on getting better windows support into an upcoming release.

Yup, it can! I've got workarounds for all issues identified. Looking forward to the next release.

[marcoferrer]

So I think I have a good solution to this problem but wanted to hold off on releasing it too early. I didnt want to hold up the coroutines update but the next release after that should have something in it to address this.

[marcoferrer]

@brandmooffin as per our convo, You can use the documentation from here to implement the windows support outlined in this issue. deployment-script-customization

[marcoferrer]

@brandmooffin We can try doing what the maven protobuf plugin does and wrap the jar using this. https://github.com/poidasmith/winrun4j

[marcoferrer]

The other half of this issue is the fact that the protoc --xxx_out flag uses : as a delimiter for options. Since protoc 3.2.0 a new flag was made available for passing plugin options (--xxx_opt). Support for this new flag was discussed in the following issue https://github.com/google/protobuf-gradle-plugin/issues/210. I currently have a PR open for adding support for this flag at https://github.com/google/protobuf-gradle-plugin/pull/290

[rocketraman]

A workaround seems to be to do this to relativize the config path before passing it to protoc:

option "ConfigPath=${file('.').toPath().toAbsolutePath().relativize(krotoConfig.toPath().toAbsolutePath()).toString()}"

Messy, but it seems to work.

One correction for anyone finding this from Google, and using these workarounds... this should be:

option "ConfigPath=${file(System.getProperty("user.dir")).toPath().toAbsolutePath().relativize(krotoConfig.toPath().toAbsolutePath()).toString()}"

The reason is that in Gradle file('.') returns the project directory, not the working directory. The path must be relative to the working directory, not the project directory.

[Abegemot]

after applying those changes I still unfortunately get an error:

Execution failed for task ':newproto:generateProto'.

protoc: stdout: . stderr: Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) Caused by: java.lang.NullPointerException at com.github.marcoferrer.krotoplus.generators.GeneratorContext$currentWorkingDir$2.invoke(GeneratorContext.kt:40)

And I'm at a lost to understand what shall I do, many thank's for any hint.

[marcoferrer]

@Abegemot Can you post a snippet of your current build configuration?

[Abegemot]

Sure, thank you so much, I really found your work not only very interesting but in my opinion you and the other people trying to bring protobuf to kotlin in a gentle and convenient way ought to be supported by jetbrains to bring up an official answer to such a technical cornerstone such as protobuf .

• the configuration file and bat file are exactly the same as in the examples.

protobuf{ protoc{ artifact=Deps2.protocc } plugins{ plugins{ id("kroto"){ path="H:\prg\artifactidfake\newproto\mykrotobat.bat" // } } } generateProtoTasks{ val krotoConfig = file("krotoPlusConfig.asciipb") // all().forEach{task-> task.inputs.files ( krotoConfig) task.plugins { id("kroto") { outputSubDir = "java" option("ConfigPath=${file(System.getProperty("user.dir")).toPath().toAbsolutePath().relativize(krotoConfig.toPath().toAbsolutePath()).toString()}")

                   }
               }
   }

}

}

[marcoferrer]

Looking at the stacktrace posted earlier, I think the null pointer is coming from this line

More specifically I think its the call to File(path).parentFile. It looks like the path isnt being relativized correctly. What output do you get when you run the following:

println("ConfigPath=${file(System.getProperty("user.dir")).toPath().toAbsolutePath().relativize(krotoConfig.toPath().toAbsolutePath()).toString()}")`?

Also could you give this a try?

option "ConfigPath=${krotoConfig.absolutePath.replace(rootProject.projectDir.path, "").drop(1)}"

[Abegemot]

Thank you that did the job, nevertheless unfortunately when I run the build I only get generated the messages but not the grpc calls, which is weird, and can't start up a server or do anything. I guess I'm doing something wrong but I'm clueless....

[Brainfree]

I managed to hack together a pure gradle solutions to fix the windows problem. It will have gradle fetch the plugin jar, copy it, then generate a batch file, where it'll point protoc to it.

configurations {
    kroto_conf
}

dependencies {
    kroto_conf "com.github.marcoferrer.krotoplus:protoc-gen-grpc-coroutines:$krotoplus_version:jvm8@jar"
    kroto_conf "com.github.marcoferrer.krotoplus:protoc-gen-kroto-plus:$krotoplus_version:jvm8@jar"
}

ext.kroto_gen_kroto_plus = new File("$buildDir/kroto/gen_kroto_plus.bat")
ext.kroto_gen_grpc_coroutines = new File("$buildDir/kroto/gen_grpc_coroutines.bat")

task setupKroto(type: Copy) {
    from configurations.kroto_conf
    into "$buildDir/kroto"

    ext.batch = { jar ->
        """
    @if "%DEBUG%" == "" @echo off

    set CMD_LINE_ARGS=%*
    set JAVA_HOME=%JAVA_HOME:"=%
    set JAVA_EXE=%JAVA_HOME%/bin/java.exe

    "%JAVA_EXE%" -jar $jar %CMD_LINE_ARGS%
    """.stripIndent()
    }

    doLast {
        kroto_gen_kroto_plus.text = batch("$buildDir/kroto/protoc-gen-kroto-plus-$krotoplus_version-jvm8.jar")
        kroto_gen_grpc_coroutines.text = batch("$buildDir/kroto/protoc-gen-grpc-coroutines-$krotoplus_version-jvm8.jar")
    }

}

protobuf {
    //download the proto compiler
    protoc {
        artifact = "com.google.protobuf:protoc:$protobuf_version"
    }

    generatedFilesBaseDir = "$projectDir/gen"

    //noinspection GroovyAssignabilityCheck
    plugins {
        grpc { artifact = "io.grpc:protoc-gen-grpc-java:$grpc_version" }
        coroutines {
            artifact = "com.github.marcoferrer.krotoplus:protoc-gen-grpc-coroutines:$krotoplus_version:jvm8@jar"
            //https://github.com/marcoferrer/kroto-plus/issues/6#issuecomment-432782481
            if(Os.isFamily(Os.FAMILY_WINDOWS)) {
                path = kroto_gen_grpc_coroutines.absolutePath
            }
        }
        kroto {
            artifact = "com.github.marcoferrer.krotoplus:protoc-gen-kroto-plus:$krotoplus_version:jvm8@jar"
            //https://github.com/marcoferrer/kroto-plus/issues/6#issuecomment-432782481
            if(Os.isFamily(Os.FAMILY_WINDOWS)) {
                path = kroto_gen_kroto_plus.absolutePath
            }
        }
    }

    generateProtoTasks {

        def krotoConfig = file("krotoPlusConfig.asciipb") // Or .json

        all().each { task ->

            //setup kroto for windows
            task.dependsOn setupKroto

            // Adding the config file to the task inputs lets UP-TO-DATE checks
            // include changes to configuration
            task.inputs.files krotoConfig

            task.plugins {
                grpc {}
                coroutines {}
                kroto {
                    // The extendable-messages generator needs the outputSubDir
                    // to be the same as 'task.builtins.java.outputSubDir' since
                    // it relies on the insertion_point api from protoc.
                    outputSubDir = "java"
                    //https://github.com/marcoferrer/kroto-plus/issues/6#issuecomment-457261961
                    option "ConfigPath=${krotoConfig.absolutePath.substring(krotoConfig.absolutePath.lastIndexOf(":") + 1)}"
                }
            }
        }
    }
}

[Brainfree]

Also, it seems a better solution for the ConfigPath problem is to simply strip away the drive ("C:"), then you can use absolute paths. I had a problem with the above solution in multi-project setup, where without absolute paths, it would not be able to find the config, if I assembled from the parent project. Solution:

task.plugins {
    kroto {
        option "ConfigPath=${krotoConfig.absolutePath.substring(krotoConfig.absolutePath.lastIndexOf(":") + 1)}"
    }
}

[rocketraman]

Also, it seems a better solution for the ConfigPath problem is to simply strip away the drive ("C:"), then you can use absolute paths.

That only works if you know everything is on the c:\ drive.

[mattdkerr]

I'm using val krotoConfig = File("${project.name}/krotoPlusConfig.yml") for loading the file in Windows, and it works fine. It starts relative to the root project, so this is fairly straight-forward.

[marcoferrer]

With 0.6.0 released, there is now an executable windows artifact available.

[fitzsia2]

Hello. I'm using kroto-generator in an Android project and I think that I still have this issue on Windows. When I build the project I receive:

Caused by: java.lang.IllegalStateException: Config file does not exist. 'C:\Users\Andrew\StudioProjects\my-app\C'

I think it may be related to my folder structure.

I have the following setup:
Project:

my-app
⨽ libraries
  ⨽ networking
    ⨽ build.gradle

// build.gradle
protobuf {
    protoc {
        artifact = Deps.protoc
    }
    plugins {
        grpc {
            artifact = Deps.grpcGenerator
        }
        javalite {
            artifact = Deps.javaLiteGenerator
        }
        kroto {
            artifact = Deps.krotoGenerator
        }
    }
    generateProtoTasks {
        def krotoConfig = file("krotoconfig.asciipb")
        all()*.plugins {
            javalite {}
        }
        ofNonTest()*.plugins {
            grpc {
                // Options added to --grpc_out
                option 'lite'
            }
            kroto {
                outputSubDir = "java"
                option "ConfigPath=$krotoConfig"
            }
        }
    }
}

I found that if I use @Brainfree's solution for Windows, then it works, but I expected that release 0.6.0 would resolve this.

Please let me know if there is some more information I can provide.

[marcoferrer]

@fitzsia2 So originally the kroto+ plugin was not able to execute on windows environments. Version 0.6.0 introduced a native executable to mitigate this issue. But the problem still exists where the : in the path to the configuration file is mistaken by the protobuf compiler as an option delimiter. Currently in windows you still need to perform a small amount of path transformation to the configuration file to resolve correctly. You can see here how the project still relies on it.

Considering you had issues discovering this means that its not immediately clear to users and should be better documented. Thank you for bringing this to light.

                kroto {
                    outputSubDir = "java"
                    if(osdetector.os == "windows") {
                        // We want to relativize the configuration path
                        // because absolute paths cause issues in windows
                        // environments
                        option "ConfigPath=${krotoConfig.absolutePath.replace(System.getProperty("user.dir"), "").drop(1)}"
                    } else {
                        option "ConfigPath=$krotoConfig"
                    }
                }

https://github.com/marcoferrer/kroto-plus/blob/2b0b447d47f14da93b536ef238b379eb4cc6075c/build.gradle#L126

[AlexOreshkevich]

Hi guys,

we just experienced somethings similar using Maven (protobuf-maven-plugin):

 Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.0:compile-custom (kroto-plus) on project entity-api: protoc failed to execute because: 'nativePluginParameter' contains illegal characters

We configured config path just like that:

<pluginParameter>
  ConfigPath=${project.basedir}/kroto.plus.config.asciipb
</pluginParameter>

(maven multimodule project)

Do we have some workaround for Maven & Windows like we have for Gradle please?