Generate error, Argument list too long.

[o2e]

I have a huge project that contains 14,982 proto files and I get an "Argument list too long" error when I compile it. Is there any way to deal with it. I can't split this project at the moment, please help me or give me a suggestion. Thank you!

Execution failed for task ':generateProto'.
> java.io.IOException: Cannot run program "/Users/Smile/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protoc/3.21.9/b5370f325a89e44a469213e65689c341f7d2a183/protoc-3.21.9-osx-x86_64.exe": error=7, Argument list too long

[rougsig]

Hello. Could you please provide a bit more details.

1 plugin version 2 used plugins 3 Windows version

[rougsig]

Also we have special logic for splitting long commands. Maybe sources can help to understand, what is going wrong.

https://github.com/google/protobuf-gradle-plugin/blob/master/src/main/groovy/com/google/protobuf/gradle/GenerateProtoTask.groovy#L178

It would be perfect if you can provide reproducible project.

[o2e]

I use the "protocol buffers(222.4345.14)" plugin bundled with idea, gradle plugin: "id("com.google.protobuf") version "0.9.1"," protoc version: "com.google.protobuf:protoc:3.21.9"" System version: MacOS X 10.15.7

Can't provide the project, but I can help to verify some ideas. project's proto file file is very very large, about 5MB in total. So far I have guessed to divide into several projects and compile them successfully. I think I can write code to generate many random proto files and try to reproduce them.

[rougsig]

Could you please write absolute path size to your proto dir? Like /Users/Smile/projects/myProject/app/src/main/proto -> 50

[o2e]

I don't quite understand what I should do, can you be more specific? Please excuse the slow response as I cannot access the external network recently.

Could you please write absolute path size to your proto dir? Like /Users/Smile/projects/myProject/app/src/main/proto -> 50

[rougsig]

Your proto files placed in some directory. In argument in cli, plugin pass <PATH_TO_PROTO_FILE>/myMessage.proto. I want to known the length of PATH_TO_PROTO_FILE. Just to calculate how much approximately you have the length of the argument.

[karthikrg]

@rougsig My invocation is ~1.7MB long. It is a legacy project which I am converting from another IDL to proto with tight interdependencies between schemas, so it is kinda hard to split it up into multiple protoc invocations.

[karthikrg]

The cleaner solution actually seems to be to support the @filename option of protoc and append all args to the file. This would be similar to the approach adopted by the maven plugin in https://github.com/xolstice/protobuf-maven-plugin/pull/42/ .

@rougsig would you be open to an intermediary hardcoded solution as proposed in #653 fast followed by a clean solution using @filename?

[rougsig]

@karthikrg Hardcoded solution is fine.

[o2e]

Have you guys reproduced the problem yet? I'm sorry I haven't been able to help you guys out because of the long time without internet, and in fact I don't have the time right now.

I think your idea of making a txt file with the paths of all .proto files to pass to the protoc program seems good. The list of .proto files could be automatically generated by gradle-plugin, split by newlines and written to a txt file. The logic wouldn't be much to change, just collect all the .proto files according to the original logic, except that it might require support from the protoc program

[rougsig]

@o2e yeah it will works too. Actually we already have splitting logic for Windows. For other OS it was disabled. Changes from @karthikrg enable that splitting.

[o2e]

👍

[ejona86]

This should be fixed by #653