Implement support for codegen with multiple languages x multiple resolves

[Eric-Arellano]

The second part of this comment https://github.com/pantsbuild/pants/issues/14484#issuecomment-1057472521, along with the solutions proposed below it to use configurations + a generators field.

Implementing this will look like:

1. Support configurations generically
2. Add a generators field, which also solves https://github.com/pantsbuild/pants/issues/14041
3. Document this. It's not very intuitive, although does solve the problem & is consistent with generic Pants mechanisms.

[stuhood]

It's also worth noting that even before configurations are supported, duplicate targets can be created manually.

[stuhood]

Hm: another very important issue (which may have made closing #14041 premature, sorry @tdyas) is that injected runtime dependencies can conflict between implementations.

For example, enabling both pants.backend.codegen.protobuf.python and pants.backend.experimental.codegen.protobuf.scala in https://github.com/pantsbuild/example-codegen/pull/10 causes JVM compilation to fail with:

12:47:43.36 [ERROR] 1 Exception encountered:

  ClasspathSourceMissing: No JVM classpath providers (from: CompileJavaSourceRequest, CompileScalaSourceRequest, CoursierFetchRequest, DeployJarClasspathEntryRequest, JvmResourcesRequest) were compatible with the combination of inputs:
  * 3rdparty/python#protobuf    (python_requirement)

...which is valid (although you could also argue that a target should ignore dependencies that it doesn't understand).

It might be the case that these injected dependencies should be "classified" / scoped (via #12794)... or that multiple target types (or at least instantiations, maybe via target generators) are necessary.

In the v2 engine experiments, this was addressed by declaring dependencies on fields (called "configurations" at that point) rather than on targets itself (i.e. target(.., sources=Python(.., dependencies=..))). In that model, codegen would generate a SourcesField with certain dependencies, rather than injecting them independently. But that also has the downside of forcing codegen to happen in order to compute dependencies.

[tdyas]

And also complicated if users want to run Java and Scala protobuf generators in the same code base.

[gniemann]

I am also running into this issue using Pants 2.15.0rc6, same error as above when trying to codegen protobuf with both Python and Java:

ClasspathSourceMissing: No JVM classpath providers (from: CompileJavaSourceRequest, CoursierFetchRequest, DeployJarClasspathEntryRequest, JvmResourcesRequest) were compatible with the combination of inputs:
  * 3rdparty/python:reqs#protobuf       (python_requirement)

I have a minimal reproduction of the issue here: https://github.com/gniemann/pants-proto-test

[jlclx]

After running into the same issue as in https://github.com/gniemann/pants-proto-test, I've attempted to fix it for java/python repos.

I narrowed it down to filtering out the python_requirements from the classpath dependencies. To do that I use PythonRequirementDependenciesField to ignore them via the classpath_dependency_requests rules in https://github.com/jlclx/pants/commit/d67078b43c7edbbae9d05f1e75f5c36b22ad4e10

This builds working jars with both the required protobuf/grpc artifacts and generated protobuf modules, without error, but it fails pre-commit hooks with

pants.engine.internals.dep_rules.DependencyRuleActionDeniedError: src/python/pants/jvm/compile.py has 1 dependency violation:

  * src/python/pants/jvm -> src/python/pants/backend/BUILD[!*] : DENY
    python_sources src/python/pants/jvm/compile.py -> python_sources src/python/pants/backend/python/target_types.py

Presumably because this sort of cross backend dependency is frowned upon, which makes sense considering it's very specific.

I tried various combinations of filtering via JVM target_types fields instead of Python ones, but was unsuccessful in creating a working jar. Presumably it needs Java/Scala/etc specific fields as well? But I'm unsure if that's going too specific again, but in the wrong direction, as I believe some java projects add non-code files to the classpath?

I'm unsure of where to go from here being a few hours into reading the code base, maybe modify codegen itself?