Unable to use Protobuf primitive wrapper types

[Zambito1]

I am trying to deserialize Protobuf messages that use the Any type, and uses some of the wrapper types for compatibility with primitive types. Particularly those found here. Here is a minimal example that demonstrates the issue I am running into. The following works:

person.proto:

syntax = "proto3";

import "google/protobuf/any.proto";

message PersonWorks {
  string name = 1;
  bool is_tall = 2;
}

message PersonBroken {
  google.protobuf.Any name = 1;
  google.protobuf.Any is_tall = 2;
}

config.yaml:

input:
  generate:
    mapping: |
      root.name = fake("name")
      root.isTall = random_int(max:1) == 1
    interval: 1s
    count: 10

pipeline:
  processors:
    - protobuf:
        operator: from_json
        message: "PersonWorks"
        import_paths: [.]
    - protobuf:
        operator: to_json
        message: "PersonWorks"
        import_paths: [.]

(worth noting that the round trip maps false to null, but that is not the issue I am trying to cover here)

But this does not: config.yaml:

input:
  generate:
    mapping: |
      root.name = fake("name")
      root.isTall = random_int(max:1) == 1
    interval: 1s
    count: 10

pipeline:
  processors:
    - mapping: |
        root.name = {
          "@type": "type.googleapis.com/google.protobuf.StringValue",
          "value": this.name
        }

        root.isTall = {
          "@type": "type.googleapis.com/google.protobuf.BoolValue",
          "value": this.isTall
        }
    - protobuf:
        operator: from_json
        message: "PersonBroken"
        import_paths: [.]
    - catch:
        - log:
            level: ERROR
            message: "Error: ${!error()}"
        - mapping: |
            root = deleted()
    # - protobuf:
        # operator: to_json
        # message: "PersonBroken"
        # import_paths: [.]

It fails with the following log:

ERRO Error: failed to unmarshal JSON message 'PersonBroken': proto: (line 1:20): unable to resolve "type.googleapis.com/google.protobuf.BoolValue": "not found"  @service=benthos label="" path=root.pipeline.processors.2.catch.0

If I remove the BoolValue field, I get the same error with the StringValue field instead. It seems to be related to this issue: https://github.com/golang/protobuf/issues/1156 but I tried adding

import (
     // ...
    _ "google.golang.org/protobuf/types/wrapperspb"
)

xor

import (
     // ...
    _ "github.com/golang/protobuf/ptypes/wrappers"
)

to internal/impl/protobuf/processor_protobuf.go, and neither worked.

[Zambito1]

Surprisingly though this actually works:

person.proto:

syntax = "proto3";

import "google/protobuf/wrappers.proto";

message PersonWorks {
  string name = 1;
  bool is_tall = 2;
}

message PersonBroken {
  google.protobuf.StringValue name = 1;
  google.protobuf.BoolValue is_tall = 2;
}

config.yaml:

input:
  generate:
    mapping: |
      root.name = fake("name")
      root.isTall = random_int(max:1) == 1
    interval: 1s
    count: 10

pipeline:
  processors:
    - protobuf:
        operator: from_json
        message: "PersonBroken"
        import_paths: [.]
    - catch:
        - log:
            level: ERROR
            message: "Error: ${!error()}"
        - mapping: |
            root = deleted()
    - protobuf:
        operator: to_json
        message: "PersonBroken"
        import_paths: [.]

This is with the latest on main rather than with my added imports too. The issue is specifically related to using these types with google.protobuf.Any

[mihaitodor]

Hey @Zambito1 I think you'll have to configure import_paths and add in there the path to wherever you install protoc on your platform (guess the platform doesn't really matter much in this case, since you only need the .proto files). You can get the latest release here: https://github.com/protocolbuffers/protobuf/releases/latest. It should contain an include folder which has the files you need.

PS: Converting to a discussion as mentioned in #2026.