Protobuf external references breaking inside docker container

[gr8routdoors]

Description

Registry Version: 2.4.11.Final and 2.4.12.Final Persistence type: Tested in both kafkasql and mysql

I am trying to take advantage of schema references for my protobuf schemas using the /apis/registry/v2/groups/{groupId}/artifacts?ifExists=RETURN_OR_UPDATE resource, and am getting back a 409 "unable to find" for the schemas references inside of the docker container. As expected, I am registering the child references first. Oddly, I went to debug this with the jar, using the same build tag and the schemas register fine with the same requests. I have verified the broken container behavior both locally and an integrated environment, to rule out any strange local issues.

Environment

OSX - both dockerized and bare metal k8s

Interacting with apicurio 2.4.11.Final and 2.4.12.Final directly via the v2 API, with the following settings enabled:

registry.rules.global.compatibility=FULL_TRANSITIVE
registry.rules.global.validity=SYNTAX_ONLY
registry.rest.artifact.deletion.enabled=true

Steps to Reproduce

1. Copy the protobuf schemas below into a local directory structure that matches up with their includes
2. Submit them in the correct order to the /apis/registry/v2/groups/{groupId}/artifacts?ifExists=RETURN_OR_UPDATE API (grandchild, child, parent) with refs (see requests below).
3. See the 409 error

Requests

Grandchild

Headers "Content-Type", "application/create.extended+json" "X-Registry-ArtifactType", "PROTOBUF" "X-Registry-ArtifactId", "foo.bar.test.external.child.AGrandchildV1" "X-Registry-Name", "external/child/a_grandchild.proto"

Body content: "<external/child/a_grandchild.proto content>" references: []

Child

Headers "Content-Type", "application/create.extended+json" "X-Registry-ArtifactType", "PROTOBUF" "X-Registry-ArtifactId", "foo.bar.test.external.child.AChildV1" "X-Registry-Name", "external/child/a_child.proto"

Body content: "<external/child/a_child.proto content>" references: [{GroupId: "default", ArtifactId: "foo.bar.test.external.child.AGrandchildV1", Name: "external/child/a_grandchild.proto", Version: "1"}]

Parent

Headers "Content-Type", "application/create.extended+json" "X-Registry-ArtifactType", "PROTOBUF" "X-Registry-ArtifactId", "foo.bar.test.external.AParentV1" "X-Registry-Name", "external/a_parent.proto"

Body content: "<external/a_parent.proto content>" references: [{GroupId: "default", ArtifactId: "foo.bar.test.external.child.AChildV1", Name: "external/child/a_child.proto", Version: "1"}]

Expected vs Actual Behaviour

Expected: Schemas are successfully stored to the registry. Actual: A 409 SchemaException is returned

Logs

"SchemaException: unable to find external/child/a_child.proto\n  searching 1 proto paths:\n    /\n  for file foo/bar/test/external/default.proto\nunable to resolve AChildV1\n  for field child (foo/bar/test/external/default.proto:19:3)\n  in message foo.bar.test.external.AParentV1 (foo/bar/test/external/default.proto:11:1)"

Strange thing for me is the nested path it's showing on the virtual filesystem vs the actual name given for the artifact (the name submitted for the artifact matches the import path in the proto). Maybe I'm submitting incorrectly?

[apicurio-bot[bot]]

Thank you for reporting an issue!

Pinging @jsenko to respond or triage.

[gr8routdoors]

external/a_parent.proto content:

syntax = "proto3";

package foo.bar.test.external;

import "external/child/a_child.proto";

// Used to test a nested protobuf message
message AParentV1 {
  // Uuid of the received content
  string item_id = 1;
  // Name of received content
  string item_name = 2;
  // Child message
  AChildV1 child = 3;
}

[gr8routdoors]

external/child/a_child.proto content:

syntax = "proto3";

package foo.bar.test.external.child;

import "external/child/a_grandchild.proto";

// Used to test a nested protobuf message
message AChildV1 {
  // Uuid of the received content
  string item_id = 1;
  // Name of received content
  string item_name = 2;
  AGrandchildV1 grandchild = 3;
}

[gr8routdoors]

external/child/a_grandchild.proto content:

syntax = "proto3";

package foo.bar.test.external.child;

// Used to test a nested protobuf message
message AGrandchildV1 {
  // Uuid of the received content
  string item_id = 1;
  // Name of received content
  string item_name = 2;
}

[EricWittmann]

Thank you for the reproducer. I will try to follow your instructions and see what I get, under different configs.

[EricWittmann]

Quick note: in your sample reproducer above the Parent references are wrong - you have an artifactId in there of riot.rebus.test.external.child.AChildV1 when you meant it to be foo.bar.test.external.child.AChildV1. I've changed it in your description. It was clearly just a sample data rename thing, and not the root/cause of your issue.

Another side note: I recommend you enable the Integrity Rule if you are going to use references.

[EricWittmann]

In my initial testing with your sample data above, I am able to successfully register a_grandchild.proto and a_child.proto but registering a_parent.proto (the last one) with this error:

{
    "causes": [],
    "message": "Syntax violation for Protobuf artifact.",
    "error_code": 409,
    "detail": "SchemaException: unable to find external/child/a_child.proto\n  searching 1 proto paths:\n    /\n  for file foo/bar/test/external/default.proto\nunable to resolve AChildV1\n  for field child (foo/bar/test/external/default.proto:19:3)\n  in message foo.bar.test.external.AParentV1 (foo/bar/test/external/default.proto:11:1)",
    "name": "RuleViolationException"
}

[EricWittmann]

I think the problem here is that you're trying to reference Child from Parent but they aren't in the same package. Parent is in the foo.bar.test.external package and it has a field like this:

AChildV1 child = 3;

I think that needs to be this:

foo.bar.test.external.child.AChildV1 child = 3;

Because AChildV1 is in a different package from AParentV1.

I'm testing this now...

[EricWittmann]

OK confirmed - if I change the package of parent to be the same as the package of the child and grandchild, then everything works.

Also if I change the child reference as described above it works.

[EricWittmann]

I've tested this locally first in my local dev environment (i.e. run from the JAR) and also run from a docker container. Behavior seems the same to me.

[gr8routdoors]

Yeah, my contrived test example was bad. (Apologies!) It worked for me as well once I fixed the reference to the child. The actual use case we had that triggered this was much more nested and complex. Let me try to put one together that actually triggers the bug and I'll report back.

[EricWittmann]

Ok no problem. I do understand how it can be hard to distill a complex bug down into a simple reproducer. I appreciate the effort to do so.

That said, if there is a case where it works when running from the JAR but fails when running from the container, I will be at a loss as to they why. :)

I'm excited though.

[gr8routdoors]

I'm still trying to pinpoint the root cause - but I do have this snippet:

Syntax violation for Protobuf artifact. 409 SchemaException: unable to find schemas/foo/dimensions/v1/content.proto\n  searching 1 proto paths:\n    /\n  for file foo/content/v1/default.proto

It looks to me like something is off with how paths are getting assembled for the either ProtobufContentValidator or ProtobufDereferencer, as these are the two that are calling FileDescriptorUtils.ProtoFileToFileDescriptor(ProtoFileElement) with no file name. Unfortunately, I do yet not know enough about this flow to speculate further.