Closed sleipnir closed 2 months ago
I can't seem to locate eigr/functions/protocol/actors/extensions.proto
to attempt to reproduce this locally.
I tried removing import eigr/functions/protocol/actors/extensions.proto
and changing the line string name = 1 [(.eigr.functions.protocol.actors.actor_id) = true];
to string name = 1;
to see if I can see the error without that file, and it reflected properly.
Can you direct me to the project that contains the extensions.proto
file?
Noting here that your error is a mismatch on the builder trying to derive types from a FileDescriptorProto
, which is unusual as FileDescriptorProto
is normally the payload you get from a reflecting server and not a descriptor output from protoc
, at least as far as I know.
Noting here that your error is a mismatch on the builder trying to derive types from a
FileDescriptorProto
, which is unusual asFileDescriptorProto
is normally the payload you get from a reflecting server and not a descriptor output fromprotoc
, at least as far as I know.
Hi @mjheilmann, thanks for responding. This occurs because the program in question generates gRPC contracts and servers on the fly.
Here is the project https://github.com/eigr/spawn
And here you find the .proto in question. https://github.com/eigr/spawn/blob/feat/embedded-grpc/priv/protos/eigr/functions/protocol/actors/extensions.proto
To run and reproduce locally use the make run-proxy-local-3 task on the branch feat/embedded-grpc.
This PR will give more context https://github.com/eigr/spawn/pull/302
@mjheilmann any workaround?
@sleipnir
from make run-proxy-local-3
** (Mix) Could not start application proxy: exited in: Proxy.Application.start(:normal, [])
** (EXIT) an exception was raised:
** (RuntimeError) Failed to start Proxy Application: {:error, {:shutdown, {:failed_to_start_child, Proxy.Supervisor, {:shutdown, {:failed_to_start_child, Sidecar.Supervisor, {:shutdown, {:failed_to_start_child, Sidecar.ProcessSupervisor, {:shutdown, {:failed_to_start_child, Statestores.Supervisor, {%DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 20997ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n 1. Ensuring your database is available and that you can connect to it\n 2. Tracking down slow queries and making sure they are running fast enough\n 3. Increasing the pool_size (although this increases resource consumption)\n 4. Allowing requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error, reason: :queue_timeout}, [{Ecto.Adapters.SQL, :raise_sql_call_error, 1, [file: ~c"lib/ecto/adapters/sql.ex", line: 1054, error_info: %{module: Exception}]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1693]}, {Ecto.Adapters.SQL, :execute_ddl, 4, [file: ~c"lib/ecto/adapters/sql.ex", line: 1161]}, {Ecto.Migrator, :verbose_schema_migration, 3, [file: ~c"lib/ecto/migrator.ex", line: 755]}, {Ecto.Migrator, :lock_for_migrations, 4, [file: ~c"lib/ecto/migrator.ex", line: 563]}, {Ecto.Migrator, :run, 4, [file: ~c"lib/ecto/migrator.ex", line: 432]}, {Ecto.Migrator, :with_repo, 3, [file: ~c"lib/ecto/migrator.ex", line: 170]}, {Statestores.Migrator, :migrate, 1, [file: ~c"lib/statestores/migrator/migrator.ex", line: 14]}]}}}}}}}}}}
(proxy 0.0.0-local.dev) lib/proxy/application.ex:39: Proxy.Application.do_start/2
(kernel 8.5.4.2) application_master.erl:293: :application_master.start_it_old/4
make: *** [run-proxy-local-3] Error 1
There is extra setup and possible configuration, I don't think its meaningful for me to start running and debugging your application.
The stack trace you posted is in types_from_descriptor/1
, which in grpc-reflection
is called in 2 places:
service.descriptor()
mod.descriptor()
where mod
is a compiled and loaded module that matches the symbol string provided.These are the only places where we have an entry to our code that could crash as you posted. We expect that the return value from descriptor/0
on these modules is either a ServiceDescriptorProto
or a DescriptorProto
. Check your pb.ex
or equivalent files like this to see what your compiled descriptors look like, I wouldn't expect any of them to be FileDescriptorProto
. My guess is either you have a module whose descriptor is returning a bad type, or you have a module that collides with a proto symbol name and is being picked up by mistake.
There is extra setup and possible configuration, I don't think its meaningful for me to start running and debugging your application.
Yeah! Sorry for this!
I will try to debug this later But I expect a ServiceDescriptorProto type too.... Thanks for now @mjheilmann, I'll try to find the error with your valuable insights.
@mjheilmann It seems that I found the error and it is quite obvious in reality. The grpc-reflection library is based on the older grpc plugin while my proto was compiled using the protobuf-generate library. I did this because protobuf-generate is the only one that supports extensions and http transcoding. But it ends up generating an elixir file that is slightly different from the one expected by the reflection library.
I think the ideal would be to include support for both in the reflection library and the choice of compiler could be done in the module options. This should be relatively simple to implement and I'll try to submit a PR in the next few days to add this functionality if that's okay with you.
Adding support for https://github.com/drowzy/protobuf_generate is likely going to be a bigger PR than you would think. A lot of the parsing and iteration that grpc-reflection does is to simulate a filesystem and generate a set of file descriptors, because we don't have access to the files to get file descriptors directly. However protobuf_generate
is providing file descriptors directly. We should leverage this directly, and have an alternate builder to support it.
Instead of collecting service and field descriptors to assemble a file descriptor, we would take and reference the existing file descriptor and instead build the symbol lookup table directly from the contents of that descriptor. I think we could build a state object, which is the post-build runtime lookup structure.
If you are up for solving that, a PR would be very welcome
If you are up for solving that, a PR would be very welcome
Okay. Let me think a little about how I could resolve this properly. Since I "depend" on it I'm likely to dedicate time to it. Thanks for detailing the process a little more.
@sleipnir I played around with protobuf_generate
, it looks like it only generates theFileDescriptorProto
for the service definitions, the regular types expose the expected descriptors. The entry point to the builder module is the service. While it would not be elegant, I bet we can fiddle with the interface to the builder so that we can provide a package, name, and service descriptor instead of a service module; this would let us unwrap the ServiceDescriptor
and pass it in to the normal builder logic.
I attempted this in https://github.com/elixir-grpc/grpc-reflection/pull/37
@sleipnir I played around with
protobuf_generate
, it looks like it only generates theFileDescriptorProto
for the service definitions, the regular types expose the expected descriptors. The entry point to the builder module is the service. While it would not be elegant, I bet we can fiddle with the interface to the builder so that we can provide a package, name, and service descriptor instead of a service module; this would let us unwrap theServiceDescriptor
and pass it in to the normal builder logic.I attempted this in #37
I agree that this approach is actually simpler. Good!
I merged in those changes, @sleipnir. Please let me know if there are further issues around, if not we can close the issue
Thank you @mjheilmann I will test and get back with feedback
The reflection worked, but the call to the function did not work accordingly. I haven't identified the problem yet.
Follow logs:
🚀 ▶ grpcurl -v -plaintext 127.0.0.1:9980 describe io.eigr.spawn.example.Joe.Sum
io.eigr.spawn.example.Joe.Sum is a method:
rpc Sum ( .io.eigr.spawn.example.MyBusinessMessage ) returns ( .io.eigr.spawn.example.MyBusinessMessage ) {
option (.google.api.http) = { post: "/v1/sum", body: "*" };
}
🚀 ▶ grpcurl -v -plaintext 127.0.0.1:9980 describe io.eigr.spawn.example.Joe
io.eigr.spawn.example.Joe is a service:
service Joe {
rpc Ping ( .io.eigr.spawn.example.MyBusinessMessage ) returns ( .io.eigr.spawn.example.MyBusinessMessage ) {
option (.google.api.http) = { get: "/v1/ping" };
}
rpc Sum ( .io.eigr.spawn.example.MyBusinessMessage ) returns ( .io.eigr.spawn.example.MyBusinessMessage ) {
option (.google.api.http) = { post: "/v1/sum", body: "*" };
}
}
🚀 ▶ grpcurl -v -plaintext 127.0.0.1:9980 describe .io.eigr.spawn.example.MyBusinessMessage
io.eigr.spawn.example.MyBusinessMessage is a message:
message MyBusinessMessage {
int32 value = 1;
}
🚀 ▶ grpcurl -v -plaintext -format text -d 'value: 1' 127.0.0.1:9980 io.eigr.spawn.example.Joe.Sum
Resolved method descriptor:
rpc Sum ( .io.eigr.spawn.example.MyBusinessMessage ) returns ( .io.eigr.spawn.example.MyBusinessMessage ) {
option (.google.api.http) = { post: "/v1/sum", body: "*" };
}
Request metadata to send:
(empty)
Response headers received:
content-type: application/grpc+proto
date: Tue, 11 Jun 2024 14:45:08 GMT
server: Cowboy
Response trailers received:
(empty)
Sent 1 request and received 0 responses
ERROR:
Code: Unknown
Message: Internal Server Error
@mjheilmann works as expect
grpcurl -v -plaintext -d '{"value": 1}' 127.0.0.1:9980 io.eigr.spawn.example.Joe/Sum
Resolved method descriptor:
rpc Sum ( .io.eigr.spawn.example.MyBusinessMessage ) returns ( .io.eigr.spawn.example.MyBusinessMessage ) {
option (.google.api.http) = { post: "/v1/sum", body: "*" };
}
Request metadata to send:
(empty)
Response headers received:
content-type: application/grpc+proto
date: Tue, 11 Jun 2024 16:32:57 GMT
server: Cowboy
Response contents:
{
"value": 75
}
Response trailers received:
(empty)
Sent 1 request and received 1 response
You can publish a new hex version?
@sleipnir I had published a new version this morning for another bugfix, which included the merged fixes here. I think you might be good to go on version 0.1.4
@sleipnir I had published a new version this morning for another bugfix, which included the merged fixes here. I think you might be good to go on version
0.1.4
Cool @mjheilmann This here is the result of our collaboration https://twitter.com/sleipni_r/status/1800584524591743205
Very nice!On Jun 12, 2024, at 12:14 AM, Adriano Santos @.***> wrote:
@sleipnir I had published a new version this morning for another bugfix, which included the merged fixes here. I think you might be good to go on version 0.1.4
Cool @mjheilmann This here is the result of our collaboration https://twitter.com/sleipni_r/status/1800584524591743205
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>
Hi guys. I have the following .proto file:
When I tried to make grpcurl request below:
I receive "Failed to resolve symbol "pinger.PingPongActor": rpc error: code = Unknown desc = Internal Server Error" message.
Follow the application logs below: