Add DynamicMessage support for C#

[jtattermusch]

Not a requirement as of now, but it would be nice to have one day in not too distant future.

[jskeet]

It would be nice to have specific requirements around this - scenarios we're trying to make simple. A few questions that arise:

• Do we need reflection support? (I assume so.)
• Will these messages be mutable, just like the rest? (I assume so.)
• Do we need to be able to mix and match, e.g. set a field in a dynamic message to a value which is a strongly-typed message? (Could be tricky.)
• Do we need C# dynamic typing support?

[xfxyjwf]

@jskeet @anandolee Is this DynamicMessage implemented in C#?

[jskeet]

Not that I'm aware of.

[dluc]

The simpler, still very useful, support would be removing the need to compile .proto files to DLLs at build time, so that a .NET application could generate the serializer/deserializer at runtime (caching it in memory) from .proto files provided by users for example.

see for example https://stackoverflow.com/a/27505474/2793555

[jskeet]

@dluc: That's far from simple - the C# code has no facility for parsing .proto files at all. That's all in protoc. If any dynamic message support is introduced, it would almost certainly work with the descriptors, not the original .proto files.

(And generating the code at execution time is far from trivial too - again, all the code generation is in protoc, not in the Google.Protobuf library.)

[jskeet]

I'm actively working on this now. The initial support is likely to be relatively bare, but it'll be there. The main reason for me doing this is for the sake of writing protoc plugins in C#.

[mcon-gr]

@jskeet Is this something you're still working on? I'm happy to contribute if you consider this a parallelisable task.

[jskeet]

I'm not working on it at the moment, no. Basically I don't have time to work on protobuf other than for features I need for other work, e.g. being able to load the descriptors dynamically. I'm afraid I wouldn't have time to review significant chunks of code either, but hopefully @anandolee will be able to help.

[mcon-gr]

Thanks Jon - I'm getting the feeling that adding DynamicMessage to C# will involve some non-trivial work, once I get some time I'll start by taking my lead from the Java implementation.

[jrimig]

@jskeet Thanks for adding FileDescriptor.BuildFromByteStrings(). Is this method intended to read the FileDescriptorSet generated by

protoc.exe --include_imports --descriptor_set_out=FILE

If not, is there a way to generate a FileDescriptor from these output files, something similar to FileDescriptorSet.parseFrom() in the Java API?

[jskeet]

It's not quite that simple, unfortunately. From memory, you can create your own message that has a repeated bytes field:

message FileDescriptorSet {
  repeated bytes file = 1;
}

... and parse the descriptor set with that, then pass the resulting RepeatedField<ByteString> into FileDescriptor.BuildFromByteStrings(). But it does take that manual step at the moment. While we could include it directly into the C# library, the current proto2 support work would make that redundant and a little odd, which is why I haven't created a pull request for it. Another option would be to have a method of FileDescriptor.BuildFromFileDescriptorSet accepting a single ByteString, of course. (That could use the internal FileDescriptorSet message.)

[jrimig]

Thanks, Jon! That works just like you said:

var fileDescriptorSet = FileDescriptorSet.Parser.ParseFrom(File.ReadAllBytes(path));
var fileDescriptors   = FileDescriptor.BuildFromByteStrings(fileDescriptorSet.File);

[michaelpearce-gain]

Is there any progress on this? currently forced to have to use the old java port https://github.com/jskeet/protobuf-csharp-port/blob/master/src/ProtocolBuffers/DynamicMessage.cs

real blocker for being able to handle dynamic streams of protobuf records on kafka with .net clients (java clients work fine as they have dynamic message). use cases is similar to using GenericRecord in avro, where consumers just need to resolve the filedescriptorset which is being held in a central repo akin to schema registries in avro.

[jtattermusch]

@michaelpearce-gain this is not actively being worked on and we don't have bandwidth to work on this. If this is something you believe lot of users need, consider contributing.

[jtattermusch]

Btw, we just merged proto2 support a few days ago, that might help when working with the descriptors (descriptors are defined as proto2, so until recently they've been implemented in a restricted fashion).

[michaelpearce-gain]

I think that will help, whats the expected sort of time, that would be until its released? +100 having that

[jtattermusch]

I think in the next 12 weeks. We might not have made it on time for the upcoming 3.10 release, but it will be in 3.11 (Google.Protobuf now has a release approx. once every 6 weeks).

[jsolana]

Hi, Is there any progress on this?

[jtattermusch]

No plans to work on DynamicMessage support at the moment.

[shouhujishu]

Hi, Is there any progress on this?

[xoofx]

FYI, I wrote a .NET library grpc-curl that provides an API to access dynamically a gRPC service by using reflection. It's available in the NuGet package DynamicGrpc

You can access a gRPC service like this:

var channel = GrpcChannel.ForAddress("http://192.168.100.1:9200");
// Fetch reflection data from server
var client = await DynamicGrpcClient.FromServerReflection(channel);

// Call the method `Handle` on the service `SpaceX.API.Device.Device`
var result = await client.AsyncUnaryCall("SpaceX.API.Device.Device", "Handle", new Dictionary<string, object>()
{
    { "get_status", new Dictionary<string, object>() }
});

The underlying DynamicMessage implementation is not directly accessible for now, because I haven't had a good use case to expose it, as it requires quite some surrounding setup. Though, that could be useful likely for scenarios that are not using gRPC.

[jskeet]

Just to update this, we have more resources for C# support in protobuf now, but they're still limited. I would definitely like to do this (as a significant piece of missing functionality present in other languages) but it's definitely non-trivial. I'll attempt to carve out some time to prototype this in the next few weeks, so that we've got a better idea of just how difficult it's likely to be.

[KerstinKeller]

@jskeet a quick question: is this required to build / parse messages when only the descpriptor of those messages is available? I have tried to create a MessageParser from a FileDescriptorSet, but I am failing:

        public static Google.Protobuf.MessageParser GetMessageParserFromDescriptorSet(Google.Protobuf.Reflection.FileDescriptorSet descriptor_set_, string typename_)
        {
          var descriptor_list = new List<ByteString>();
          foreach (var file in descriptor_set_.File)
          {
            descriptor_list.Add(file.ToByteString());
          }
          var all_descriptors = Google.Protobuf.Reflection.FileDescriptor.BuildFromByteStrings(descriptor_list);
          var registry = Google.Protobuf.Reflection.TypeRegistry.FromFiles(all_descriptors);
          var message_descriptor = registry.Find(typename_);

          return message_descriptor.Parser;
        }

Unfortunately, the Parser Field is always null, although the rest of the MessageDescriptor seems to contain the appropriate information.

It would be great if you could confirm if deserializing a Message where only the FileDescriptorSet is available, is doable at this very moment, or rather not.

[jskeet]

@KerstinKeller: No, it isn't possible at the moment I'm afraid.

[qqiu-rbx]

@jskeet , do you happen to have an execution plan written down somewhere in case someone may have enough bandwidth to work on this feature?

Will this be a large undertaking or maybe medium sized and will need couple weeks to implement?

[jskeet]

No, I don't have anything like that I'm afraid. I'd expect it to take someone who is very familiar with protobuf and the .NET library at least a couple of weeks of full time work - and for someone less familiar, significantly more, I'm afraid.

[qqiu-rbx]

No, I don't have anything like that I'm afraid. I'd expect it to take someone who is very familiar with protobuf and the .NET library at least a couple of weeks of full time work - and for someone less familiar, significantly more, I'm afraid.

Oh bummer, this is unfortunate. Thanks for getting back to me. If only this problem can be broken down into small pieces.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago. This issue will be closed and archived after 14 additional days without activity.

[learn-more]

This is something that I am interested in.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago. This issue will be closed and archived after 14 additional days without activity.

[learn-more]

I am still interested in this.