JS generator missing jspb.generate_from_object option

[hochhaus]

The comment for jspb.Message.GENERATE_FROM_OBJECT states:

 *     NOTE: By default no protos actually have a fromObject method. You need to
 *     add the jspb.generate_from_object options to the proto definition to
 *     activate the feature.

However, no such option is currently implemented in the open sourced version. I see the code which generates output for this option (GenerateClassFromObject) but it is invoked 0 times.

~/protobuf$ grep -r GenerateClassFromObject .
./src/google/protobuf/compiler/js/js_generator.cc:void Generator::GenerateClassFromObject(const GeneratorOptions& options,
./src/google/protobuf/compiler/js/js_generator.h:  void GenerateClassFromObject(const GeneratorOptions& options,

Can the jspb.generate_from_object option be open sourced?

[hochhaus]

Also, the jspb.generate_xid option appears to be missing as well.

/**
 * The xid of this proto type (The same for all instances of a proto). Provides
 * a way to identify a proto by stable obfuscated name.
 * @see {xid}.
 * Available if {@link jspb.generate_xid} is added as a Message option to
 * a protocol buffer.
 * @const {!xid.String|undefined} The xid or undefined if message is
 *     annotated to generate the xid.
 */
jspb.Message.prototype.messageXid;

[hochhaus]

@xfxyjwf Any chance this bug could be addressed prior to the beta 4 release? It is a large issue for certain use cases of the JS implementation.

[pcj]

+1

[hochhaus]

Adding @haberman as he was able to help with a few other JS issues.

Any chance this could be looked into?

[haberman]

Many of the options we have internally are legacy/deprecated, so we haven't open-sourced them. The ones that we want to keep we will likely want to integrate into descriptor.proto.

Could you say more about how you want to use toObject() and fromObject()? I ask because this format is not standardized, and it is similar to (but not the same as) proto3 JSON.

Also how do you want to use xids?

[hochhaus]

Thanks Joshua. All I need is proto3 JSON support (I was not aware of the different toObject() JSON format). Also I have no current need for xid, I was only trying to be complete.

Could the proto3 JSON support be open sourced?

[haberman]

We don't currently have proto3 JSON support implemented for JavaScript (even internally). Inside Google most JavaScript projects use a wire format called JSPB that is specific to JavaScript and encodes protobufs as arrays indexed by field number like [123, "abc", true]. This format has code-size benefits over binary or proto3 JSON formats (basically it can be parsed with JSON.parse() without the need for any generated parsing code). But the JSPB format isn't very standardized (even inside Google) and changes a lot so we haven't open-sourced it.

For JavaScript we've implemented binary support which works across proto2/proto3 and can interoperate with other implementations. proto3 JSON support would be possible to implement, but we haven't done it yet.

[hochhaus]

@haberman Thanks for the additional context.

For what it is worth, the JSPB format that you mentioned looks very similar to the goog.proto2 PBLite format from closure-library. I've been using this for some time as part of the protoclosure project.

Given that proto3 JSON isn't implemented would it be possible to include the toObject and fromObject JSPB support?

[haberman]

For what it is worth, the JSPB format that you mentioned looks very similar to the goog.proto2 PBLite format from closure-library.

Indeed -- that is an example of a format that is similar but not the same as JSPB. I didn't realize that had been open-sourced.

[pcj]

I was under the impression from https://groups.google.com/d/msg/closure-library-discuss/oRZIfaoFGRQ/0FZpl-RIAgAJ that someone is working on grpc support for javascript that may or may not be supported by closure library goog.net.rpc.*. Any insight on JSON support with those efforts?

[haberman]

@stanley-cheung do you have any insight on @pcj's question?

[stanley-cheung]

I am not sure about the full context here, but yes we are working on a javascript client library that will use JS protobuf and the Closure library to support making RPC calls from browsers to gRPC backend via a gRPC gateway. It is still early in that project so nothing concrete is released yet. The current estimate is late Q3 the earliest, probably Q4.

[pcj]

@stanley-cheung Nice. Keep me posted, I may be interested in helping with that.

[seishun]

@haberman

Could you say more about how you want to use toObject() and fromObject()?

Here's my use case. I maintain a library that interoperates with a network that uses protobufs (proto2) extensively. Since I don't want to tie my library's API to a specific protobuf implementation, its methods accept protobufs as plain objects - same goes for events that provide protobufs (more detailed description here).

There's also the issue that I don't convert to camelCase, but toObject in this implementation does and there's no way around it. I was hoping to address this after protocolbuffers/protobuf#1922 is resolved.

[alexeykomov]

Hello, all, I'm interested in this issue being resolved as well.

Answering @haberman on question in https://github.com/google/protobuf/issues/1591#issuecomment-237001956. My use case is to use toObject and fromObject in order to use JSON.stringify and JSON.parse with them correspondingly, in order to transmit human readable JSON over the wire.

I see that at least fromObject being missing could be resolved by adding GenerateClassFromObject(options, printer, desc); after this line https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/js/js_generator.cc#L1841.

Is there any dangers with this solution?

[wiktortomczak]

What's the status of this? Are there plans to include fromObject method in the generated JS proto message classes?

My use case is same as @alexeykomov's. Also, Alexey's solution just works. See wiktortomczak/protobuf@68ad1c80cefc93e17f26b3a337b253102bbece27.

[someValue]

+1

[jonnyreeves]

I would also like to make use of fromObject; our use-case is to incorporate it with https://github.com/improbable-eng/ts-protoc-gen and provide a less verbose API for constructing Protobuf objects for transmission via https://github.com/improbable-eng/grpc-web

I'm happy to raise a PR for this if it will be considered by the grpc maintainers.

[shanshanzhu]

What is the status of this one? We would like to have fromObject exposed as well.

[anandolee]

@haberman any updates?

[hartmut-co-uk]

having the same requirement as @jonnyreeves...

less verbose API for constructing Protobuf objects

[metled]

+1, any updates?

[met-pub]

hope fromObject() generated.

[amgadani]

Pinging this ticket again, any update ?

[parano]

Tried the following workaround and it seems to work correctly. Although I haven't tested it extensively: https://github.com/protocolbuffers/protobuf/compare/master...parano:js-fromobject-hack?expand=1

First add the following line to js_generator.cc:

diff --git a/src/google/protobuf/compiler/js/js_generator.cc b/src/google/protobuf/compiler/js/js_generator.cc
index 5779d5e2..439893b4 100644
--- a/src/google/protobuf/compiler/js/js_generator.cc
+++ b/src/google/protobuf/compiler/js/js_generator.cc
@@ -1910,6 +1910,7 @@ void Generator::GenerateClass(const GeneratorOptions& options,

     GenerateClassToObject(options, printer, desc);
+    GenerateClassFromObject(options, printer, desc);
     // These must come *before* the extension-field info generation in
     // GenerateClassRegistration so that references to the binary
     // serialization/deserialization functions may be placed in the extension

The Generator::GenerateClass method now looks like:

void Generator::GenerateClass(const GeneratorOptions& options,
                              io::Printer* printer,
                              const Descriptor* desc) const {
  ...
  if (!NamespaceOnly(desc)) {
    printer->Print("\n");
    GenerateClassConstructor(options, printer, desc);
    GenerateClassFieldInfo(options, printer, desc);

    GenerateClassToObject(options, printer, desc);

    GenerateClassFromObject(options, printer, desc); // ADD THIS LINE

    // These must come *before* the extension-field info generation in
    // GenerateClassRegistration so that references to the binary
    // serialization/deserialization functions may be placed in the extension
    // objects.
    GenerateClassDeserializeBinary(options, printer, desc);
    GenerateClassSerializeBinary(options, printer, desc);
  }
  ...
}

Rebuild protoc following instructions here, now the protoc generated js code should include fromObject method. Although it is missing goog.isDef and goog.array method. A quick hack, I added the following to all the generated file:

goog.isDef = function(a) {
  if (a === undefined || a === null) {
    return false;
  }
  return true;
}

goog.array = {
  map: function(arr, func) {
    return arr.map(func);
  }
}

It works for most simple cases. The only problem I still have is the generated fileld name are not what I'd expected. For example, for field repeated myObjects = 1, the json object field name will be myobjectsList.

Looks like I need the Use proto field name instead of lowerCamelCase name option that is documented here: https://developers.google.com/protocol-buffers/docs/proto3#json_options Which I believe is not yet implemented in js.

[sophiawisdom]

@TeBoring any thoughts/plans on this?

[thetimetraveler]

any updates on this?

[elvizlai]

LOL? a 2016 issue still not solved.

[BichengWang]

we need fromObject.......

[4nte]

.fromObject would be neat, +1

[ohroy]

+1

[juansalvatella]

+1

[dtiziani]

We really need this! Any updates at all?

[sophiawisdom]

I just used protobufjs for javascript instead of the default implementation, and it was better in every respect.

[create-share]

Any luck on this ?

[jontroncoso]

@TeBoring Is this a priority? Or can you mark this as won't fix or pass this off to somebody else? This is an urgent bug from 4 years ago that makes this library kind of useless (or at least extremely convoluted).

[jontroncoso]

For future people looking, here's a lame hack that I came up with. We're only using int32 type ints and strings, so this works for us. There's a good chance this won't work for you if your using other scalars:

import { Message } from 'google-protobuf';
import {
  StringValue,
  Int32Value,
} from 'google-protobuf/google/protobuf/wrappers_pb';

export function fromObject(_return: Message, obj: {}): Message {
  Object.keys(obj).forEach((key: string): void => {
    const setMethodName = `set${key.charAt(0).toUpperCase()}${key.slice(1)}`;
    const hasMethodName = `has${key.charAt(0).toUpperCase()}${key.slice(1)}`;
    let value = obj[key];
    if (typeof _return[hasMethodName] === 'function') {
      if (typeof value === 'string') {
        value = new StringValue(value);
      } else if (typeof value === 'number') {
        value = new Int32Value(value);
      } else if (typeof value !== 'object') {
        value = null;
      }
    }
    if (
      typeof _return[setMethodName] === 'function' &&
      (!!value || typeof value === 'number')
    ) {
      _return[setMethodName](value);
    }
  });
  return _return;
}

[mihai1voicescu]

Without .fromObject gRPC seems to be unusable in Node.js. Any plans in fixing this?

[jmzwcn]

black tech: bytes=>string=>bystes localStorage.setItem('user', Message.bytesAsB64(user.serializeBinary())); User.deserializeBinary(Message.bytesAsU8(localStorage.getItem('user

Message is from jspb.Message.

[edvardchen]

I write a tool to generate complex message object from JSX.

const feature: Feature = (
  <Feature name="hello" location={<Point {...point} />}></Feature>
);

https://github.com/edvardchen/protobuf-jsx

Try it as a friendly workaround before official support.

[connor4312]

Lacking this option is is making it hard to support communication with etcd in the browser (see referenced issue) 🙁

[Teivaz]

Any luck get this issue resolved? It's been 4 years since it was raised.

[otto-dev]

My use case is same as @alexeykomov's. Also, Alexey's solution just works. See wiktortomczak/protobuf@68ad1c8.

I don't know what I'm looking at - So we could have this solved with one line of code?

[otto-dev]

I've found a way to convert any plain JavaScript object reliably to its Protobuf counterpart for use with any grpc-web implementation. Includes nested messages / all types.

The answer is that protocolbuffers/protobuf does not support fromObject, but protobuf.js does with .encode.

So you can serialize your plain object into the protobuf binary format using protobuf.js, and then read that serialized binary back into another protobuf library of your choice. (improbable-eng/grpc-web in my case)

  const ProtoJsMyMessage = root.lookupType('something.MyMessage'); // protobuf.js
  const whatTheHeck = { ... }; // plain JS object containing the message properties
  const result = MyMessage.deserializeBinary(ProtoJsMyMessage.encode(whatTheHeck).finish()); // MyMessage is grpc-web

So I'm using the protobuf binary format as an interface between the two protobuf libraries, so that I can use gRPC as an interface to communicate with my backend. I'm glad that I have finally arrived at a solution that is more "agile" (/s) than just using JSON. Shoot me now.

[lucaslcode]

That should be looked at. Saying "toObject/fromObject has some issues and we are discussing it internally" really goes against the point of open source. What are the issues? toObject works well. We are ready to help with pull requests but they get ignored. Is the internal discussion 4 years long?

[Leo7654]

+1

[vjpai]

@murgatroid99 @nicolasnoble Can y'all comment on protocolbuffers/protobuf-javascript#96

[murgatroid99]

As far as I am aware, people have been successfully using code generated by protoc with Node gRPC just fine for a while. This requested feature would absolutely improve usability, but I assume the statement that it is "unusable" is just hyperbole.

[mkosieradzki]

As far as I am aware, people have been successfully using code generated by protoc with Node gRPC just fine for a while.

You mean people used to imperative paradigm (I bet they have backend implemented in Java :-) ). Today people practising functional paradigm have no practical means to use this library.

From my perspective lack of support for functional style is good enough reason to keep using protobuf.js.

[xiemeilong]

Add this feature， please ， please， please！