[JavaScript] Type Forward/Backward compatible object serialization for Fury JavaScript

[chaokunyang]

Is your feature request related to a problem? Please describe.

Type forward and backward compatible serialization is a common requirement for RPC between microservices. As the calling and receiving ends of the service are deployed and upgraded independently, the serialized objects may exist in different versions across services.

The object structure may have a different number of fields between services. Serialization frameworks need to be able to efficiently support object serialization under situations of type inconsistency between the two ends, ensuring accuracy while providing more compact data and extreme performance.

The Flatbuffer serialization framework suffers from significant data inflation and lacks flexibility, whereas Protobuf achieves type forward and backward compatibility through a KV structure. For each field of the object, during serialization, the object's TAG and type are written.

During deserialization, this information can be used to skip fields not present in the current version. This approach incurs additional performance overhead and space overhead for Message List serialization.

Fury has designed a type compatibility protocol based on metadata compression shared, which at runtime, performs extreme compression of the object's field names and types and other metadata, and shares it across multiple object serializations to minimize metadata overhead. Furthermore, based on this metadata, a deserializer fully consistent with serialization can be dynamically generated during deserialization, completely solving the issues of performance, usability, and space redundancy.

The detailed spec can be fount at https://fury.apache.org/docs/specification/fury_xlang_serialization_spec/#schema-evolution

Describe the solution you'd like

• [ ] #1671
• [ ] #1672
• [ ] #1673
• [ ] #1674

Additional context

[Forchapeatl]

Hi @chaokunyang , please Could you provide more details on the specific requirements , expected results and constraints for the JavaScript implementations. Are these features independent or does one affect the other ?

[Forchapeatl]

Hello @theweipeng , please can shed some light here.

[theweipeng]

Hello @theweipeng , please can shed some light here.

Thanks for your great work on the MetaString. The main goal of Forward/Backward compatibility is to generate code at runtime. Currently, we have implemented the MetaString compression algorithm, but we are unable to pass the JSON string to the other endpoint, as the metadata we communicate across languages will be packed into binary. The binary layout is described in https://fury.apache.org/docs/specification/fury_xlang_serialization_spec#type-meta. Therefore, could you please implement the 'type meta' part of our protocol.