InvalidProtocolBufferException when executing signed transaction (golang) on Java server

robkainz commented 2 years ago

Description

There seems to some kind of incompatibility between the Go and Java Hedera SDKs and their use of Google's protobuf libraries and this is currently blocking us on our WW HTS release due to a last-minute, but important, feature. I generate a transaction on our server (Java), have the client sign it (golang) and send it back to the server signed for execution (so the server pays all fees.) Unfortunately, when run on the server I get this error:

2021-10-25 08:14:22,957 ERROR | http-nio-8080-exec-2 | o.a.c.c.C.[Tomcat].[localhost].[/].[dispatcherServlet] | Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero). at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:125) at com.google.protobuf.ArrayDecoders.decodeUnknownField(ArrayDecoders.java:1036) at com.google.protobuf.ArrayDecoders.decodeUnknownField(ArrayDecoders.java:1028) at com.google.protobuf.MessageSchema.parseProto3Message(MessageSchema.java:5426) at com.google.protobuf.MessageSchema.mergeFrom(MessageSchema.java:5442) at com.google.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1567) at com.google.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1671) at com.hedera.hashgraph.sdk.proto.TransactionList.parseFrom(TransactionList.java:148) at com.hedera.hashgraph.sdk.Transaction.fromBytes(Transaction.java:149) at ...

Steps to reproduce

Create a token association transaction in Java - get the bytes (I convert them to a hex string to pass back to the golang client)
Have the golang client sign the transaction and return the updated signed bytes to the server
Execute the transaction on the server

Additional context

No response

Hedera network

testnet

Version

v2.3.0

Operating system

macOS

andrewantar commented 2 years ago

This is a blocker for walla wallet!

SimiHunjan commented 2 years ago

Apologies for the delay here. @danielakhterov @andrix10 please review this bug asap.

janaakhterov commented 2 years ago

@robkainz @andrewantar Which version of the SDKs are you guys using? I just did a mini manual test and everything seems to work.

Create TokenCreateTransaction in Java

System.out.println(Hex.toHexString(new TokenAssociateTransaction()
            .setTransactionId(TransactionId.generate(testEnv.client.getOperatorAccountId()))
            .setNodeAccountIds(Collections.singletonList(new AccountId(3)))
            .setAccountId(accountId)
            .setTokenIds(Collections.singletonList(tokenId))
            .freeze()
            .toBytes()
    ));

Decode string and sign it then re-encode it to string in Go

 bytes, err := hex.DecodeString("0a372a350a310a130a0b08f2f8eb8b0610a79aec2a12041888c401120218031880cab5ee0122020878c2020b0a0318a80e12041889c4011200")
assert.NoError(t, err)

transaction, err := TransactionFromBytes(bytes)
assert.NoError(t, err)

switch tx := transaction.(type) {
case TokenAssociateTransaction:
    _, err := tx.SignWithOperator(env.Client)
    assert.NoError(t, err)

    bytes, err =tx.ToBytes()
    assert.NoError(t, err)

    println(hex.EncodeToString(bytes))
}

Decode string in Java

    var hex = "0a9e012a9b010a310a130a0b08f2f8eb8b0610a79aec2a12041888c401120218031880cab5ee0122020878c2020b0a0318a80e12041889c40112660a640a20f0d2879e3cc156dab7008dcb71f85402db9aebc310785aa1c48f60b2d6b9f0961a40870eeee74b4b3b58389ce15b87537d1d43bf2562de6f7bfc7c3435a1668f81d3692e9f2b9a52119cd1ecb79cee545b76ce5428f88ee2614fd0c9ec50eddda403";

    var transaction = Transaction.fromBytes(Hex.decode(hex));

    if (transaction instanceof TokenAssociateTransaction) {
        for (var nodeEntries : transaction.getSignatures().entrySet()) {
            for (var signatures : nodeEntries.getValue().entrySet()) {
                System.out.println("Node: " + nodeEntries.getKey() + " PublicKey: " + signatures.getKey() + " Signature: " + Hex.toHexString(signatures.getValue()));
            }
        }
        System.out.println(transaction);
    } else {
        System.out.println("TRANSACTION IS NOT TOKENASSOCIATETRANSACTION");
    }

robkainz commented 2 years ago

It's failing for me both on Oracle JDK 11.0.13 running on Mac OSX 11.3.1 (Big Sur) and also the standard appengine environment on Google Cloud using Java 11.

Java Hedera SDK:

implementation 'com.hedera.hashgraph:sdk:2.1.1'

Go Hedera SDK:

github.com/hashgraph/hedera-sdk-go/v2 v2.3.0

Here are the relevant code snippets:

create the token associate transaction in Java:

TokenAssociateTransaction tokenAssociateTransaction = new TokenAssociateTransaction()
        .setTokenIds(tokenIdList)
        .setAccountId(AccountId.fromString(accountId))
        .setTransactionValidDuration(Duration.ofMinutes(2))
        .freezeWith(client);

return Hex.toHexString(tokenAssociateTransaction.toBytes());

the method used to sign the token associate transaction in go

func SignTransaction(transactionHexBytes string, privateKey string) (string, error) {
var err error
var privKey hedera.PrivateKey
var transactionBytes []byte
var signedTransaction []byte

privKey, err = hedera.PrivateKeyFromString(privateKey)
if err!=nil {
    return "", err
}

transactionBytes, err = hex.DecodeString(transactionHexBytes)
if err!=nil {
    return "", err
}

transaction, err := hedera.TransactionFromBytes(transactionBytes)
if err!=nil {
    return "", err
}

switch newTx := transaction.(type) {
case hedera.TokenAssociateTransaction:
    signedTransaction, err = privKey.SignTransaction(&newTx.Transaction)
}
if err!=nil {
    return "", err
}

return hex.EncodeToString(signedTransaction), nil
}

the java code to read the token associate transaction - it fails in fromBytes(), before I can even execute it.

System.out.println("Client Signed Transaction Hex Bytes: " + transactionHexBytes);
byte[] bytes = Hex.decode(transactionHexBytes);
TokenAssociateTransaction transaction = (TokenAssociateTransaction) TokenAssociateTransaction.fromBytes(bytes);

And here's the full stack trace (Mac OSX environment) with the original transaction bytes (hex) as created by Java and the signed transaction bytes (hex) created by golang:

new token associate transaction hex bytes: 0a3a2a380a340a140a0c08c4bbef8b06108ac49ec303120418b8af03120218031880c2d72f22020878c2020e0a051892c8aa01120518c6b781011200 Client Signed Transaction Hex Bytes: 6fc5c71076dd86d109b93d5b07d9ea969a95173e9f634ee14c9b49513b013be7db6a485c43b7bb5726336d5aa859c1b5b9298b812e1af49e781e1db414823004 2021-10-29 07:41:05,510 ERROR | http-nio-8080-exec-2 | o.a.c.c.C.[Tomcat].[localhost].[/].[dispatcherServlet] | Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero). at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:125) at com.google.protobuf.ArrayDecoders.decodeUnknownField(ArrayDecoders.java:1036) at com.google.protobuf.MessageSchema.parseProto3Message(MessageSchema.java:5426) at com.google.protobuf.MessageSchema.mergeFrom(MessageSchema.java:5442) at com.google.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1567) at com.google.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1671) at com.hedera.hashgraph.sdk.proto.TransactionList.parseFrom(TransactionList.java:148) at com.hedera.hashgraph.sdk.Transaction.fromBytes(Transaction.java:149) at com.ledgerama.wallawalletservice.airdrop.AirdropServiceImpl.executeAssociateTokenTransaction(AirdropServiceImpl.java:654) at com.ledgerama.wallawalletservice.airdrop.AirdropServiceImpl$$FastClassBySpringCGLIB$$94f7dfab.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at com.ledgerama.wallawalletservice.airdrop.AirdropServiceImpl$$EnhancerBySpringCGLIB$$d36ea72e.executeAssociateTokenTransaction() at com.ledgerama.wallawalletservice.airdrop.AirdropController.executeAssociateTokensTransactionAndAirdrop(AirdropController.java:136) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:97) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834)

janaakhterov commented 2 years ago

I think I see the issue.

    switch newTx := transaction.(type) {
    case hedera.TokenAssociateTransaction:
        signedTransaction, err = privKey.SignTransaction(&newTx.Transaction)
    }
    if err!=nil {
        return "", err
    }

    return hex.EncodeToString(signedTransaction), nil

Specifically signedTransaction, err = privKey.SignTransaction(...). This does not return a signed transaction, but instead returns the signature. The transaction that is signed will still be newTx so I recommend changing hex.EncodeToString(signedTransaction), nil to hex.EncodeToString(newTx.ToBytes())

robkainz commented 2 years ago

Thank you so much Daniel, you were correct. Your code suggestion isn't fully correct, but it was easy to get it working from there. I'm sorry I missed that, it should have been obvious to me. I now have our airdrop feature fully working!

janaakhterov commented 2 years ago

I'm glad to hear that everything worked out, and no worries about missing that it happens to the best of us.

hashgraph / hedera-sdk-go