search

tink-crypto / tink

Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse.
https://developers.google.com/tink
Apache License 2.0
13.5k stars 1.18k forks source link

Unable to initialize Tink #617

Closed ferozed closed 2 years ago

ferozed commented 2 years ago

Help us help you

We’d like to know more about your Tink deployment.

Describe the bug:

I am trying to write a kafka message encryption/decryption mechanism using Tink. I followed the docs, and tried to initialize it.

Provide a clear and concise description of the bug.

Calling register() method throws an exception:

tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class com.google.crypto.tink.proto.HmacKeyFormat
java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class com.google.crypto.tink.proto.HmacKeyFormat
    at com.google.crypto.tink.proto.HmacKeyFormat.getSerializedSize(HmacKeyFormat.java:182)
    at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62)
    at com.google.crypto.tink.Registry.registerKeyTemplates(Registry.java:653)
    at com.google.crypto.tink.Registry.registerKeyManager(Registry.java:556)
    at com.google.crypto.tink.mac.HmacKeyManager.register(HmacKeyManager.java:232)
    at com.google.crypto.tink.mac.MacConfig.register(MacConfig.java:86)
    at com.google.crypto.tink.mac.MacConfig.init(MacConfig.java:74)
    at com.google.crypto.tink.mac.MacConfig.<clinit>(MacConfig.java:59)
    at com.google.crypto.tink.aead.AeadConfig.register(AeadConfig.java:102)
    at com.google.crypto.tink.aead.AeadConfig.init(AeadConfig.java:86)
    at com.google.crypto.tink.aead.AeadConfig.<clinit>(AeadConfig.java:68)
    at com.zillow.kafka.message.MessageEncryptorTest.testSimple(MessageEncryptorTest.java:25)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
    at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
    at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)

What was the expected behavior?

Did not expect an exception.

How can we reproduce the bug?

I use IntelliJ IDEA to write up a unit test.

package com.zillow.kafka.message;

import com.google.crypto.tink.aead.AeadConfig;
import io.confluent.kafka.schemaregistry.avro.AvroSchema;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.GenericRecordBuilder;
import org.junit.jupiter.api.Test;

import java.security.GeneralSecurityException;
import java.util.HashMap;
import java.util.Map;

public class MessageEncryptorTest
{
    String schema = "{\"type\":\"record\",\"namespace\":\"com.example\",\"name\":\"FullName\",\"fields\":[{\"name\":\"first\",\"type\":\"string\"},{\"name\":\"last\",\"type\":\"string\"}]}";

    @Test
    public void testSimple() {
        Map<String,String> config = new HashMap<String,String>();

        try {
            AeadConfig.register();
        } catch (GeneralSecurityException e) {
            e.printStackTrace();
        }

        /*
        GenericRecord record = new GenericData.Record(new Schema.Parser().parse(schema));

        byte [] data = encryptor.serialize("topic", record);

         */
    }
}

Here is build.gradle

buildscript {
    repositories {
        jcenter()
        mavenCentral()
        maven {
            url = "https://packages.confluent.io/maven"
        }
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.1'
    }
}

plugins {
    id 'java' // so that we can use 'implementation', 'testImplementation' for dependencies
}

apply plugin: 'com.github.johnrengelman.shadow'

repositories {
    jcenter()
    mavenCentral()
    maven {
        url = "https://packages.confluent.io/maven"
    }
    maven {
        url = "https://jitpack.io"
    }
}

dependencies {

    implementation 'org.apache.avro:avro:1.9.0'
    implementation 'org.apache.avro:avro-tools:1.9.0'
    // Kafka
    implementation "org.apache.kafka:kafka_2.12:2.2.0"
    implementation 'org.apache.avro:avro:1.9.0'
    implementation 'io.confluent:kafka-schema-registry-client:6.2.1'
    implementation 'io.confluent:kafka-avro-serializer:6.2.1'
    implementation 'io.confluent:kafka-json-serializer:6.2.1'
    implementation 'io.confluent:kafka-json-schema-provider:6.2.1'
    implementation 'com.google.crypto.tink:tink:1.6.1'

    //test
    testImplementation(platform('org.junit:junit-bom:5.9.0'))
    testImplementation('org.junit.jupiter:junit-jupiter')
}

sourceSets {
    main {
        java {
            srcDirs = ["src/main/java"]
        }

        resources {
            srcDirs = ["src/main/avro", "src/main/resources"]
        }
    }
    test {
        java {
            srcDirs = ["src/test/java"]
        }
    }
}

test {
    useJUnitPlatform()
    testLogging {
        events "passed", "skipped", "failed"
    }
}

plugins.withId("com.github.johnrengelman.shadow"){

    //this block requires the java plugin to be applied first.
    plugins.withId("java"){

        shadowJar {
            //We are overriding the default jar to be the shadow jar
            classifier = null
            exclude 'META-INF'
            exclude 'META-INF/*.INF'
            exclude 'META-INF/license/*'
        }

        jar {
            manifest {
                attributes(
                        'Built-By'       : System.properties['user.name'],
                        'Build-Timestamp': new java.text.SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ").format(new Date()),
                        'Created-By'     : "Gradle ${gradle.gradleVersion}",
                        'Build-Jdk'      : "${System.properties['java.version']} (${System.properties['java.vendor']} ${System.properties['java.vm.version']})",
                        'Build-OS'       : "${System.properties['os.name']} ${System.properties['os.arch']} ${System.properties['os.version']}" ,
                        'Main-Class'     : "com.zillow.com.zillow.avro.test.HelloWorld"
                )
            }
        }

        tasks.build.dependsOn tasks.shadowJar
        tasks.shadowJar.mustRunAfter tasks.jar
    }

}

Do you have any debugging information?

tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class com.google.crypto.tink.proto.HmacKeyFormat
java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class com.google.crypto.tink.proto.HmacKeyFormat
    at com.google.crypto.tink.proto.HmacKeyFormat.getSerializedSize(HmacKeyFormat.java:182)
    at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62)
    at com.google.crypto.tink.Registry.registerKeyTemplates(Registry.java:653)
    at com.google.crypto.tink.Registry.registerKeyManager(Registry.java:556)
    at com.google.crypto.tink.mac.HmacKeyManager.register(HmacKeyManager.java:232)
    at com.google.crypto.tink.mac.MacConfig.register(MacConfig.java:86)
    at com.google.crypto.tink.mac.MacConfig.init(MacConfig.java:74)
    at com.google.crypto.tink.mac.MacConfig.<clinit>(MacConfig.java:59)
    at com.google.crypto.tink.aead.AeadConfig.register(AeadConfig.java:102)
    at com.google.crypto.tink.aead.AeadConfig.init(AeadConfig.java:86)
    at com.google.crypto.tink.aead.AeadConfig.<clinit>(AeadConfig.java:68)
    at com.zillow.kafka.message.MessageEncryptorTest.testSimple(MessageEncryptorTest.java:25)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
    at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
    at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at com.sun.proxy.$Proxy2.stop(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:135)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
    at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
    at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
    at java.lang.Thread.run(Thread.java:748)

Provide your version information:

Is there anything else you’d like to add?

None.

ferozed commented 2 years ago

FYI... this does not happen in 1.5.0 version. So looks like something broke in 1.6.1

morambro commented 2 years ago

Hi @ferozed, could you check if any other dependency is pulling in protobuf (com.google.protobuf:protobuf-java)? Tink 1.6.1 requires com.google.protobuf:protobuf-java:3.14.0

ferozed commented 2 years ago

No, there is no other dependency pulling in protobuf. I verified it using gradle dependencies

Added output of gradle dependencies

dependencies.txt

tholenst commented 2 years ago

This is very weird then -- if you look at the error there must be some protobuf version conflict. Remember that Protobuf creates some code from the ".proto" files in Tink, and then uses it. If these files are created with a different version then the protobuf library used, then such mistakes happen.

Maybe there is some leftover in your installation from Tink 1.5.0?

ferozed commented 2 years ago

No.. originally i was using 1.6.1 and it occurs. Move to 1.5.0 and it doesnt. Move back to 1.6.1 and it does.

It happens in IntelliJ as well as when running using gradle.

If there was a mixed version conflict, it should show up in gradle dependencies right?

Is there any compile time action running? Meaning is it trying to use .proto files and generate java classes? Maybe it is ending up using a different version of protobuf?

I have protobuf 3.19.4 installed systemwide

$ protoc --version
libprotoc 3.19.4
ferozed commented 2 years ago

nvm... my intellij project import was corrupted. reimporting the project fixed it.

daveclay commented 1 year ago

Just a note for posterity: including avro-tools in a project that uses tink 1.7.0 causes problems because avro-tools is a "fat jar" that includes its own dependencies - including com/google/protobuf - in its jar. At runtime, tink may load up the protobuf classes from the avro-tools jar rather than its own protobuf dependency, leading to compatibility issues.

Our solution was to remove avro-tools from dependencies, as it's not appropriate unless you're really using the jar file as an executable on the command line.