Allow irsa service account

[antontreushchenko]

Can you add to your root and avro kafka connect converter pom.xml these dependencies to allow the irsa service account?

            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-sts</artifactId>
                <version>${aws.sdk.v1.version}</version>
            </dependency>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>sts</artifactId>
                <version>${aws.sdk.v2.version}</version>
            </dependency>

[blacktooth]

Hi,

Can you please provide more context on this?

1. What is the use-case you are trying to implement?
2. What is irsa?

Thanks!

[antontreushchenko]

Hi

1. This use-case fix the #151. Also allows you to run kafka streams images or kafka connectors on the EKS cluster.
2. Irsa is IAM Roles for Service Accounts on the EKS cluster.

Now I've patched your lib, but it's not convenient to use. If you make this fix, it will save me from constant patches and will be useful to many users Thank you!

[blacktooth]

Thanks for explaining it. Do we need dependencies on both SDKv1 and SDKv2? Can we just use SDKv2? Will you be willing to submit a PR?

[antontreushchenko]

Hi! Yes, we need dependencies on both SDKv1 and SDKv2. Yes, I will provide you with a PR within 1 hour!

[blacktooth]

Yes, we need dependencies on both SDKv1 and SDKv2.

What's the reasoning behind using both? Do you use both versions of the SDK? I am concerned of any class path conflicts and increase in JAR size.

[antontreushchenko]

Yes, we use both versions of the SDK. One for kafka converters irsa and one more for kafka streams irsa

[hhkkxxx133]

Hello, the change is included in our latest release 1.1.10. Thanks for your contribution!

[ali-raza-rizvi]

@antontreushchenko We have added both v1 and v2 sdks and using the 1.1.10 version of glue sdk but its still using the node role instead of service account role , could you please share what patches you had applied to resolve this issue ?

[ahsan-n]

Hi @antontreushchenko thanks for the fix. We are trying to achieve the same but still getting error:

software.amazon.awssdk.services.glue.model.AccessDeniedException: User: arn:aws:sts::****************:assumed-role/NODE_ROLE/i-833fu7203a782371 is not authorized to perform: glue:GetSchemaByDefinition on resource: arn:aws:glue:us-east-1:****************:registry/schema-registry because no identity-based policy allows the glue:GetSchemaByDefinition action (Service: Glue, Status Code: 400, Request ID: 74269899-8eaf-48dc-831b-7j271209231j71)

However same configuration uses secret manager too which is working fine i-e utilizing IRSA.

this is our config

package com.kafka.gluedemo.config.msk;

import com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistryKafkaSerializer;
import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
import com.kafka.gluedemo.config.glue.GlueProperties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import software.amazon.awssdk.services.glue.model.DataFormat;

import java.util.Properties;

@Configuration
public class MskConfig {

    private final MskProperties mskProperties;

    public MskConfig(
                     final MskProperties mskProperties) {

        this.mskProperties = mskProperties;
    }

    //TODO:: @Ali @Shakeel Need to add IAM Jar config inorder to with work MSK IAM
    @Bean
    public Properties mskConfigProperties() {

        //producer config for MSK

        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, mskProperties.getProducer().getBootstrapServers());
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, GlueSchemaRegistryKafkaSerializer.class.getName());
        props.put(AWSSchemaRegistryConstants.SCHEMA_NAME, "my-schema");
        props.put(AWSSchemaRegistryConstants.DATA_FORMAT, DataFormat.AVRO.name());
        props.put(AWSSchemaRegistryConstants.AWS_REGION, "ap-southeast-1");
        props.put(AWSSchemaRegistryConstants.REGISTRY_NAME, "prod-schema-registry");
        return props;
    }

    @Bean("producer")
    public KafkaProducer<String, Object> kafkaProducer(final Properties mskConfigProperties) {
        return new KafkaProducer<String, Object>(mskConfigProperties);
    }
}

any chance can you further guide us or share your configs?