search

aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
142 stars 65 forks source link

Incorrect downstream dependency name leads to build errors #54

Closed Anthuang closed 2 years ago

Anthuang commented 2 years ago

Hello!

We're running into the following issue when importing this package while submitting a Spark job:

Exception in thread "main" java.io.FileNotFoundException: File file:/home/hadoop/.ivy2/jars/io.netty_netty-transport-native-epoll-4.1.63.Final.jar does not exist

This seems to be due to the fact that the actual downloaded jar is io.netty_netty-transport-native-epoll-4.1.63.Final-linux-x86_64.jar rather than io.netty_netty-transport-native-epoll-4.1.63.Final.jar (note that the expected jar has no OS architecture in its name). We verified this by manually renaming the jar in our build cache, and the build succeeded on a subsequent run.

This is the command we're using (redacted):

spark-submit -v --master yarn --deploy-mode cluster ... --packages '...software.amazon.msk:aws-msk-iam-auth:1.1.0'

This is the machine I'm running on:

Distributor ID: Amazon
Description:    Amazon Linux release 2 (Karoo)
Release:    2
Codename:   Karoo

We're using 1.1.0 of aws-msk-iam-auth, and since it doesn't seem like there's any mention of this issue I haven't tried it with the newer versions.

I'm not completely familiar with Java, but from some digging it seems like the issue might be because the OS arch is included with the jar name: https://github.com/netty/netty/blob/4.1/transport-native-epoll/pom.xml#L151. Is this something you can fix on your end, or is this something that should be brought up with the dependency? Also let me know if you need more info.

Thank you in advance :)

sayantacC commented 2 years ago

@Anthuang Were you using aws-msk-iam-auth with the uber jar (built with the shadowJar target) that includes all the packaged dependencies or were you including aws-msk-iam-auth as a dependency of the project?

I am sorry I could not tell exactly.

There have been some changes in how the uber jar is generated but it will only affect if you are using that.

Anthuang commented 2 years ago

I used the generated jar from running gradle clean build. Would using the uber jar solve the issue for us? Sounds like it might

sayantacC commented 2 years ago

I would suggest trying out using the uber jar. The section on Building from Source in the README outlines how the uber jar can be built and used. Please let me know if that helps.

Anthuang commented 2 years ago

Thank you for the suggestion! It builds successfully now, but I'm seeing errors (see below) from the Spark application itself. This might be a long shot, but have you seen this error before? I made sure the Kafka client and the slf4-api jars are included in the classpath.

[2022-02-03 17:38:40.688]Container exited with a non-zero exit code 50. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Error files: stderr, stderr.txt.
Last 4096 bytes of stderr :
eam.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
Anthuang commented 2 years ago

Sorry turns out that was a red herring. The uber jar works well! I also found an issue that referenced a very similar issue (on another version): https://github.com/netty/netty/issues/10887.

sayantacC commented 2 years ago

Closing the issue. Will update troubleshooting section to include guidance on using uber jar in such situations.