failed to access class org.apache.iceberg.flink.FlinkEnvironmentContext from class org.apache.iceberg.flink.TableLoader$HadoopTableLoader

hashmapybx commented 4 weeks ago

Apache Iceberg version

1.4.3

Query engine

Flink

Please describe the bug 🐞

I try to execution compaction small flie by flink-rewrite-action,https://iceberg.apache.org/docs/nightly/flink-actions/.When I submit my program, there is a error: java.lang.IllegalAccessError: failed to access class org.apache.iceberg.flink.FlinkEnvironmentContext from class org.apache.iceberg.flink.TableLoader$HadoopTableLoader (org.apache.iceberg.flink.FlinkEnvironmentContext is in unnamed module of loader 'app'; org.apache.iceberg.flink.TableLoader$HadoopTableLoader is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @878537d) at org.apache.iceberg.flink.TableLoader$HadoopTableLoader.loadTable(TableLoader.java:106) at broadcast.iceberg.FlinkrewriteData2.main(FlinkrewriteData2.java:25) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:102) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1172) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:272) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1430) at org.apache.flink.client.cli.CliFrontend.lambda$mainWithReturnCodeAndException$5(CliFrontend.java:617) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2065) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.mainWithReturnCodeAndException(CliFrontend.java:617) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1491)

Willingness to contribute

[X] I can contribute a fix for this bug independently
[X] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
[X] I cannot contribute a fix for this bug at this time

hashmapybx commented 4 weeks ago

   System.out.println("执行合并操作");
    LOG.info("===============执行合并操作=====================");
    ParameterTool tool = ParameterTool.fromArgs(args);
    StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();

    TableLoader tableLoader = TableLoader.fromHadoopTable(tool.get("hdfs-path-table"));
    Table table = tableLoader.loadTable();
    RewriteDataFilesActionResult result = Actions.forTable(environment,table)
            .rewriteDataFiles()
            .maxParallelism(1)
            .targetSizeInBytes(128 * 1024 * 1024)
            .execute();

    System.out.println("合并任务启动");

hashmapybx commented 4 weeks ago

pvary commented 4 weeks ago

Normally, this error is caught by the compiler; this error can only occur at run time if the definition of a class has incompatibly changed.

Could you please check the class versions on your class path?

hashmapybx commented 3 weeks ago

Normally, this error is caught by the compiler; this error can only occur at run time if the definition of a class has incompatibly changed.

Could you please check the class versions on your class path?

do you mean i need to check flink and iceberg version?

pvary commented 2 weeks ago

I would guess that are multiple versions of this class on your classpath, or you have a different version of the class on your classpath, than you complied your code with.

apache / iceberg