search

exasol / cloud-storage-extension

Exasol Cloud Storage Extension for accessing formatted data Avro, Orc and Parquet, on public cloud storage systems
MIT License
7 stars 11 forks source link

Update alluxio dependency version #263

Closed morazow closed 1 month ago

morazow commented 11 months ago

Situation

We should update the alluxio dependency to 302 version. With this it there are exceptions:

  java.sql.SQLDataException: VM error: F-UDF-CL-LIB-1127: F-UDF-CL-SL-JAVA-1002: F-UDF-CL-SL-JAVA-1013:                                                                                         
com.exasol.ExaUDFException: F-UDF-CL-SL-JAVA-1080: Exception during run                                                                                                                         
java.lang.IllegalArgumentException: No Under File System Factory found for: /opt/alluxio/underFSStorage                                                                                         
alluxio.underfs.UnderFileSystem$Factory.createWithRecorder(UnderFileSystem.java:114)                                                                                                            
alluxio.underfs.UnderFileSystem$Factory.create(UnderFileSystem.java:94)                                                                                                                         
alluxio.underfs.UnderFileSystem$Factory.create(UnderFileSystem.java:81)                                                                                                                         
alluxio.client.file.ufs.UfsBaseFileSystem.lambda$new$0(UfsBaseFileSystem.java:109)                                                                                                              
alluxio.underfs.UfsManager$UfsClient.acquireUfsResource(UfsManager.java:62)                                                                                                                     
alluxio.client.file.ufs.UfsBaseFileSystem.<init>(UfsBaseFileSystem.java:131)                                                                                                                    
alluxio.client.file.ufs.UfsBaseFileSystem.<init>(UfsBaseFileSystem.java:108)                                                                                                                    
alluxio.client.file.FileSystem$Factory.create(FileSystem.java:183)                                                                                                                              
alluxio.client.file.FileSystem$Factory.create(FileSystem.java:166)                                                                                                                              
alluxio.client.file.FileSystem$Factory.create(FileSystem.java:158)                                                                                                                              
alluxio.hadoop.AbstractFileSystem.initialize(AbstractFileSystem.java:572)                                                                                                                       
alluxio.hadoop.AbstractFileSystem.initialize(AbstractFileSystem.java:509)                                                                                                                       
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)                                                                                                                          
org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)                                                                                                                                 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712)                                                                                                                         
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663)                                                                                                                                 
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)                                                                                                                                        
org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)                                                                                                                                          
org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58)                                                                                                              
org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:677)
com.exasol.cloudetl.parquet.ParquetRowWriter$.apply(ParquetRowWriter.scala:37)
com.exasol.cloudetl.sink.BatchSizedSink$$anon$1.<init>(BatchSizedSink.scala:54)
com.exasol.cloudetl.sink.BatchSizedSink.createWriter(BatchSizedSink.scala:51)
com.exasol.cloudetl.sink.BatchSizedSink.openNewFile(BatchSizedSink.scala:106)
com.exasol.cloudetl.sink.BatchSizedSink.write(BatchSizedSink.scala:75)
com.exasol.cloudetl.scriptclasses.TableDataExporter$.runExport(TableDataExporter.scala:53)
com.exasol.cloudetl.scriptclasses.TableDataExporter$.run(TableDataExporter.scala:36)
com.exasol.cloudetl.scriptclasses.TableDataExporter.run(TableDataExporter.scala)
com.exasol.ExaWrapper.run(ExaWrapper.java:197)
 (Session: 1773757907427328000)
  at com.exasol.jdbc.ExceptionFactory.createSQLException(ExceptionFactory.java:33)
  at com.exasol.jdbc.EXASQLException.getSQLExceptionIntern(EXASQLException.java:50)
  at com.exasol.jdbc.AbstractEXAStatement.execute(AbstractEXAStatement.java:478)
  at com.exasol.jdbc.EXAStatement.execute(EXAStatement.java:289)
  at com.exasol.cloudetl.BaseIntegrationTest.executeStmt(BaseIntegrationTest.scala:52)
  at com.exasol.cloudetl.BaseIntegrationTest.executeStmt$(BaseIntegrationTest.scala:51)
  at com.exasol.cloudetl.alluxio.AlluxioExportImportIT.executeStmt(AlluxioExportImportIT.scala:16)
  at com.exasol.cloudetl.alluxio.AlluxioExportImportIT.exportIntoAlluxio(AlluxioExportImportIT.scala:120)
  at com.exasol.cloudetl.alluxio.AlluxioExportImportIT.$anonfun$new$1(AlluxioExportImportIT.scala:76)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
  ...

Acceptance Criteria

morazow commented 11 months ago

The two following dependencies should be updated properly. They break or miss classes for integration tests.

        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-netty</artifactId>
            <version>1.56.1</version>
        </dependency>
        <dependency>
            <groupId>org.alluxio</groupId>
            <artifactId>alluxio-core-client-hdfs</artifactId>
            <version>300</version>
        </dependency>