TestHudiIntegration.testDemoQuery and other test failures in pipeline

czentgr commented 7 months ago

https://github.com/prestodb/presto/actions/runs/8070233507/job/22047081223?pr=22024

Your Environment

n/a

Expected Behavior

Current Behavior

 Error:  Failures: 
Error:    TestHudiIntegration.testDemoQuery1:94->AbstractTestQueryFramework.assertQuery:159 Execution of 'actual' query failed: SELECT symbol, max(ts) FROM stock_ticks_mor_rt GROUP BY symbol HAVING symbol = 'GOOG'
Error:    TestHudiIntegration.testDemoQuery2:117->AbstractTestQueryFramework.assertQuery:159 Execution of 'actual' query failed: SELECT "_hoodie_record_key", symbol, ts, volume, open, close  FROM stock_ticks_mor_rt WHERE symbol = 'GOOG'
Error:    TestHudiIntegration.testQueryWithPartitionColumn:137->AbstractTestQueryFramework.assertQuery:159 Execution of 'actual' query failed: SELECT symbol, ts, dt FROM stock_ticks_mor_rt WHERE symbol = 'GOOG' AND dt = '2018-08-31'

Main cause appears to be

Caused by: java.lang.UnsupportedOperationException: Not implemented by the RawLocalFileSystem FileSystem implementation
    at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:219)
    at org.apache.hadoop.fs.HadoopExtendedFileSystem.getScheme(HadoopExtendedFileSystem.java:71)
    at org.apache.hudi.common.fs.FSUtils.isGCSFileSystem(FSUtils.java:649)
    at org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:501)
    at org.apache.hudi.common.table.log.HoodieLogFileReader.<init>(HoodieLogFileReader.java:120)
    at org.apache.hudi.common.table.log.HoodieLogFormatReader.<init>(HoodieLogFormatReader.java:70)
    at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:245)
    ... 32 more

Possible Solution

Steps to Reproduce

n/a

Screenshots (if appropriate)

Context

imjalpreet commented 6 months ago

Also observed this here: https://github.com/prestodb/presto/actions/runs/8530290591/job/23367795134?pr=21622

elharo commented 6 months ago

seems to be one of the more common flakes in the last few days

czentgr commented 5 months ago

I was looking into this. Somehow the hudi library is trying to call a function that isn't supported by the file system (the function is getScheme()).

This originates from

at org.apache.hudi.common.fs.FSUtils.isGCSFileSystem(FSUtils.java:649)

This function is curious because of the file system is a RawFileSystem the getScheme() isn't applied but the code in Hudi nonetheless would try to call it. So not clear which paths this would normally take.

Currently, this function is in HadoopFSUtils (was refactored away from FSUtils)

  /**
   * This is due to HUDI-140 GCS has a different behavior for detecting EOF during seek().
   *
   * @param fs fileSystem instance.
   * @return true if the inputstream or the wrapped one is of type GoogleHadoopFSInputStream
   */
  public static boolean isGCSFileSystem(FileSystem fs) {
    return fs.getScheme().equals(StorageSchemes.GCS.getScheme());
  }

There seems to be some assumption that all filesystems that can come here support getScheme(). But it is not clear how the test can switch and the error isn't consistent. For testing purposes I would assume that the file system is always a local file system.

This code path was changed recently completely and perhaps upgrading the Hudi jar might help.

elharo commented 5 months ago

Again today:

TestHudiIntegration.testDemoQuery2 flakes:

Error: Tests run: 2977, Failures: 1, Errors: 0, Skipped: 90, Time elapsed: 2,915.17 s <<< FAILURE! - in TestSuite Error: com.facebook.presto.hive.hudi.TestHudiIntegration.testDemoQuery2 Time elapsed: 0.972 s <<< FAILURE! java.lang.AssertionError: Execution of 'actual' query failed: SELECT "_hoodie_record_key", symbol, ts, volume, open, close FROM stock_ticks_mor_rt WHERE symbol = 'GOOG' at org.testng.Assert.fail(Assert.java:98) at com.facebook.presto.tests.QueryAssertions.assertQuery(QueryAssertions.java:178) at com.facebook.presto.tests.QueryAssertions.assertQuery(QueryAssertions.java:106) at com.facebook.presto.tests.AbstractTestQueryFramework.assertQuery(AbstractTestQueryFramework.java:164) at com.facebook.presto.hive.hudi.TestHudiIntegration.testDemoQuery2(TestHudiIntegration.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:135) at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:673) at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:220) at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50) at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:945) at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:193) at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146) at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.RuntimeException: Exception when constructing record reader at com.facebook.presto.tests.AbstractTestingPrestoClient.execute(AbstractTestingPrestoClient.java:124) at com.facebook.presto.tests.DistributedQueryRunner.execute(DistributedQueryRunner.java:733) at com.facebook.presto.tests.DistributedQueryRunner.execute(DistributedQueryRunner.java:701) at com.facebook.presto.tests.QueryAssertions.assertQuery(QueryAssertions.java:175) ... 18 more Caused by: org.apache.hudi.exception.HoodieException: Exception when constructing record reader at org.apache.hudi.hadoop.realtime.HoodieRealtimeRecordReader.constructRecordReader(HoodieRealtimeRecordReader.java:80) at org.apache.hudi.hadoop.realtime.HoodieRealtimeRecordReader.(HoodieRealtimeRecordReader.java:47) at org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat.getRecordReader(HoodieParquetRealtimeInputFormat.java:85) at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:293) at com.facebook.presto.hive.GenericHiveRecordCursorProvider.lambda$createRecordCursor$0(GenericHiveRecordCursorProvider.java:81) at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23) at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:81) at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:80) at com.facebook.presto.hive.HivePageSourceProvider.getPageSourceFromCursorProvider(HivePageSourceProvider.java:601) at com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:465) at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:217) at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:63) at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:80) at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:250) at com.facebook.presto.operator.Driver.processInternal(Driver.java:441) at com.facebook.presto.operator.Driver.lambda$processFor$10(Driver.java:324) at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:750) at com.facebook.presto.operator.Driver.processFor(Driver.java:317) at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1079) at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:165) at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:621) at com.facebook.presto.$gen.Presto_0_288_SNAPSHOT_3258939testversion__20240509_183530_10442.run(Unknown Source) ... 3 more Caused by: org.apache.hudi.exception.HoodieException: Could not create HoodieRealtimeRecordReader on path file:/tmp/PrestoTest8239072451865457221/data/stock_ticks_mor/2018/08/31/21863880-1cc8-4cda-9a39-44c5f76595f9-0_0-28-26_20220310031208890.parquet at org.apache.hudi.hadoop.realtime.AbstractRealtimeRecordReader.(AbstractRealtimeRecordReader.java:101) at org.apache.hudi.hadoop.realtime.RealtimeCompactedRecordReader.(RealtimeCompactedRecordReader.java:68) at org.apache.hudi.hadoop.realtime.HoodieRealtimeRecordReader.constructRecordReader(HoodieRealtimeRecordReader.java:70) ... 24 more Caused by: org.apache.hudi.exception.HoodieException: Error ordering fields for storage read. #fieldNames: 9, #fieldPositions: 5 at org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils.orderFields(HoodieRealtimeRecordReaderUtils.java:277) at org.apache.hudi.hadoop.realtime.AbstractRealtimeRecordReader.init(AbstractRealtimeRecordReader.java:153) at org.apache.hudi.hadoop.realtime.AbstractRealtimeRecordReader.(AbstractRealtimeRecordReader.java:98) ... 26 more

2024-05-09T13:20:06.101-0600 INFO Thread-3214 org.apache.ranger.audit.provider.AuditProviderFactory ==> JVMShutdownHook.run() 2024-05-09T13:20:06.101-0600 INFO Thread-3214 org.apache.ranger.audit.provider.AuditProviderFactory JVMShutdownHook: Signalling async audit cleanup to start. [INFO] [INFO] Results: [INFO] Error: Failures: Error: TestHudiIntegration.testDemoQuery2:117->AbstractTestQueryFramework.assertQuery:164 Execution of 'actual' query failed: SELECT "_hoodie_record_key", symbol, ts, volume, open, close FROM stock_ticks_mor_rt WHERE symbol = 'GOOG'

hantangwangd commented 2 months ago

+1

https://github.com/prestodb/presto/actions/runs/10190762809/job/28190997119?pr=23347

imjalpreet commented 2 months ago

Observed it again today: https://github.com/prestodb/presto/actions/runs/10366329899/job/28695346187?pr=23401

prestodb / presto