Error reported when upgrading to version 0.282

[shiwl]

After upgrading to version 0.282, there was no problem with the master node, but the two slave nodes kept reporting errors:

2023-06-20T11:25:49.919+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-175-44 alluxio.client.block.stream.GrpcDataReader Closing gRPC read stream took longer than 5000ms, moving on. blockId: 5949217570816, address: WorkerNetAddress{host=alluxio04, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio04, rack=null)} 2023-06-20T11:25:49.919+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-179-46 alluxio.client.file.AlluxioFileInStream Failed to read block 5949418897408 of file /warehouse/iceberg_db/ods_weapon_ddos_statistics/data/02e807dc-f06e-4820-ab73-1f0b306a10d3.parquet from worker WorkerNetAddress{host=alluxio06, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio06, rack=null)}. This worker will be skipped for future read operations, will retry: alluxio.exception.status.CancelledException: Interrupted while waiting for response. (Zero Copy GrpcDataReader). 2023-06-20T11:25:49.919+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-175-44 alluxio.client.file.AlluxioFileInStream Failed to read block 5949217570816 of file /warehouse/iceberg_db/ods_weapon_ddos_statistics/data/8ff7a359-347e-4f23-8194-69d0a98ba496.parquet from worker WorkerNetAddress{host=alluxio04, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio04, rack=null)}. This worker will be skipped for future read operations, will retry: alluxio.exception.status.CancelledException: Interrupted while waiting for response. (Zero Copy GrpcDataReader). 2023-06-20T11:25:49.919+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-178-47 alluxio.client.file.AlluxioFileInStream Failed to read block 5949385342976 of file /warehouse/iceberg_db/ods_weapon_ddos_statistics/data/c26546a8-c75f-45f5-a034-fe6dc6a61f83.parquet from worker WorkerNetAddress{host=alluxio05, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio05, rack=null)}. This worker will be skipped for future read operations, will retry: alluxio.exception.status.CancelledException: Interrupted while waiting for response. (Zero Copy GrpcDataReader). 2023-06-20T11:25:49.920+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-182-45 alluxio.client.file.AlluxioFileInStream Failed to read block 5949586669568 of file /warehouse/iceberg_db/ods_weapon_ddos_statistics/data/9c44e8e8-cd03-4a73-9558-fd12a599a7b4.parquet from worker WorkerNetAddress{host=alluxio05, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio05, rack=null)}. This worker will be skipped for future read operations, will retry: alluxio.exception.status.CancelledException: Interrupted while waiting for response. (Zero Copy GrpcDataReader). 2023-06-20T11:25:49.920+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-176-41 alluxio.client.file.AlluxioFileInStream Failed to read block 5949318234112 of file /warehouse/iceberg_db/ods_weapon_ddos_statistics/data/c2734b0d-f3e3-42b7-9ce8-f21a605848aa.parquet from worker WorkerNetAddress{host=alluxio06, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio06, rack=null)}. This worker will be skipped for future read operations, will retry: alluxio.exception.status.CancelledException: Interrupted while waiting for response. (Zero Copy GrpcDataReader). 2023-06-20T11:25:49.921+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-180-40 alluxio.client.file.AlluxioFileInStream Failed to read block 5949486006272 of file /warehouse/iceberg_db/ods_weapon_ddos_statistics/data/99a4190b-5668-47bd-91db-4ef7c652eeca.parquet from worker WorkerNetAddress{host=alluxio01, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio01, rack=null)}. This worker will be skipped for future read operations, will retry: alluxio.exception.status.CancelledException: Interrupted while waiting for response. (Zero Copy GrpcDataReader). 2023-06-20T11:25:49.921+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-177-43 alluxio.client.file.AlluxioFileInStream Failed to read block 5949402120192 of file /warehouse/iceberg_db/ods_weapon_ddos_statistics/data/e99ac63c-6432-4d89-93e2-3ba7582dc84c.parquet from worker WorkerNetAddress{host=alluxio03, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio03, rack=null)}. This worker will be skipped for future read operations, will retry: alluxio.exception.status.CancelledException: Interrupted while waiting for response. (Zero Copy GrpcDataReader). 2023-06-20T11:25:49.921+0800 WARN 20230620_032517_00264_q3yrz.1.0.1.0-181-42 alluxio.client.file.AlluxioFileInStream Failed to read block 5949502783488 of file /warehouse/iceberg_db/ods_weapon_ddos_statistics/data/bc78f956-3278-42d9-8566-7b3b8a9cf387.parquet from worker WorkerNetAddress{host=alluxio06, containerHost=, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=alluxio06, rack=null)}. This worker will be skipped for future read operations, will retry: alluxio.exception.status.CancelledException: Interrupted while waiting for response. (Zero Copy GrpcDataReader).

[imjalpreet]

@shiwl Is this an intermittent issue or are you facing this on all the queries?

CC: @beinan @apc999

[shiwl]

@shiwl Is this an intermittent issue or are you facing this on all the queries?

CC: @beinan @apc999

From the log, it seems that every query will print this warning

[tdcmeehan]

CC: @beinan