search

GoogleCloudDataproc / spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.
Apache License 2.0
375 stars 197 forks source link

failed to read bigquery table in internal network #985

Closed stallapalli20 closed 6 months ago

stallapalli20 commented 1 year ago

Hi !

I am trying to connect to read bigquery table but failing after many retries, can you please help me with this issue.

I am using the following sample code 

from pyspark.sql import SparkSession
spark = SparkSession \
    .builder \
    .appName('spark-bigquery-demo') \
    .getOrCreate()

spark.conf.set("viewsEnabled", "true")
spark.conf.set("materializationDataset", "SampleDataset")
spark.sparkContext.setLogLevel('DEBUG')
# spark.conf.set("httpMaxRetry", '3')

emails = spark.read.format('bigquery') \
    .option("credentialsFile", "big_query_key.json") \
    .option("parentProject", "myproject") \
    .load('select * from `myproject.SampleDataset.sample_data` ')
print(emails.schema.json())
print(emails.show())

Below is the spark submit command I am using

spark-submit --deploy-mode "client" --jars "spark-3.2-bigquery-0.30.0.jar" \
    --conf "spark.driver.extraJavaOptions=-Djavax.net.ssl.trustStore=cert-bundle.jks -Djavax.net.ssl.trustStorePassword=mypass  -Djavax.net.ssl.keyStore=cert-bundle.jks -Djavax.net.ssl.keyStorePassword=mypass"  \
    --verbose tests/bigquery/spark_example.py

I am able to see the column details as below

23/06/01 10:24:36 DEBUG PoolingHttpClientConnectionManager: Connection manager is shutting down
23/06/01 10:24:36 DEBUG PoolingHttpClientConnectionManager: Connection manager shut down
23/06/01 10:24:36 DEBUG BigQueryClient: destinationTable is %s
23/06/01 10:24:36 DEBUG BigQueryClient: running query %s
23/06/01 10:24:36 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol https
23/06/01 10:24:36 DEBUG FsUrlStreamHandlerFactory: Unknown protocol https, delegating to default implementation
23/06/01 10:24:40 DEBUG BigQueryClient: job has finished. %s
{"fields":[{"metadata":{},"name":"id","nullable":true,"type":"long"},{"metadata":{},"name":"name","nullable":true,"type":"string"},{"metadata":{},"name":"greet","nullable":true,"type":"string"}],"type":"struct"}
23/06/01 10:24:44 INFO V2ScanRelationPushDown: 
Output: id#0L, name#1, greet#2

But I dont see the results, and after many retries the job fails with below error. 

23/05/30 14:47:03 DEBUG NettyClientHandler: [id: 0xed96bc5e, L:/xx.xxx.xxx.xxx:60103 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=65536}
23/05/30 14:47:03 DEBUG NettyClientHandler: [id: 0xed96bc5e, L:/xx.xxx.xxx.xxx:60103 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] OUTBOUND SETTINGS: ack=true
23/05/30 14:48:51 DEBUG NettyClientHandler: [id: 0xed96bc5e, L:/xx.xxx.xxx.xxx:60103 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] OUTBOUND RST_STREAM: streamId=3 errorCode=8

py4j.protocol.Py4JJavaError: An error occurred while calling o33.showString.
: com.google.cloud.spark.bigquery.repackaged.com.google.api.gax.rpc.DeadlineExceededException: com.google.cloud.spark.bigquery.repackaged.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 108.177819958s. [closed=[], open=[[buffered_nanos=172853875, remote_addr=bigquerystorage.googleapis.com/xxx.xxx.xx.xxx:443]]]

NOTE : I am able to retrieve the results with the same service account using the non spark approach using python libraries, so I think its not issue with permissions or service account.. from google.cloud import bigquery

davidrabinowitz commented 1 year ago

Is it possible that the Spark is restricted by firewall? Also, is the credentials file is installed on all the cluster? Can you set the full path to the file?

stallapalli20 commented 1 year ago 
23/06/01 12:10:05 DEBUG InternalLoggerFactory: Using SLF4J as the default logging framework
23/06/01 12:10:05 DEBUG PlatformDependent0: -Dio.netty.noUnsafe: false
23/06/01 12:10:05 DEBUG PlatformDependent0: Java version: 8
23/06/01 12:10:05 DEBUG PlatformDependent0: sun.misc.Unsafe.theUnsafe: available
23/06/01 12:10:05 DEBUG PlatformDependent0: sun.misc.Unsafe.copyMemory: available
23/06/01 12:10:05 DEBUG PlatformDependent0: sun.misc.Unsafe.storeFence: available
23/06/01 12:10:05 DEBUG PlatformDependent0: java.nio.Buffer.address: available
23/06/01 12:10:05 DEBUG PlatformDependent0: direct buffer constructor: available
23/06/01 12:10:05 DEBUG PlatformDependent0: java.nio.Bits.unaligned: available, true
23/06/01 12:10:05 DEBUG PlatformDependent0: jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
23/06/01 12:10:05 DEBUG PlatformDependent0: java.nio.DirectByteBuffer.<init>(long, int): available
23/06/01 12:10:05 DEBUG PlatformDependent: sun.misc.Unsafe: available
23/06/01 12:10:05 DEBUG PlatformDependent: -Dio.netty.tmpdir: /var/folders/kw/b3z_q1ys2zv_2lq89qlds2mw0000gn/T (java.io.tmpdir)
23/06/01 12:10:05 DEBUG PlatformDependent: -Dio.netty.bitMode: 64 (sun.arch.data.model)
23/06/01 12:10:05 DEBUG PlatformDependent: Platform: MacOS
23/06/01 12:10:05 DEBUG PlatformDependent: -Dio.netty.maxDirectMemory: 954728448 bytes
23/06/01 12:10:05 DEBUG PlatformDependent: -Dio.netty.uninitializedArrayAllocationThreshold: -1
23/06/01 12:10:05 DEBUG CleanerJava6: java.nio.ByteBuffer.cleaner(): available
23/06/01 12:10:05 DEBUG PlatformDependent: -Dio.netty.noPreferDirect: false
23/06/01 12:10:05 DEBUG NativeLibraryLoader: -Dio.netty.native.workdir: /var/folders/kw/b3z_q1ys2zv_2lq89qlds2mw0000gn/T (io.netty.tmpdir)
23/06/01 12:10:05 DEBUG NativeLibraryLoader: -Dio.netty.native.deleteLibAfterLoading: true
23/06/01 12:10:05 DEBUG NativeLibraryLoader: -Dio.netty.native.tryPatchShadedId: true
23/06/01 12:10:05 DEBUG NativeLibraryLoader: -Dio.netty.native.detectNativeLibraryDuplicates: true
23/06/01 12:10:05 DEBUG NativeLibraryLoader: Execution of 'install_name_tool -id TXTLuqhkYRyO0DOEODXJbshtY /var/folders/kw/b3z_q1ys2zv_2lq89qlds2mw0000gn/T/libcom_google_cloud_spark_bigquery_repackaged_netty_tcnative_osx_x86_645320150711680462113.dylib' succeed: 0
23/06/01 12:10:05 DEBUG NativeLibraryLoader: Execution of 'codesign -s - /var/folders/kw/b3z_q1ys2zv_2lq89qlds2mw0000gn/T/libcom_google_cloud_spark_bigquery_repackaged_netty_tcnative_osx_x86_645320150711680462113.dylib' succeed: 0
23/06/01 12:10:05 DEBUG NativeLibraryLoader: Successfully loaded the library /var/folders/kw/b3z_q1ys2zv_2lq89qlds2mw0000gn/T/libcom_google_cloud_spark_bigquery_repackaged_netty_tcnative_osx_x86_645320150711680462113.dylib
23/06/01 12:10:05 DEBUG NativeLibraryLoader: Loaded library with name 'netty_tcnative_osx_x86_64'
23/06/01 12:10:05 DEBUG OpenSsl: Initialize netty-tcnative using engine: 'default'
23/06/01 12:10:05 DEBUG OpenSsl: netty-tcnative using native library: BoringSSL
23/06/01 12:10:06 DEBUG ResourceLeakDetector: -Dcom.google.cloud.spark.bigquery.repackaged.io.netty.leakDetection.level: simple
23/06/01 12:10:06 DEBUG ResourceLeakDetector: -Dcom.google.cloud.spark.bigquery.repackaged.io.netty.leakDetection.targetRecords: 4
23/06/01 12:10:06 DEBUG AbstractByteBuf: -Dcom.google.cloud.spark.bigquery.repackaged.io.netty.buffer.checkAccessible: true
23/06/01 12:10:06 DEBUG AbstractByteBuf: -Dcom.google.cloud.spark.bigquery.repackaged.io.netty.buffer.checkBounds: true
23/06/01 12:10:06 DEBUG ResourceLeakDetectorFactory: Loaded default ResourceLeakDetector: com.google.cloud.spark.bigquery.repackaged.io.netty.util.ResourceLeakDetector@71961450
23/06/01 12:10:06 DEBUG InternalThreadLocalMap: -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
23/06/01 12:10:06 DEBUG InternalThreadLocalMap: -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.numHeapArenas: 16
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.numDirectArenas: 16
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.pageSize: 8192
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxOrder: 9
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.chunkSize: 4194304
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.smallCacheSize: 256
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.normalCacheSize: 64
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxCachedBufferCapacity: 32768
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.cacheTrimInterval: 8192
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.cacheTrimIntervalMillis: 0
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.useCacheForAllThreads: false
23/06/01 12:10:06 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
23/06/01 12:10:06 DEBUG ByteBufUtil: -Dio.netty.allocator.type: pooled
23/06/01 12:10:06 DEBUG ByteBufUtil: -Dio.netty.threadLocalDirectBufferSize: 0
23/06/01 12:10:06 DEBUG ByteBufUtil: -Dio.netty.maxThreadLocalCharBufferSize: 16384
23/06/01 12:10:06 DEBUG ResourceLeakDetectorFactory: Loaded default ResourceLeakDetector: com.google.cloud.spark.bigquery.repackaged.io.netty.util.ResourceLeakDetector@5d05cceb
23/06/01 12:10:06 DEBUG Recycler: -Dio.netty.recycler.maxCapacityPerThread: 4096
23/06/01 12:10:06 DEBUG Recycler: -Dio.netty.recycler.ratio: 8
23/06/01 12:10:06 DEBUG Recycler: -Dio.netty.recycler.chunkSize: 32
23/06/01 12:10:06 DEBUG Recycler: -Dio.netty.recycler.blocking: false
23/06/01 12:10:06 DEBUG Recycler: -Dio.netty.recycler.batchFastThreadLocalOnly: true
23/06/01 12:10:06 DEBUG PlatformDependent: org.jctools-core.MpscChunkedArrayQueue: available
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 => ECDHE-ECDSA-AES128-GCM-SHA256
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 => ECDHE-ECDSA-AES128-GCM-SHA256
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 => ECDHE-RSA-AES128-GCM-SHA256
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_GCM_SHA256 => ECDHE-RSA-AES128-GCM-SHA256
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-ECDSA-CHACHA20-POLY1305
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-ECDSA-CHACHA20-POLY1305
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-RSA-CHACHA20-POLY1305
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-RSA-CHACHA20-POLY1305
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-PSK-CHACHA20-POLY1305
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-PSK-CHACHA20-POLY1305
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA => ECDHE-ECDSA-AES128-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA => ECDHE-ECDSA-AES128-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA => ECDHE-RSA-AES128-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA => ECDHE-RSA-AES128-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA => ECDHE-PSK-AES128-CBC-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_PSK_WITH_AES_128_CBC_SHA => ECDHE-PSK-AES128-CBC-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA => ECDHE-ECDSA-AES256-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA => ECDHE-ECDSA-AES256-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA => ECDHE-RSA-AES256-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA => ECDHE-RSA-AES256-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA => ECDHE-PSK-AES256-CBC-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_ECDHE_PSK_WITH_AES_256_CBC_SHA => ECDHE-PSK-AES256-CBC-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_RSA_WITH_AES_128_GCM_SHA256 => AES128-GCM-SHA256
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_RSA_WITH_AES_128_GCM_SHA256 => AES128-GCM-SHA256
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_RSA_WITH_AES_256_GCM_SHA384 => AES256-GCM-SHA384
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_RSA_WITH_AES_256_GCM_SHA384 => AES256-GCM-SHA384
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_RSA_WITH_AES_128_CBC_SHA => AES128-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_RSA_WITH_AES_128_CBC_SHA => AES128-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_PSK_WITH_AES_128_CBC_SHA => PSK-AES128-CBC-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_PSK_WITH_AES_128_CBC_SHA => PSK-AES128-CBC-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_RSA_WITH_AES_256_CBC_SHA => AES256-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_RSA_WITH_AES_256_CBC_SHA => AES256-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_PSK_WITH_AES_256_CBC_SHA => PSK-AES256-CBC-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_PSK_WITH_AES_256_CBC_SHA => PSK-AES256-CBC-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: TLS_RSA_WITH_3DES_EDE_CBC_SHA => DES-CBC3-SHA
23/06/01 12:10:06 DEBUG CipherSuiteConverter: Cipher suite mapping: SSL_RSA_WITH_3DES_EDE_CBC_SHA => DES-CBC3-SHA
23/06/01 12:10:06 DEBUG OpenSsl: Supported protocols (OpenSSL): [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2, TLSv1.3] 
23/06/01 12:10:06 DEBUG OpenSsl: Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256]
23/06/01 12:10:06 DEBUG MultithreadEventLoopGroup: -Dio.netty.eventLoopThreads: 16
23/06/01 12:10:06 DEBUG GlobalEventExecutor: -Dio.netty.globalEventExecutor.quietPeriodSeconds: 1
23/06/01 12:10:06 DEBUG NioEventLoop: -Dio.netty.noKeySetOptimization: false
23/06/01 12:10:06 DEBUG NioEventLoop: -Dio.netty.selectorAutoRebuildThreshold: 512
23/06/01 12:10:06 DEBUG ReadSessionCreator: using default preferred min parallelism [24]
23/06/01 12:10:06 DEBUG ReadSessionCreator: using default max parallelism [20000]
23/06/01 12:10:06 DEBUG DefaultChannelId: -Dio.netty.processId: 81080 (auto-detected)
23/06/01 12:10:06 DEBUG NetUtil: -Djava.net.preferIPv4Stack: false
23/06/01 12:10:06 DEBUG NetUtil: -Djava.net.preferIPv6Addresses: false
23/06/01 12:10:06 DEBUG NetUtilInitializations: Loopback interface: lo0 (lo0, 0:0:0:0:0:0:0:1%lo0)
23/06/01 12:10:06 DEBUG NetUtil: Failed to get SOMAXCONN from sysctl and file /proc/sys/net/core/somaxconn. Default: 128
23/06/01 12:10:06 DEBUG DefaultChannelId: -Dio.netty.machineId: bc:d0:74:ff:fe:13:cd:95 (auto-detected)
23/06/01 12:10:06 DEBUG ResourceLeakDetectorFactory: Loaded default ResourceLeakDetector: com.google.cloud.spark.bigquery.repackaged.io.netty.util.ResourceLeakDetector@ca70b7
23/06/01 12:10:06 DEBUG SslHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] HANDSHAKEN: protocol:TLSv1.3 cipher suite:TLS_AES_256_GCM_SHA384
23/06/01 12:10:06 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}
23/06/01 12:10:06 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
23/06/01 12:10:06 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] INBOUND SETTINGS: ack=false settings={HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=100}
23/06/01 12:10:06 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] OUTBOUND SETTINGS: ack=true
23/06/01 12:10:06 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] INBOUND SETTINGS: ack=true
23/06/01 12:10:06 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:authority: bigquerystorage.googleapis.com:443, :path: /google.cloud.bigquery.storage.v1.BigQueryRead/CreateReadSession, :method: POST, :scheme: https, content-type: application/grpc, te: trailers, user-agent: spark-bigquery-connector/0.30.0 spark/3.2.1 java/1.8.0_301-b09 scala/2.12.15 datasource/v2 grpc-java-netty/1.54.0, x-goog-request-params: read_session.table=projects/myproject/datasets/SampleDataset/tables/_bqc_06d2f13c4213400fa26780a4d16a25dc, grpc-accept-encoding: gzip, authorization: Bearer ..-ELNreR0K-Iev8B6jhp2dNbc6vf0xRMPwz_gtnjXQ_JW9bNDEaVNwUyOh7iKM7GldkxowuW6y-S7B4cStDn-iUCNyolgkNPArD11ltAHr_1peiKMQLzNPDw5Fl-D0sxV_zvXqIh2yZa3fN81DRlpqTQuTdvGUFcVfbn0hgqzNroVa9WMOm4678JqJ2WwSSpEe49ABy9ajx8s545viOAaQZXk9378lQLcDm6THSmYatd6ZZpkM_S_lP37U1MH_dmCUp0sTHDHQFhQ, grpc-timeout: 599465m] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
23/06/01 12:10:06 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] OUTBOUND DATA: streamId=3 padding=0 endStream=true length=170 bytes=00000000a50a1f70726f6a656374732f6d6173736d757475616c2d73382d3230796361613331127c1802326370726f6a656374732f6d6173736d757475616c2d...
23/06/01 12:10:07 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=65536}
23/06/01 12:10:07 DEBUG NettyClientHandler: [id: 0x6921a4fc, L:/xx.xxx.xx.xxx:60132 - R:bigquerystorage.googleapis.com/xx.xxx.xx.xxx:443] OUTBOUND SETTINGS: ack=true

@davidrabinowitz I am not sure if firewall is the issue. Above is the full log message, Please suggest if any parameters I can set and try. Also I tried giving absolute path for the key file, but no luck. I am running in my local machine in client mode.

I am already setting ssl parameters in spark.driver.extraJavaOptions as mentioned in the above spark-submit command

isha97 commented 7 months ago

@stallapalli20 Are you still facing this issue?

isha97 commented 6 months ago

Please reopen the issue if you still face this.