Open theosib-amazon opened 1 year ago
If I use a vanilla Spark driver, then I can run the tests to completion. I can see that the worker and master nodes are using CPU. So it seems like something about the Gluten plugin or the configuration breaks the driver.
The deployment is a big issue now. we can't put all dependency library in jar. So we have to manually install the libraries on each worker node manually now. We are creating a script to install these libraries as temp solution. In long term I'd think the conda env is the best solution. We just need to copy the conda env to each worker node and set the LD_LIBRARY_PATH.
Do you load data from S3? We haven't test S3 internally. Currently only localfs and hdfs are tested.
I install all the necessary jars on the worker nodes, and I can access S3. I just realized that if I use a vanilla driver, it's not going to do planning properly for using Velox. So we gotta get past this hang in the driver.
Looks there are some issues to use S3. we will try to reproduce it and see where is the issue. @weiting-chen can you try this?
I had a similar problem but different from him, I checked the stderr of the Executor and I found out that there was an error in the initialization of libvelox.so by the executor. The error message is: libvelox.so: undefined symbol: _ZN6google21kLogSiteUninitializedE I checked some information about this and it seems to be a glog version problem, the variable kLogSiteUninitialized was removed in the latest version of glog. Although gluten compiled velox specifying the glog version as 0.4.0, I turned on vcpkg and it seems to be downloading the latest 0.6.0 version of glog for me. btw, The problem is occasional, not inevitable.
I have configured two clusters of six r5d.8xlarge nodes on EC2 and manually configured Spark 3.2.2 with hadoop 3.2, which I downloaded from the Apache Spark website.
On the baseline cluster, I configured vanilla Spark, and I'm able to run all TPC-DS queries. The main things I had to add to get that to work were a few jars to enable accessing S3:
And I put this in the spark-defaults.conf file for every node:
The second cluster is basically the same, but I added the following jars to spark-3.2.2-bin-hadoop3.2/jars on all nodes:
And I had to install some system libraries manually. This is from my script that configures all the nodes:
And finally, the spark-defaults.conf file is different:
After all this, I can get master and worker nodes to start and find each other. However, when I try to run a query on the cluster, the driver hangs. None of the worker or master log files indicate any problems. The driver just stops forever. Here's the tail end of what is printed by the driver: