elmiko
commented
2 years ago first off, thank you for such a detailed issue @vemonet
unfortunately, i do not have much in the way of advice as i am not using these projects actively. in the past, when i have seen issues with collect going off and doing it's own thing it's usually because there is a communication issue between the compute instances and the controller instance.
if you haven't seen the radanalytics website tutorials, you might investigate those for some inspiration. although i don't think we have a dataframes example specifically, you will see several different application deployment styles and many of them are using pySpark.
hope that helps =)
edit:
just an afterthought, you might need to expose some sort of service on your driver application. it's possible that the collected results are not returning properly to that process. although, i would expect to see something in the logs if that were the case.
vemonet
jkremser
Description:
I deployed the radanalytics/spark-operator on OKD 4.6 (using OpenDataHub, you can find the full ODH manifests we are using here: https://github.com/MaastrichtU-IDS/odh-manifests)
From this spark-operator I started a Spark cluster (1 master, 10 workers, no limit)
When I am trying to simply create and collect a simple dataframe on this Spark cluster, creating works, but collecting get stuck
Creating runs in about 3s:
Collecting just get stuck:
I made sure to use the exact same Spark version (3.0.1) everywhere (spark cluster images, spark local executable, pyspark version: everything is 3.0.1)
These are the logs displayed by the Master and Workers nodes does not seems to contain any interesting informations:
Workers spam this exact code block every 3 seconds (just changing a bit the IDs). The words are english, they can be read, but the sentences they are producing are not giving any relevant informations on what's happening:
Master gives even less informations:
I tried different number of cores/limitations when deploying the Spark cluster, but every single Spark cluster deployed with the Radanalytics operator never manage to
.collect()anything. But we always manage to connect to the Spark cluster and somehow create a dataframe on it (at least it seems like, not sure if the dataframe is really created)Steps to reproduce:
alphachannel I guess, as usual with Operators there is not an easy way to share which version we use (not sure why it was designed this way, it really make reproducibility hard to achieve, anyway who cares about reproducibility in computer science?), but you can check the OpenDataHub subscription: https://github.com/MaastrichtU-IDS/odh-manifests/blob/dsri/radanalyticsio/spark/cluster/base/subscription.yamlspark_cluster_url = "spark://spark-cluster:7077" spark = SparkSession.builder.master(spark_cluster_url).getOrCreate() df = spark.createDataFrame([ Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)), Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)), Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0)) ]) df.collect()