No service type 'SPARK' available for cluster with version 'CDH 6.3.0'

[bvkudupa]

Hi, I am able to install HDFS, YARN and Zookeeper. But facing issue while installing spark. when I run python deploycloudera.py -ihivepass -fCloudera@123 -nCloudera@123 -rCloudera@123

Traceback (most recent call last): File "deploycloudera.py", line 1079, in main() File "deploycloudera.py", line 1028, in main spark_service = deploy_spark(CLUSTER, SPARK_SERVICE_NAME, SPARK_SERVICE_CONFIG, SPARK_MASTER_HOST, SPARK_MASTER_CONFIG, SPARK_WORKER_HOSTS, SPARK_WORKER_CONFIG, SPARK_GW_HOSTS, SPARK_GW_CONFIG) File "deploycloudera.py", line 689, in deploy_spark spark_service = cluster.create_service(spark_service_name, "SPARK") File "/usr/lib/python2.7/site-packages/cm_api/endpoints/clusters.py", line 161, in create_service service_type, self.name) File "/usr/lib/python2.7/site-packages/cm_api/endpoints/services.py", line 44, in create_service ApiService, True, data=[apiservice])[0] File "/usr/lib/python2.7/site-packages/cm_api/endpoints/types.py", line 137, in call ret = method(path, data=data, params=params) File "/usr/lib/python2.7/site-packages/cm_api/resource.py", line 148, in post self._make_headers(contenttype)) File "/usr/lib/python2.7/site-packages/cm_api/resource.py", line 73, in invoke headers=headers) File "/usr/lib/python2.7/site-packages/cm_api/http_client.py", line 193, in execute raise self._exc_class(ex) cm_api.api_client.ApiException: No service type 'SPARK' available for cluster with version 'CDH 6.3.0'. (error 400).

What is the service name for spark in CDH 6.3.0?

[funes79]

SPARK2 or SPARK2_ON_YARN

[bvkudupa]

with SPARK2 I get the same exception as No service type 'SPARK2' available for cluster with version 'CDH 6.3.0'. (error 400) When I use SPARK2_ON_YARN, I get below error. cm_api.api_client.ApiException: Unknown configuration attribute 'hdfs_service' for service (type: 'SPARK2_ON_YARN', name: 'SPARK2_ON_YARN'). (error 400)

Please find below relevant code snippet: `### Spark ### SPARK_SERVICE_NAME = "SPARK2_ON_YARN" SPARK_SERVICE_CONFIG = { 'hdfs_service': HDFS_SERVICE_NAME, } SPARK_MASTER_HOST = SPARK_HOST SPARK_MASTER_CONFIG = { 'master_max_heapsize': 67108864, } SPARK_WORKER_HOSTS = list(CLUSTER_HOSTS) SPARK_WORKER_CONFIG = { 'executor_total_max_heapsize': 67108864, 'worker_max_heapsize': 67108864, } SPARK_GW_HOSTS = list(CLUSTER_HOSTS) SPARK_GW_CONFIG = { }

Deploys spark - master, workers, gateways

def deploy_spark(cluster, spark_service_name, spark_service_config, spark_master_host, spark_master_config, spark_worker_hosts, spark_worker_config, spark_gw_hosts, spark_gw_config): spark_service = cluster.create_service(spark_service_name, "SPARK2_ON_YARN") spark_service.update_config(spark_service_config)

sm = spark_service.get_role_config_group("{0}-SPARK_MASTER-BASE".format(spark_service_name)) sm.update_config(spark_master_config) spark_service.create_role("{0}-sm".format(spark_service_name), "SPARK_MASTER", spark_master_host)

sw = spark_service.get_role_config_group("{0}-SPARK_WORKER-BASE".format(spark_service_name)) sw.update_config(spark_worker_config)

worker = 0 for host in spark_worker_hosts: worker += 1 spark_service.create_role("{0}-sw-".format(spark_service_name) + str(worker), "SPARK_WORKER", host)

gw = spark_service.get_role_config_group("{0}-GATEWAY-BASE".format(spark_service_name)) gw.update_config(spark_gw_config)

gateway = 0 for host in spark_gw_hosts: gateway += 1 spark_service.create_role("{0}-gw-".format(spark_service_name) + str(gateway), "GATEWAY", host)

return spark_service`

Please let me know if any other places to change. Thanks :)

[smareti]

I am also getting same kind of issue..below. I am trying to install on CDH 5.14.4

Traceback (most recent call last): File "cm-api-deploy.py", line 1001, in deploy.gen_service_types_and_objects() File "cm-api-deploy.py", line 339, in gen_service_types_and_objects self.service_types_and_names[svc_type], svc_type) # Service name & type File "/usr/lib/python2.6/site-packages/cm_api/endpoints/clusters.py", line 161, in create_service service_type, self.name) File "/usr/lib/python2.6/site-packages/cm_api/endpoints/services.py", line 44, in create_service ApiService, True, data=[apiservice])[0] File "/usr/lib/python2.6/site-packages/cm_api/endpoints/types.py", line 137, in call ret = method(path, data=data, params=params) File "/usr/lib/python2.6/site-packages/cm_api/resource.py", line 148, in post self._make_headers(contenttype)) File "/usr/lib/python2.6/site-packages/cm_api/resource.py", line 73, in invoke headers=headers) File "/usr/lib/python2.6/site-packages/cm_api/http_client.py", line 193, in execute raise self._exc_class(ex) cm_api.api_client.ApiException: No service type 'SPARK2_ON_YARN' available for cluster with version 'CDH 5.0.0'. (error 400)

[smareti]

I have installed another parcel to get install spark 2 service. here is the example parcel: SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el6.parcel