search

Azure / Azure-Orbital-Analytics-Samples

Sample solution that demonstrates how to deploy and analyze spaceborne data using Azure Synapse Analytics
https://aka.ms/synapse-geospatial-analytics
MIT License
30 stars 24 forks source link

A Spark pool cannot have 1 nodes; it must have between 3 and 200 nodes. #75

Closed ivo-andreev closed 2 years ago

ivo-andreev commented 2 years ago

While running the installation script, there is another error about Spark autoscaling configuration

{"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"Conflict\",\r\n \"message\": \"{\r\n \\"status\\": \\"Failed\\",\r\n \\"error\\": {\r\n \\"code\\": \\"ResourceDeploymentFailure\\",\r\n \\"message\\": \\"The resource operation completed with terminal provisioning state 'Failed'.\\",\r\n \\"details\\": [\r\n {\r\n \\"code\\": \\"DeploymentFailed\\",\r\n \\"message\\": \\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\\",\r\n \\"details\\": [\r\n {\r\n \\"code\\": \\"BadRequest\\",\r\n \\"message\\": \\"{\\r\\n \\\\"error\\\\": {\\r\\n \\\\"code\\\\": \\\\"ValidationFailed\\\\",\\r\\n \\\\"message\\\\": \\\\"Spark pool request validation failed.\\\\",\\r\\n \\\\"details\\\\": [\\r\\n {\\r\\n \\\\"code\\\\": \\\\"NodeCountNotValid\\\\",\\r\\n \\\\"message\\\\": \\\\"The autoscale minimum node count is not valid. A Spark pool cannot have 1 nodes; it must have between 3 and 200 nodes.\\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"}]}}

ivo-andreev commented 2 years ago

Setting up synapseAutoScaleMinNodeCount int = 3 in synapse.sparkpool.bicep and pipeline.bicep fixes the issue

sjyang18 commented 2 years ago

I guess that Azure Synapse team changed the min scaling number recently. When we worked on this project, we didn't notice this restriction. It would be nice if we see contributions from outside.

ivo-andreev commented 2 years ago

I guess that Azure Synapse team changed the min scaling number recently. When we worked on this project, we didn't notice this restriction. It would be nice if we see contributions from outside.

I have a proposal about how this could be fixed, as pointed out above, but I am not convinced in triggering a pull request.