Synapse Spark Pool "Workspace packages" Option

olivergaudreault commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

In the Synapse Spark Pool portal there is an option for "Workspace Packages" but this isn't available in terraform. Can it be added?

New or Affected Resource(s)

azurerm_synapse_spark_pool

Potential Terraform Configuration

N/A

References

N/A

DesaCh01 commented 3 years ago

Hello @olivergaudreault,

I tried to install some python libraries like treelib in the spark pool while its deployment via Terraform. I specified the code to that in library_requirement code block for azurerm_synapse_spark_pool resource.

But, I encountered the following error while doing so,

azurerm_synapse_spark_pool.SparkPool_withlib: Still creating... [19m30s elapsed]
azurerm_synapse_spark_pool.SparkPool_withlib: Still creating... [19m40s elapsed]
azurerm_synapse_spark_pool.SparkPool_withlib: Still creating... [19m50s elapsed]
azurerm_synapse_spark_pool.SparkPool_withlib: Still creating... [20m0s elapsed]

Error: waiting on creating future for Synapse Spark Pool "sparkpoollib" (Workspace "fidospark-dev" / Resource Group "shared-fido-dev-eastus2"): Code="CreateOrUpdateSparkComputeFailed" Message="DurableTask.Core.Exceptions.TaskFailedException : LibraryManagement - Spark Job for sparkpoollib in workspace fidospark-dev in subscription 29da7f08-209a-465d-bc97-c3ee3ab0e1db failed with status: \r\n{\"id\":3,\"appId\":\"application_1626877786327_0001\",\"appInfo\":{\"driverLogUrl\":\"http://e7ab47e4c5244301a852f638d14607ac00262629548:8042/node/containerlogs/container_1626877786327_0001_02_000001/trusted-service-user\",\"sparkUiUrl\":\"http://e7ab47e4c5244301a852f638d14607ac00262629548:8088/proxy/application_1626877786327_0001/\",\"isSessionTimedOut\":null},\"state\":\"dead\",\"log\":[\"\\t user: trusted-service-user\",\"21/07/21 14:30:02 INFO ShutdownHookManager: Shutdown hook called\",\"21/07/21 14:30:02 INFO ShutdownHookManager: Deleting directory /tmp/spark-f19b257f-ab2f-4060-a766-65d0d909756b\",\"21/07/21 14:30:02 INFO ShutdownHookManager: Deleting directory /tmp/spark-b8e304d0-a0e2-4241-9943-2a028f043237\",\"21/07/21 14:30:02 INFO MetricsSystemImpl: Stopping azure-file-system metrics system...\",\"21/07/21 14:30:02 INFO MetricsSystemImpl: azure-file-system metrics system stopped.\",\"21/07/21 14:30:02 INFO MetricsSystemImpl: azure-file-system metrics system shutdown complete.\",\"\\nstderr: \",\"\\nYARN Diagnostics: \",\"User application exited with status 1\"]}\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at DurableTask.Core.TaskOrchestrationContext.<ScheduleTaskInternal>d__17.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at DurableTask.Core.TaskOrchestrationContext.<ScheduleTaskToWorker>d__16`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at DurableTask.Core.TaskOrchestrationContext.<ScheduleTask>d__15`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.ProvisioningWorkerSvc.Workflows.SparkCompute.Orchestration.CreateOrUpdateSparkComputeOrchestration.<ExecuteProvisioningWorkflow>d__7.MoveNext() in S:\\src\\services\\provisioning\\Microsoft.Analytics.ProvisioningWorkerSvc\\Workflows\\SparkCompute\\Orchestration\\CreateOrUpdateSparkComputeOrchestration.cs:line 84"

  on spark-pool-lib.tf line 1, in resource "azurerm_synapse_spark_pool" "SparkPool_withlib":
   1: resource "azurerm_synapse_spark_pool" "SparkPool_withlib"

Also, I'm pasting the code block used by me.

resource "azurerm_synapse_spark_pool" "SparkPool_withlib" {
  name                 = "sparkpoollib"
  synapse_workspace_id = azurerm_synapse_workspace.synapse_2.id
  node_size_family     = "MemoryOptimized"
  node_size            = "Small"
  spark_version        = "2.4"
  spark_log_folder     = "/logs"
  spark_events_folder  = "/events"

  auto_scale {
    max_node_count = 10
    min_node_count = 3
  }

  auto_pause {
    delay_in_minutes = 5
  }

  library_requirement {
    content  = <<EOF
scipy==1.5.2
joblib==0.16.0
threadpoolctl==2.1.0
scikit-learn==0.23.2
Boruta==0.3.0
graphviz==0.14.1
treelib==1.6.1
EOF
    filename = "requirements.txt"
  }

  tags = module.metadata.tags

}

Would appreciate any help on this.

Thanks.

olivergaudreault commented 3 years ago

Hi @DesaCh01 Does it work without the library requirement? What version of terraform and providers are you running? When I did it i used something like <<REQUIREMENT pyYaml==.... REQUIREMENT

DesaCh01 commented 3 years ago

Yes @olivergaudreault , it works fine without the library requirement code block.

olivergaudreault commented 3 years ago

What versions you running? I am running ad: 1.6.0 rm: 2.67.0 terraform: 1.0.0

olivergaudreault commented 3 years ago

If the version isn't it then try changing <<EOF

to REQUIREMENTS or REQUIREMENT - i don't recall which one I used. I don't think this is the problem but just try I guess and let's see.

DesaCh01 commented 3 years ago

Hi @olivergaudreault ,

I tried by changing <<EOF to REQUIREMENT, but it gave the same error.

I'm using

Azurerm : 2.64.0 Terraform : 0.13.5

I also gave it a try using Azurerm: 2.68.0 Terraform : 1.0.0 But to no avail :(

olivergaudreault commented 3 years ago

@DesaCh01 Does the problem happen on the plan or apply?

olivergaudreault commented 3 years ago

@DesaCh01 I am also having issues on my side. For some reason I cannot apply due to some plugin issue. Trying to work through it.

DesaCh01 commented 3 years ago

@DesaCh01 Does the problem happen on the plan or apply?

The issue occurs during apply.

hashicorp / terraform-provider-azurerm