[FEATURE REQUEST]: Documentation to use .net for spark on Azure Synapse

[guruvonline]

Is there any documentation for using on Azure synapse? currently i have a job running in Azure Databricks, now i want to move/migrate this job to azure synapse. job has lot of dependent assemblies, which today are added to cluster as part of init script.

Describe the solution you'd like A step by step guide on how to create jobs and upload application packages and run end to end.

[imback82]

Does https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/spark-dotnet help? If it's missing some details you need, please let us know. Thanks.

[MikeRys]

.NET for Spark is already available in Synapse out of the box. Documentation is available in the Azure Synapse documentation. For example: Use .NET for Apache Spark - Azure Synapse Analytics | Microsoft Docshttps://docs.microsoft.com/en-us/azure/synapse-analytics/spark/spark-dotnet

I hope this helps Michael

From: guruvonline @.> Sent: Wednesday, August 25, 2021 4:17 PM To: dotnet/spark @.> Cc: Subscribed @.***> Subject: [dotnet/spark] [FEATURE REQUEST]: Documentation to use .net for spark on Azure Synapse (#963)

Is there any documentation for using on Azure synapse? currently i have a job running in Azure Databricks, now i want to move/migrate this job to azure synapse.

Describe the solution you'd like A step by step guide on how to create jobs and upload application packages and run end to end.

- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fspark%2Fissues%2F963&data=04%7C01%7Cmrys%40microsoft.com%7C395b7bb5c06f430d244308d9681e7985%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637655302370583905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HgIuEMbguJcSPskH%2BMrZ4m8%2FLWKuKUSdMDjpctJpFGY%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZXGJDASTKEYT7GBWVVQPTT6V2XVANCNFSM5C2EF4EQ&data=04%7C01%7Cmrys%40microsoft.com%7C395b7bb5c06f430d244308d9681e7985%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637655302370593901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FCs%2FnKYDrRipVz4InCBrD3pkXMihxCZolfoaIqC0%2Fok%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cmrys%40microsoft.com%7C395b7bb5c06f430d244308d9681e7985%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637655302370603896%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fJWjw7kj4Wb0g5kozv41MAeswNG9h26QX86YaNjFJ8U%3D&reserved=0 or Androidhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26utm_campaign%3Dnotification-email&data=04%7C01%7Cmrys%40microsoft.com%7C395b7bb5c06f430d244308d9681e7985%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637655302370603896%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tJ93RvUmCpZwWWP7RYm0%2F0Gn98MZRfq%2FG%2BQ9O6xlxos%3D&reserved=0.

[guruvonline]

Thanks for reply. I am following the same links, but document is not very clear on how to part when moving from azure databricks to synapse, who to handle different scenarios. for example.

• my scenario have structured streaming C# app, with c# udf and in Databricks we used to have db_init scripts to copy dependent dlls to cluster. in synapse do i have to upload these dlls as "Reference files " in c# job?
• App pacakge needs some secret (to connect to azure blob), in ADB we use to read from dbscript and pass as argument to DotnetRunner.main. how to do same in Synapse

below is the steps that i am following

• used the same application package(after publish) which i upload in ADB to Synpase.
• app uses "azure Event hub for spark " package, uploaded the package jar file as workspace package and added to spark pool.
• Created a new C# job in synapse.
• uploaded all the binaries from application in "Reference Files" section of job.
• published and Submit job.
• it fails with attached message SynapseError.txt

[guruvonline]

can we add step in doc to use .NET for spark higher than 1. I was using less and it was not working.