spark_version not capturing from OS environ in Azure databricks cluster

[sujitgh]

Describe the bug spark_version not capturing from OS environ in Azure databricks cluster. To Reproduce Steps to reproduce the behavior:

1. Go to 'Azure databricks workspace'
2. Click on 'notebook'
3. Scroll down to 'try to run the pydeequ as a job'
4. See error

Expected behavior We can use spark session to retrieve spark version.

Screenshots

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

[mykola-yesypchuk-inflection]

AWS Glue is also failed with the same error.

[chenliu0831]

There's a workaround and we are tracking the improvement here https://github.com/awslabs/python-deequ/issues/140.