Because we are using spark version "2.4" for spark pool, it's not an issue with current setup. But in general, if the spark version >=3.1 (python 3.8) is selected, the application insights package load is failing with the following error:
The installation of the library to the pool does work properly:
import pkg_resources
for d in pkg_resources.working_set:
if "opencensus" in str(d):
print(d)
In "E2E - Parking Sensors Synapse" sample, "opencensus" library is used to write logs to Application Insights. There are two potential issues:
https://pypi.org/project/opencensus-ext-azure/ https://pypi.org/project/opencensus/ (doesn't have 1.1.7 version)
This file is used in deploy_synapse_artifacts.sh script.
The installation of the library to the pool does work properly:
But the importing of "AzureLogHandler" fails: