Azure-Samples / modern-data-warehouse-dataops

DataOps for Microsoft Data Platform technologies. https://aka.ms/dataops-repo
MIT License
588 stars 459 forks source link

[Bug] E2E Parking Sensors Synapse - opencensus package issue #587

Closed promisinganuj closed 1 year ago

promisinganuj commented 1 year ago

In "E2E - Parking Sensors Synapse" sample, "opencensus" library is used to write logs to Application Insights. There are two potential issues:

  1. The requirement.txt file for Synapse mentions "opencensus==1.1.7" which should be "opencensus-ext-azure==1.1.7" instead:

https://pypi.org/project/opencensus-ext-azure/ https://pypi.org/project/opencensus/ (doesn't have 1.1.7 version)

This file is used in deploy_synapse_artifacts.sh script.

  1. Because we are using spark version "2.4" for spark pool, it's not an issue with current setup. But in general, if the spark version >=3.1 (python 3.8) is selected, the application insights package load is failing with the following error:

The installation of the library to the pool does work properly:

import pkg_resources

for d in pkg_resources.working_set:
    if "opencensus" in str(d):
        print(d)
opencensus 0.11.0
opencensus-ext-azure 1.1.7
opencensus-context 0.1.3

But the importing of "AzureLogHandler" fails:

from opencensus.ext.azure.log_exporter import AzureLogHandler
from opencensus.ext.azure.log_exporter import AzureEventHandler
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-8097069> in <module>
----> 1 from opencensus.ext.azure.log_exporter import AzureLogHandler
      2 from opencensus.ext.azure.log_exporter import AzureEventHandler

~/cluster-env/clonedenv/lib/python3.8/site-packages/opencensus/ext/azure/log_exporter/__init__.py in <module>
     34     TransportStatusCode,
...
~/cluster-env/clonedenv/lib/python3.8/site-packages/google/protobuf/internal/api_implementation.py in <module>
    102   try:
    103     # pylint: disable=g-import-not-at-top
--> 104     from google.protobuf.pyext import _message
    105     _c_module = _message
    106     del _message

TypeError: bases must be types