Issue Pyodbc library installation on Azure Databricks 13.3 DBR version

[MPavankumar1]

Issue: On Azure Databricks clusters with DBX 13.3 runtime, to download the msodbcsql18 package it requires some additional package ( “odbcinst”).

On Log comparison b/w 12.2 and 13.3 runtimes. In 12.2 log file to download the msodbcsql17 driver, it doesn’t require odbcinst package. Whereas to download msodbcsql18 driver the package file is showing a dependency on “odbcinst” package as shown above

In the 'Packages' section, the details indicate version 2.3.7(refer screenshot below), still the system is attempting to download version 2.3.9. Our understanding is since version 2.3.9 is not found, it defaults to downloading from [http://archive.ubuntu.com]

Please check this difference from 12.2 to 13.3 DBR versions and update packages.microsoft.com with 2.3.9 version for updated Ubuntu on 13.3 DBR

Steps to Reproduce Installing with following code

os.system('curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -') os.system('curl https://packages.microsoft.com/config/ubuntu/**22.04**/prod.list > /etc/apt/sources.list.d/mssql-release.list') os.system('sudo apt-get update') os.system('sudo ACCEPT_EULA=Y apt-get -q -y install msodbcsql18') os.system('sudo apt-get install python3-pip -y')

[sdherr]

I have sent an internal message asking the relevant team to take a look at this.

[v-chojas]

I was unable to reproduce this on a newly created Ubuntu 22.04 Azure VM. msodbcsql18 installs successfully with no errors (and is version 18.3.2.1.) The fact that you are seeing 18.2.2.1 suggests the apt metadata has not been updated, please apt-get update to fetch the latest metadata from the repo and try again.

[MPavankumar1]

I did more analysis on standard out logs. When installing pyodbc and msodbcsql17 on Databricks DBR 12.2, the logs doesn't show dependency of odbcinst, whereas installing msodbcsql18 and pyodbc on Databricks DBR 13.3 shows dependency of odbcinst package.

12.2 installation logs:

13.3 installation logs:

Attached log files.

odbcinst package is downloaded from archive.ubuntu.com which is missing in previous installations.

Our end customer has requirement to have dependency packages available in the [packages.microsoft.com] rather than download it from [http://archive.ubuntu.com] which is on port 80 where wouldn’t allow any external traffic.

Can we know why this behavioral change?

What is the workaround?

12_2DBR_msodbcsql17_pyodbc.txt

This 13.3 testing is on our lab environment where traffic is allowed on port 80 to reach archive.ubuntu.com

13_3DBR_msodbcsql18_pyodbc.txt

[v-chojas]

In both of those logs you are downloading packages from archive.ubuntu.com .

[MPavankumar1]

The attachments are for working lab environment and shared for difference in installing dependency package odbcinst in 13.3 which is not present in 12.2

Here is attachment for 13.3 where errors occurring only in 13.3 for odbcinst on port 80 13_3DBR_error.txt

[v-chojas]

You can run dpkg-deb -I on the msodbcsql17 and msodbcsql18 packages to inspect their dependencies. You will see they are exactly the same.

[sanyamgoyal26]

I have captured a screenshot down on the dependencies which was shared before. “Odbcinst” is the additional dependency we see for msodbcsql18 package.

In the 'Packages' section, the details indicate version 2.3.7 (screenshot 2, package file) , still the system is attempting to download version 2.3.9. Our understanding is since version 2.3.9 is not found, it defaults to downloading from [http://archive.ubuntu.com]

[v-chojas]

Look at the dependencies written in the package itself with the dpkg-deb -I command. You will see that they are identical, and thus what you are experiencing is not a difference there.

[sanyamgoyal26]

Can you upload the results here?

[v-chojas]

I am asking you to do it so that you can see the results for yourself.

[sanyamgoyal26]

Hi,

We tried to run dpkg-deb -I msodbcsql17 command and it returned attached error. Not sure if we are executing it correctly. PFA the error we encountered.

Also, we tried to install msodbcsql18 package both on DBR runtime 12.2 and 13.3. It got installed successfully for 12.2 runtime. However, it's not getting installed for 13.3 runtime. PFA the error details in the logs screenshots.

[sanyamgoyal26]

Just to mention, 2nd screenshot contains the error details while installing msodbcsql18 package on 13.3 DBx runtime. It is not able to install odbcinst package in 13.3 whereas in 12.2 runtime cluster, it is able to install odbcinst package.

[mike-f50]

Just to note, I'm seeing the same issue here.

[mbearup]

@mike-f50 please clarify which issue you're referring to.

• For the original error, @v-chojas may need to provide further clarity
• The more recent error (screenshot on 18Jan2024) appears to be a transient connection error with archive.ubuntu.com. We don't provide support for those mirrors in this forum.

[mbearup]

@mike-f50, I've synced with @v-chojas and determined what's happening here. The odbc packages hosted in the packages.microsoft.com (PMC) repo are version 2.3.7 (vs 2.3.9 on Ubuntu). Apt will naturally find and install the newest packages unless instructed otherwise. You can instruct apt to use the packages from PMC as follows: sudo apt-get install libodbc1=2.3.7 odbcinst1debian2=2.3.7 odbcinst=2.3.7 unixodbc=2.3.7

Note that there are some essential dependencies that can only be retrieved from the distro (Ubuntu). Presumably these are already present in your environment, so they shouldn't pose a problem during install libc6 (>= 2.19), libstdc++6 (>= 4.8.2), libkrb5-3, openssl, debconf (>= 0.5)

[mike-f50]

Thanks @mbearup. Adding the following to my notebook seems to have got me up and running on 13.3 😀

%sh
apt-get -y install libodbc1=2.3.7 odbcinst1debian2=2.3.7 odbcinst=2.3.7 unixodbc=2.3.7  --allow-downgrades