search

databricks / containers

Sample base images for Databricks Container Services
Apache License 2.0
167 stars 118 forks source link

missing dependencies in python:14.3-lts #188

Open ericfeunekes opened 7 months ago

ericfeunekes commented 7 months ago

I used the databricksruntime/python:14.3-LTS image but when loading it in Databricks it's missing some dependencies for IPywidgets. As well as the dependencies for Black formatting to work.

Regarding the former. I tried using tqdm but it fails, saying I need to update jupyter and ipywidgets. So I pip install those. Then when I try running tqdm it just says that it fails to load.

I haven't tried installing the dependencies for black formatting yet.

This is mostly to say I understand you want a minimal version for the python image (from comments on #150), but it would be good if it at least allowed the image to work with Databricks features. I'm not sure what libraries I'd have to install in my custom image to make sure all Databricks features work.

xinzhao-db commented 2 weeks ago

@joveyuan-db Can you take a look?

Btw, starting from 15.4, we install the same set of python dependencies as DBR in DCS

joveyuan-db commented 1 week ago

Hi @ericfeunekes, as @xinzhao-db mentioned, DCS images corresponding to DBR 14.3 and older had a more minimal set of Python dependencies and did not necessarily support all Databricks features out of the box. You should be able to install the missing ones manually to get the expected behavior though. Could you share an example of what error you're seeing when trying to run tqdm?