databricks / containers

Sample base images for Databricks Container Services
Apache License 2.0
168 stars 118 forks source link

Install pyccolo in gpu cuda-11.8 #204

Closed es94129 closed 2 months ago

es94129 commented 2 months ago

The cuda-11.8 image hasn't been updated for a while, and it has Python REPL issue on a cluster with DBR 15.x; meanwhile, DBR 14.3 works. The cluster driver logs showed that the python package pyccolo was missing.

Changes:

  1. Move requirements for cuda-11.8/venv to a dedicated requirements.txt
  2. Add pyccolo to the file

Tested that Python REPL now works:

image