Error - ModuleNotFoundError: No module named 'dbt.main'

[vitorfraga]

Hello my friends. I am facing the following problem after installing the extension in jupyterlab.

"ModuleNotFoundError: No module named 'dbt.main'"

My code:

import json
import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.types import *
import ipywidgets as widgets
out = widgets.Output()
with out:
    spark = SparkSession.builder.getOrCreate()
%load_ext jupyterlab_sql_editor.ipython_magic.sparksql

Version of jupyterlab: I'm running jupyterlab through a docker image: jupyter/all-spark-notebook

Can you help me please? I performed tests with other versions of the extension, I always get the same problem.

[zorrofox]

When I start the jupyter lab also have the error:

[W 2023-07-01 03:43:24.140 ServerApp] Failed to load language server spec finder `sparksql-language-server`: 
    No module named 'dbt.main'

[eddspark]

I beleive this is related to this commit on dbt https://github.com/dbt-labs/dbt-core/commit/b1b830643ee6a8e15042d7b2f111f5896872f802

in the package dbt, dbt.main no longer exists after version 1.5.0, instead its located in dbt.cli.main

I was able to fix the error by changing the import command in jupyterlab_sql_editor/ipython_magic/sparksql/sparksql.py

from import dbt.main to

import dbt
import dbt.cli.main
dbt.main = dbt.cli.main

I tried to create a pull request so that the package will work with older and new versions of dbt. But I don't have permissions

The change is pretty simple, it's just

try:
    import dbt.main
except ModuleNotFoundError as E:
    import dbt
    import dbt.cli.main
    dbt.main = dbt.cli.main

Modifying the pyspark magic file should fix it. Perhaps someone with perms could fix it, also the pyproj needs bokeh to be added as a dependency.

[gmrqs]

Oi @vitorfraga !

Managed to make it work by forcing a older version of dbt

pip3 install dbt-core==1.01

[cccs-nik]

Sorry about that, we use a slightly older version of dbt internally and we weren't running into the issue. I removed dbt as a hard dependency and added error handling for those imports (https://github.com/CybercentreCanada/jupyterlab-sql-editor/pull/113/files#diff-6ac80edb3babc248f8c37b3080b138b267f521697dd1346dbc46aced45003ba0). Anyway those imports are used for a hack to stop double logging specifically for dbt. I'll probably revisit when we start using newer dbt.

[cccs-nik]

Fixed in #113