[Regression] TypeError: expected string or bytes-like object, got 'PosixPath'

digitalghost-dev commented 4 months ago

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Installing dbt-core on a new machine, unable to run dbt init.

TypeError: expected string or bytes-like object, got 'PosixPath'

Expected Behavior

Hoping to initialize a new dbt profile.

Steps To Reproduce

Install dbt-bigquery
Run dbt init
Error

Relevant log output

16:29:17  Encountered an error:
[ConfigFolderDirectory]: Unable to parse dict {'dir': PosixPath('/Users/cs/.dbt')}
16:29:17  Traceback (most recent call last):
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/google/protobuf/json_format.py", line 608, in _ConvertFieldValuePair
    _ConvertScalarFieldValue(value, field,
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/google/protobuf/json_format.py", line 779, in _ConvertScalarFieldValue
    if _UNPAIRED_SURROGATE_PATTERN.search(value):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'PosixPath'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/dbt/events/base_types.py", line 72, in __init__
    self.pb_msg = ParseDict(kwargs, msg_cls())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/google/protobuf/json_format.py", line 446, in ParseDict
    parser.ConvertMessage(js_dict, message, '')
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/google/protobuf/json_format.py", line 487, in ConvertMessage
    self._ConvertFieldValuePair(value, message, path)
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/google/protobuf/json_format.py", line 622, in _ConvertFieldValuePair
    raise ParseError(
google.protobuf.json_format.ParseError: Failed to parse dir field: expected string or bytes-like object, got 'PosixPath'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/dbt/cli/requires.py", line 91, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/dbt/cli/requires.py", line 76, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/dbt/cli/main.py", line 515, in init
    results = task.run()
              ^^^^^^^^^^
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/dbt/task/init.py", line 298, in run
    self.create_profiles_dir(profiles_dir)
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/dbt/task/init.py", line 70, in create_profiles_dir
    fire_event(ConfigFolderDirectory(dir=profiles_dir))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cs/premier-league/venv/lib/python3.12/site-packages/dbt/events/base_types.py", line 81, in __init__
    raise Exception(error_msg)
Exception: [ConfigFolderDirectory]: Unable to parse dict {'dir': PosixPath('/Users/cs/.dbt')}

Environment

- OS: Sonoma 14.3
- Python: 12
Core:
  - installed: 1.7.10
  - latest:    1.7.10 - Up to date!

Plugins:
  - bigquery: 1.7.6 - Up to date!

Which database adapter are you using with dbt?

bigquery

Additional Context

No response

digitalghost-dev commented 4 months ago

I edited core/dbt/task/init.py on my machine and it works when editing the create_profiles_dir function:

    def create_profiles_dir(self, profiles_dir: str) -> bool:
        """Create the user's profiles directory if it doesn't already exist."""
        profiles_path = Path(profiles_dir)
        profiles_dir_str = str(profiles_path)  # Convert PosixPath object to string
        if not profiles_path.exists():
            fire_event(ConfigFolderDirectory(dir=profiles_dir_str))
            dbt.clients.system.make_directory(profiles_dir_str)
            return True
        return False

dbeatty10 commented 3 months ago

Thanks for reaching out about this @digitalghost-dev ! And thanks for testing out that code change as well 🤩

I tried to replicate this error, but running dbt init worked for me with Python 3.10.10 and dbt-core 1.7.10.

A couple follow-up questions: Which version of protobuf do you have installed?

If you used pip to install your dependencies, you can discover the version by running pip list | grep protobuf
dbt-core==1.7.10 requires protobuf>=4.0.0,<5.0.0 -- 5.26.0 will not work (see https://github.com/dbt-labs/dbt-core/issues/9759)

If protobuf is not the issue, could you try installing and running in an environment other than Python 3.12 (like 3.11 or 3.10) and see if it still has the same problem?

digitalghost-dev commented 3 months ago

Hey @dbeatty10 - protobuf is showing version 4.25.3

I tried it with Python v3.11.6 and looks like it worked.

dbeatty10 commented 3 months ago

Thanks for testing that out @digitalghost-dev 💪

The root cause appears to be that dir is expected to be a string here and here.

A minimal fix might be updating this: https://github.com/dbt-labs/dbt-core/blob/65b366bca904ef741bedae255dfdbc3837f4d2e9/core/dbt/task/init.py#L69

to this:

 fire_event(ConfigFolderDirectory(dir=str(profiles_dir)))

But we'll leave it to one of our software engineers to make the decision how they want to address this.

digitalghost-dev commented 3 months ago

Alright, @dbeatty10, thanks for looking at this with me. The small patch I made got me up and running for the time being.

CalRobert commented 3 months ago

Thanks for the conversation here, I hit the same issue with dbt as part of dagster-dbt. For me I was able to simply make the .dbt directory manually.

dbt-labs / dbt-core