conf source location doesn't exist

[puneeter]

Description

My resource file:

resources:
    jobs:
        job_name:
            name: job_name
            tasks:
            -   task_key: job_name_raw
                depends_on: []
                python_wheel_task:
                    package_name: my_package
                    entry_point: databricks_run
                    parameters:
                    - --nodes
                    - nodes.raw
                    - --conf-source
                    - /dbfs/FileStore/my_package/conf
                    - --package-name
                    - my_package
                libraries:
                -   whl: ../dist/*.whl
                -   whl: "/Workspace/Users/my_username/.bundle/package_name/dev/files/extra_packages/package_1/package_1.whl"

Error:

MissingConfigException: Given configuration path either does not exist or is not a valid directory: /dbfs/FileStore/my_package/conf/base

Are using the right path for conf_source here?

[JenspederM]

Hi there!

The config path seems correct, given that you use the 'kedro databricks deploy' command.

Beyond deploying your project, this command will also upload your conf and data directories to dbfs under the project name.

[puneeter]

Is there a way to verify that? I can quickly check and debug. I am using the following command tho: kedro databricks deploy --debug && databricks bundle run "job_name"

[JenspederM]

You can always list the files in dbfs with the databricks cli, I.e. 'databricks fs ls/dbfs/FileStore/'

[puneeter]

Aah, it seems like I don't have write access to dbfs which is why I was facing this error. Thank you!