search

duckdb / dbt-duckdb

dbt (http://getdbt.com) adapter for DuckDB (http://duckdb.org)
Apache License 2.0
880 stars 78 forks source link

use_credential_provider does not work when using a k8s cluster role binding #321

Open JRocki opened 8 months ago

JRocki commented 8 months ago

Hello! I'm running dbt-duckdb on a k8s cluster via EKS that has a cluster role binding rather than using an IAM role. The project uses parquet files on S3 as the data source. It seems that using use_credential_provider does not work for role bindings as it leads to an HTTP 400 error. I can work around this, but I'm wondering if there is an alternative setting the botocore session instead of using aws_creds = session.get_credentials().get_frozen_credentials() as it is here.

example code:

temp_project:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: "/temp.duckdb"
      extensions:
        - httpfs
        - parquet
      use_credential_provider: aws
jwills commented 8 months ago

Ah, there isn't one right now, but I would be happy to help add one if I can-- is there somewhere I can read up on what is needed to make this work? Or can you just tell me?

JRocki commented 8 months ago

I am also currently looking into this and trying to track down how boto3 handles it. If I can find something, I will send it your way or I will open a PR. This is a great package and I'd be happy to help contribute if I can.

nclaeys commented 7 months ago

@JRocki what do you mean with using cluster role binding for k8s? The use credential provider uses boto3 behind the scenes. So it uses the standard aws credential provider chain. If it does not work, this means that your configuration is probably wrong or unsupported by boto3. Can you give more details on your configuration/settings?