BigQuery adapter Unrecognized name: _dbt_max_partition on model.ga4.fct_ga4__pages

[scottsgoingon]

console_output.txt

Trying to deploy this package and encounter this issue as per issue name and log file. All the other models successfully ran except this. model.ga4.fct_ga4__pages

I ran dbt seed, followed by dbt run

[dgitis]

We recently deprecated dynamic partitioning that relies on the _dbt_max_partition macro that's failing in that model.

Install the current latest version, rather than the last major release.

packages:
  - git: "https://github.com/Velir/dbt-ga4.git"
    revision: 78973a2c82da216aa7f7766a8f8ab18acae46330

You can switch back to the normal install when we release the next version.

[scottsgoingon]

I use DBT Cloud, not CLI.

I updated the packages.yml with the following.

packages:

• package: Velir/ga4 version: [">=3.2.1", "<=4.0.0"]
• package: dbt-labs/dbt_utils version: 1.0.0

The variables are also set in the projects file vars: ga4: project: "datagemsadmin" dataset: "analytics_257102154" start_date: "20220531" frequency: "daily"

I then run the following dbt deps -- success dbt seed -- success dbt run success and populated all of the tables from the package on the dev dataset that it created at dbt_selliott including fct_ga4__pages

I then attempt to run this in Production and encounter an error. Database Error Invalid project ID 'datagemsadmin.datagemsadmin'. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.

Which makes no sense.

I go back to console and rerun dbt run and it's started erroring out on fct_ga4__pages with the previous error Unrecognized name: _dbt_max_partition

What am I missing here?

[adamribaudo-velir]

@scottsgoingon , Damon was suggesting you update your packages.yml file to what he posted above so that you pull the latest code from GitHub rather than the latest release which is out of date.

Also, if you want to disable a single model (like fct_ga4__pages) you can easily do so by updating your dbt_project.yml file and disabling any model with enabled: false. See these docs:

https://docs.getdbt.com/reference/resource-configs/enabled https://docs.getdbt.com/reference/model-configs

[adamribaudo-velir]

@scottsgoingon we released v5.0.0 recently. want to give it a spin and see if your issues are resolved? As a reminder, you can disable the tables in the marts folder if they are giving you issues.

[scottsgoingon]

Hi Adam,

When I rerun DBT deps for version 5.0.0 from DBT cloud it doesn't complete.

I get the error below and no other details.

Exception 'NoneType' object cannot be interpreted as an integer

On Mon, 7 Aug 2023 at 03:01, Adam Ribaudo @.***> wrote:

@scottsgoingon https://github.com/scottsgoingon we released v5.0.0 recently. want to give it a spin and see if your issues are resolved? As a reminder, you can disable the tables in the marts folder if they are giving you issues.

— Reply to this email directly, view it on GitHub https://github.com/Velir/dbt-ga4/issues/253#issuecomment-1666888289, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE2VSJHTORCMXKIWHFIKFWTXT6WVDANCNFSM6AAAAAA22X4SVU . You are receiving this because you were mentioned.Message ID: @.***>

[dgitis]

That's normal and very annoying. The dbt Cloud parser is complaining that static_incremental_days is missing. The package updates. Set the variable and you'll be fine.

[scottsgoingon]

Hi Damon,

doesn't run in production either. I've attached the log

On Mon, 7 Aug 2023 at 10:35, Damon Gudaitis @.***> wrote:

That's normal and very annoying. The dbt Cloud parser is complaining that static_incremental_days is missing. The package updates. Set the variable and you'll be fine.

— Reply to this email directly, view it on GitHub https://github.com/Velir/dbt-ga4/issues/253#issuecomment-1666992758, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE2VSJHB5P6R4FEXXX73AGTXUAL2LANCNFSM6AAAAAA22X4SVU . You are receiving this because you were mentioned.Message ID: @.***>

[dgitis]

Maybe I wasn't very clear with my comment. I was commenting from my phone which is annoying.

Your attachment didn't make it to GitHub. The GitHub email reply functionality is limited. You need to include everything as text.

Are you sure that static_incremental_days is set? Because that error is exactly the one I got when upgrading. I just had to ignore it until I'd fixed it.

Are there other integer variables that could be not set?

[scottsgoingon]

My Bad- I hadn't added the variable. Added now. I achieve success in Dev. But when deployed to Prod get error about ProjectID which was one of the errors I had last week.
debug.log

From my dbt_project.yml file below

vars: ga4: project: "datagemsadmin" dataset: "analytics_257102154" start_date: "20220531" frequency: "daily" static_incremental_days: 3

New to BigQuery, so I'm not quite understanding the error-

[dgitis]

I've got a feeling this might be fixed with a --full-refresh.

You might want to check the BigQuery connections. I usually don't have any listed but the log seems to suggest that you are using one on line 30 that also has some suspiciously long names that could fall afoul of the Project IDs must contain 6-63 lowercase letters, digits, or dashes error.

If you have a connection there, I'd think about deleting it.

[scottsgoingon]

Hi Damon,

do you mean add --full-refresh into the commands so it would be dbt run --full-refresh If yes, Tried that didn't work.

Also tried adding to dbt_projects yml as per documentation https://docs.getdbt.com/reference/resource-configs/full_refresh

models: +full_refresh: true Didn't work. And the YML file didn't seem to like as it showed a warning in the editor window.

The Project id for Bigquery is quite short datagemsadmin . I don't know if I have set up the service account correctly. I don't understand BigQuery well enough to know what I need to fix. What I don't understand is why it runs in Dev with no issue and fails in Production

The error in the log that sticks out for me is Invalid project ID 'datagemsadmin.datagemsadmin' It's like it's trying to pass the project id twice into a variable- But I only speculate as I wouldn't know where to fix this or at what point it falls over.

[adamribaudo-velir]

It would be helpful to understand which specific query contains that invalid project ID 'datagemsadmin.datagemsadmin'.

Can you go into the project history and paste the query here that fails?

[scottsgoingon]

Hi Damon,

Here's the query from the project history. / {"app": "dbt", "dbt_version": "1.5.4", "profile_name": "user", "target_name": "default", "connection_name": "create_datagemsadmin_datagemsadmin.analytics_257102154"} / create schema if not exists datagemsadmin.datagemsadmin.analytics_257102154

[dgitis]

Do you have a dataset with the same name as the project? Maybe try changing the name of the target dataset.

Otherwise, we're going to need a lot more information. I'm pretty confident that this is not a package issue.

Are you on dbt Cloud? If so, you might be able to get an answer from their support much faster as they'll be able to see what you've set up.

Also, none of the Yaml that you've pasted in here has indent spacing. Are you spacing your Yaml files? Yaml is very strict with spacing and I know that pasting Yaml into GitHub preserves the spacing so either you're removing the spacing when pasting here or your Yaml files aren't spaced properly.

[scottsgoingon]

vars: ga4: project: "datagemsadmin" dataset: "analytics_257102154" start_date: "20220531" # Earliest date to load frequency: "daily" # daily|streaming|daily+streaming. See 'Export Frequency' below. static_incremental_days: 3

[scottsgoingon]

Hi Damon,

Ticket to DBT helped. It was the deployment set up. Removed the dataset name from that and I now have it running in DBT cloud. It's always the configuration :(

Going through all of the models via Looker Studio.

thanks for your help :)

[scottsgoingon]

Thanks everyone for your help