search

kristeligt-dagblad / dbt_ml

Package for dbt that allows users to train, audit and use BigQuery ML models.
Apache License 2.0
62 stars 26 forks source link

Training [type] models is not available for on-demand usage. To train, please set up a reservation #21

Closed Startouf closed 3 years ago

Startouf commented 3 years ago

I have the following error when trying to run a model

Database Error in model ml_gatling_train (models/ml/models/ml_gatling_train.sql) Training Matrix Factorization models is not available for on-demand usage. To train, please set up a reservation (flex or regular) based on instructions in BigQuery public docs. compiled SQL at target/run/myproject/models/ml/models/ml_gatling_train.sql

Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

It seems like I need additional config on my BQ adapter ? Do you have some info regarding what must be done ? (that could then be added to the readme ?)

Startouf commented 3 years ago

Working with reservation slots is actually quite simple to activate (but can become quite expensive if you're not careful to "clean up" after you)

Do you think we should mention this in the readme ? (and maybe give a link to https://cloud.google.com/bigquery/docs/reservations-tasks#working_with_reservations)

Note that this subject was discussed within the dbt bigquery channel here https://getdbt.slack.com/archives/C99SNSRTK/p1593451232464300 and there is apparently a bigquery autoscaling alpha available

rbjerrum commented 3 years ago

Thank you for bringing this up, @Startouf. I think handling reservations is out of the scope of this specific package, however, I'm open to the idea of mentioning it in the readme. The database error is quite detailed so I don't think we need to handle it on execution.

Feel free to open a PR that mentions reservations in the readme - otherwise I'll try to get to it next week.