Closed allengour closed 1 year ago
@TheMichaelHu :D
Aparently, the feature doesn't allow a specific table as it creates the table after the Batches Prediction. Using projectID.datasetID works around the error.
Hi @allengour, thanks for the detailed issue. Our recommendation is to follow the guidance of the docstring:
When only the project is specified, the Dataset and Table is created.
When the full table reference is specified, the Dataset must exist and table must not exist.
and the error message: InvalidArgument: 400 Table name is not allowed in Bigquery output uri.
I agree that bq://projectId.bqDatasetId.bqTableId
shouldn't be an accepted format and that it can be confusing. The docstring has been edited in https://github.com/googleapis/python-aiplatform/commit/794cedd0e9e34544d6c8d5bf6892ef2c28eaaf5d.
FYI Custom Models can accept destination tables during BP, however models trained using AutoML products cannot, the most granular they accept is destination dataset. I have filed a feature request to remove this restriction, but that is how it is right now.
It appears the workaround has been ruined, now Vertex forces you to select a table, so AutoML batch predictions are essentially impossible to export into BigQuery atm
Hi @gfcornejo thanks for pointing this out. I'm also seeing this behavior in the UI, which is definitely not intentional. Are you seeing an issue exporting to BQ via the Vertex SDK as well?
I haven’t tried the SDK.
After the 10th refresh, the issue went away on its own!
On Fri, Nov 10, 2023 at 12:50 PM Michael Hu @.***> wrote:
Hi @gfcornejo https://github.com/gfcornejo thanks for pointing this out. I'm also seeing this behavior in the UI, which is definitely not intentional. Are you seeing an issue exporting to BQ via the Vertex SDK as well?
— Reply to this email directly, view it on GitHub https://github.com/googleapis/python-aiplatform/issues/1615#issuecomment-1806267090, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADVFDSUPVWCOAUU6L3ZHDHLYDZZQRAVCNFSM57U3QSSKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBQGYZDMNZQHEYA . You are receiving this because you were mentioned.Message ID: @.***>
Per the docs at https://googleapis.dev/python/aiplatform/latest/aiplatform/services.html?#google.cloud.aiplatform.Model.batch_predict and https://googleapis.dev/python/aiplatform/latest/aiplatform.html?#google.cloud.aiplatform.BatchPredictionJob.create
A prefix uri like
bq://projectId.bqDatasetId.bqTableId
is an acceptable input tobigquery_destination_prefix
. However, when that is passed in, I receive:The same error can be observed in the UI, even though UI suggests
tableId
is allowed and passes validation.Note that the documentation does not mention allowing a user-specified output table in
bigquery_destination_prefix
here:Code example
or
Steps to reproduce
Model type: AutoMLForecasting
On UI:
Stack trace
Environment details
google-cloud-aiplatform
version: 1.16.