galeone
commented
1 year ago Update: I understood that for tabular data the schema is the csv (or bigquery) structure itself. And that's fine. So there's no need to execute the ImportDataRequest.
However, the association between the storage and the dataset should be performed.
Reading the python source code I found this line:
if gcs_source:
dataset_metadata = {"inputConfig": {"gcsSource": {"uri": gcs_source}}}
elif bq_source:
dataset_metadata = {"inputConfig": {"bigquerySource": {"uri": bq_source}}}In practice, it's the equivalent of setting the *vaipb.Dataset.MetaData field to the equivalent Go struct.
I tried in several ways, with 0 luck.
First attempt: the struct value:
metadata, err := structpb.NewStruct(map[string]interface{}{
"inputConfig": map[string]interface{}{
"gcsSource": map[string]interface{}{
"uri": gcsSource,
},
},
}
req := &vaipb.CreateDatasetRequest{
// Required. The resource name of the Location to create the Dataset in.
// Format: `projects/{project}/locations/{location}`
Parent: fmt.Sprintf("projects/%s/locations/%s", _vaiProjectID, _vaiLocation),
Dataset: &vaipb.Dataset{
DisplayName: userDataset,
Description: fmt.Sprintf("User %d data", user.ID),
// No metadata schema because it's a tabular dataset, and "tabular dataset does not support data import"
// ref: https://github.com/googleapis/python-aiplatform/blob/6f3b34b39824717e7a995ca1f279230b41491f15/google/cloud/aiplatform/datasets/_datasources.py#LL223C30-L223C75
MetadataSchemaUri: "gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml",
// But we need to pass the metadata as a structpb.Value
// https://github.com/googleapis/python-aiplatform/blob/6f3b34b39824717e7a995ca1f279230b41491f15/google/cloud/aiplatform/datasets/_datasources.py#L48
Metadata: structpb.NewStructValue(metadata),
},
}But the server returns:
rpc error: code = InvalidArgument desc = Invalid structured dataset value: [struct_value {
fields {
key: "inputConfig"
value {
struct_value {
fields {
key: "gcsSource"
value {
struct_value {
fields {
key: "uri"
value {
string_value: "gs://train-and-deploy-experiment-user-data/1-2023-05-20_2023-01-13.csv"
}
}
}
}
}
}
}
}
}Second attempt: passing a string containing the JSON:
// same request with the Metadata field set in this way
Metadata: structpb.NewStrintValue(fmt.Sprintf(`{"inputConfig": {"gcsSource": {"uri": "%s"}}}`, gcsSource)),In this case, the creation works, but in the Vertex AI page of the dataset, there's no association between the file on the bucket and the dataset.
I can do this process manually from the webpage and the association is possible. However, I need to do this from code in Go.
So, yes, this is definitely a google-cloud-go issue now
noahdietz
dizcology
codyoss
I have created the empty dataset (using the Go SDK, but maybe the language doesn't matter), and the next step is to associate the data source with the dataset. When creating the empty dataset, I specified the "metadata schema URI" as
gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml. The empty dataset has been created with no problem, and it appeared on the vertex AI dashboard.The data is already uploaded to a bucket on Google Storage. So far so good. However the "import data request" requires the attribute "Import Schema URI" to have a valid value.
Unfortunately, all the available schemas are not for tabular data AFAIK. When listing the content of the bucket
gs://google-cloud-aiplatform/schema/dataset/ioformatin fact, there's nothing about tabular data.Am I doing something wrong or is this schema for the IO of tabular data missing?
e.g.
Expected behavior
I should be able to import the tabular data into the dataset.
Actual behavior
If I leave the
ImportSchemaUriattribute empty, I receive this messageIf I put a wrong schema (like one of the other ioformat that's not related to tabular data, I receive the message
Anyway, I do really thing the problem is in the missing schema for the tabular data. Is this correct? Is there a workaround for this?
Thanks!