request body for aiplatformClient.predict(request) is not clearly defined; gRPC error is not clear

[lucksp]

Environment details

• which product (packages/*) :
• OS: Mac
• Node.js version: 18.16.0
• npm version: 9.5.1
• google-cloud-node version: "@google-cloud/aiplatform": "^2.14.0"

Steps to reproduce

// Instantiates a client
const aiplatformClient = new PredictionServiceClient({
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
});

async function callPredict() {
  // Construct request
  const request = {
    endpoint: `projects/${projectId}/locations/us-central1/endpoints/${endpointId}`,
    instances: [sampleInstance],
  };
  // Run request
  const response = await aiplatformClient.predict(request);
  return response;

}

Error from predict requesst:

node:internal/process/promises:288 triggerUncaughtException(err, true / fromPromise /); ^ Error: 3 INVALID_ARGUMENT: at callErrorFromStatus (/Users/lucksp/dev/myflyid/api/run/node_modules/@grpc/grpc-js/build/src/call.js:31:19) at Object.onReceiveStatus (/Users/lucksp/dev/myflyid/api/run/node_modules/@grpc/grpc-js/build/src/client.js:192:76) at Object.onReceiveStatus (/Users/lucksp/dev/myflyid/api/run/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141) at Object.onReceiveStatus (/Users/lucksp/dev/myflyid/api/run/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181) at /Users/lucksp/dev/myflyid/api/run/node_modules/@grpc/grpc-js/build/src/resolving-call.js:94:78 at process.processTicksAndRejections (node:internal/process/task_queues:77:11) for call at at ServiceClientImpl.makeUnaryRequest (/Users/lucksp/dev/myflyid/api/run/node_modules/@grpc/grpc-js/build/src/client.js:160:34) at ServiceClientImpl. (/Users/lucksp/dev/myflyid/api/run/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19) at /Users/lucksp/dev/myflyid/api/run/node_modules/@google-cloud/aiplatform/build/src/v1/prediction_service_client.js:218:29 at /Users/lucksp/dev/myflyid/api/run/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16 at OngoingCallPromise.call (/Users/lucksp/dev/myflyid/api/run/node_modules/google-gax/build/src/call.js:67:27) at NormalApiCaller.call (/Users/lucksp/dev/myflyid/api/run/node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19) at /Users/lucksp/dev/myflyid/api/run/node_modules/google-gax/build/src/createApiCall.js:84:30 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { code: 3, details: '', metadata: Metadata { internalRepr: Map(2) { 'endpoint-load-metrics-bin' => [ Buffer(27) [Uint8Array] [ 9, 160, 73, 4, 20, 177, 31, 204, 63, 49, 225, 28, 120, 240, 178, 244, 116, 64, 57, 91, 154, 116, 6, 2, 176, 223, 63 ] ], 'grpc-server-stats-bin' => [ Buffer(10) [Uint8Array] [ 0, 0, 245, 65, 218, 1, 0, 0, 0, 0 ] ] }, options: {} } }

The predict method is typed as a IPredictRequest:

interface IPredictRequest {

                    /** PredictRequest endpoint */
                    endpoint?: (string|null);

                    /** PredictRequest instances */
                    instances?: (google.protobuf.IValue[]|null);

                    /** PredictRequest parameters */
                    parameters?: (google.protobuf.IValue|null);
                }

based on the error message 3 INVALID_ARGUMENT: it seems like one of my arguments is incorrect? But which one? I am guessing the endpoint being passed in? But what format should it be? the typing is just string with no other details. This table of codes doesn't help clarify either

[iamcrisb]

Hi @danielbankhead did you figure out the correct format for this? facing the same issue and this library typing and docs are lacking a lot of info.

[danielbankhead]

Looking at the autogenerated sample, the request looks correct:

https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-aiplatform/samples/generated/v1/prediction_service.predict.js

I don’t work on this product, but someone should chime in soon.

[iamcrisb]

meanwhile...

    const [response] = await this.predictionClient.predict(request);
    const prediction = helpers.fromValue(response.predictions[0] as any);
    const metadata = helpers.fromValue(response.metadata as any);

[Seth-McKilla]

I'm running into the same issue, I was able to successfully retrieve a prediction by using the fetch API but am unable to translate this to the SDK package (so that I don't have to add the complexity of refreshing access tokens). For reference, here's the fetch API version:

import { convertImageToBase64 } from "@/lib/utils";

const {
  GCP_ACCESS_TOKEN,
  GCP_LOCATION,
  GCP_PROJECT_ID,
  GCP_PREDICTION_ENDPOINT_ID,
} = process.env;

export async function POST(request: Request) {
  try {
    const { imageUrl } = await request.json();
    const base64Image = await convertImageToBase64(imageUrl);

    const apiRoute = `https://${GCP_LOCATION}-aiplatform.googleapis.com/v1/projects/${GCP_PROJECT_ID}/locations/${GCP_LOCATION}/endpoints/${GCP_PREDICTION_ENDPOINT_ID}:predict`;

    const payload = {
      instances: [
        {
          content: base64Image,
        },
      ],
      parameters: {
        confidenceThreshold: 0.5,
        maxPredictions: 5,
      },
    };

    const response = await fetch(apiRoute, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${GCP_ACCESS_TOKEN}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(payload),
    });
    const data = await response.json();

    return new Response(
      JSON.stringify({
        predition: data.predictions && data.predictions[0]?.displayNames[0],
      }),
      {
        headers: { "Content-Type": "application/json" },
      }
    );
  } catch (error: any) {
    return new Response(
      JSON.stringify({
        error: error?.message || "Server error",
      }),
      {
        headers: { "Content-Type": "application/json" },
        status: 500,
      }
    );
  }
}

And here's the attempt at the SDK:

import aiPlatform from "@google-cloud/aiplatform";

import { convertImageToBase64 } from "@/lib/utils";

const {
  GCP_PROJECT_ID,
  GCP_CLIENT_EMAIL,
  GCP_PRIVATE_KEY,
  GCP_LOCATION,
  GCP_PREDICTION_ENDPOINT_ID,
} = process.env;

const predictionServiceClient = new aiPlatform.v1.PredictionServiceClient({
  projectId: GCP_PROJECT_ID!,
  credentials: {
    client_email: GCP_CLIENT_EMAIL!,
    private_key: GCP_PRIVATE_KEY!,
  },
  apiEndpoint: `${GCP_LOCATION}-aiplatform.googleapis.com`,
});

export async function predictImage(imageUrl: string) {
  const endpoint = `projects/${GCP_PROJECT_ID}/locations/${GCP_LOCATION}/endpoints/${GCP_PREDICTION_ENDPOINT_ID}`;

  const base64Image = await convertImageToBase64(imageUrl);

  const payload = {
    endpoint,
    instances: [
      {
        content: base64Image,
      },
    ],
    parameters: {
      confidenceThreshold: 0.5,
      maxPredictions: 5,
    },
  };

  const response = await predictionServiceClient.predict(payload);
  return response.predictions && response.predictions[0]?.displayNames[0];
}

I'm getting errors that await has no effect on this type of expression (when invoking predictionServiceClient.predict()) and the payload is throwing the following Typescript error:

Argument of type '{ endpoint: string; instances: { content: string; }[]; parameters: { confidenceThreshold: number; maxPredictions: number; }; }' is not assignable to parameter of type 'IPredictRequest'.
  Types of property 'instances' are incompatible.
    Type '{ content: string; }[]' is not assignable to type 'IValue[]'.
      Type '{ content: string; }' has no properties in common with type 'IValue'.ts(2345)

I'm also getting the same error as @lucksp when trying to execute the function.

[Seth-McKilla]

Apologies for the back-to-back posts! But implementing token refreshing was actually a breeze with the google-auth-library package. So I wanted to share an interim workaround until the predictionServiceClient is fixed; here's the full function that I'm using (error handling excluded for sake of simplicity):

import { GoogleAuth } from "google-auth-library";

const {
  GCP_PROJECT_ID,
  GCP_CLIENT_EMAIL,
  GCP_PRIVATE_KEY,
  GCP_LOCATION,
  GCP_PREDICTION_ENDPOINT_ID,
} = process.env;

const googleAuth = new GoogleAuth({
  projectId: GCP_PROJECT_ID!,
  credentials: {
    client_email: GCP_CLIENT_EMAIL!,
    private_key: GCP_PRIVATE_KEY!,
  },
  scopes: ["https://www.googleapis.com/auth/cloud-platform"],
});

export async function predictImage(imageUrl: string) {
  const base64Image = await convertImageToBase64(imageUrl);

  const apiRoute = `https://${GCP_LOCATION}-aiplatform.googleapis.com/v1/projects/${GCP_PROJECT_ID}/locations/${GCP_LOCATION}/endpoints/${GCP_PREDICTION_ENDPOINT_ID}:predict`;

  const payload = {
    instances: [
      {
        content: base64Image,
      },
    ],
    parameters: {
      confidenceThreshold: 0.5,
      maxPredictions: 5,
    },
  };

  const accessToken = await googleAuth.getAccessToken();

  const response = await fetch(apiRoute, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${accessToken}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(payload),
  });
  const data = await response.json();
  return data.predictions;
}

export async function convertImageToBase64(imageUrl: string): Promise<string> {
  const response = await fetch(imageUrl);
  const buffer = await response.arrayBuffer();
  return btoa(
    new Uint8Array(buffer).reduce(
      (data, byte) => data + String.fromCharCode(byte),
      ""
    )
  );
}

[danielbankhead]

Hey folks, I'm waiting for someone more familiar with the product to get back to help everyone. In the meantime, perhaps the instances value in the payload references an int ID of sorts; here's a sample:

https://github.com/googleapis/google-cloud-node/blob/f826ea9f8e1325d48dd8b343b9af5e994e871108/packages/google-cloud-aiplatform/samples/generated/v1/prediction_service.predict.js#L25-L76

[dizcology]

For image classification, does this sample help? https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/ai-platform/snippets/predict-image-classification.js#L57-L60

(Note the helper to_value method.)

Alternatively, you can build a protobuf.Value message directly, which are often deeply nested with number_value, list_value, etc. keys, https://protobuf.dev/reference/protobuf/google.protobuf/#value

[Seth-McKilla]

For image classification, does this sample help? https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/ai-platform/snippets/predict-image-classification.js#L57-L60

(Note the helper to_value method.)

Alternatively, you can build a protobuf.Value message directly, which are often deeply nested with number_value, list_value, etc. keys, https://protobuf.dev/reference/protobuf/google.protobuf/#value

This is the example I was using but got errors when trying to import the { instance, params, prediction } found here: https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/main/ai-platform/snippets/predict-image-classification.js#L31-L32

It was saying that they didn't exist

[dizcology]

Looks like those imports refer to files here: https://github.com/googleapis/google-cloud-node/tree/main/packages/google-cloud-aiplatform/protos/google/cloud/aiplatform/v1/schema/predict

Could you confirm that they have been installed?

[ryuzaki01]

This is how it's work

const prompt = {
    content: base64Image,
}
const instance = helpers.toValue(prompt);
const instances = [instance];
const parameter ={
  confidenceThreshold: 0.5,
  maxPredictions: 5,
};
const parameters = helpers.toValue(parameter);
const predictRequest = PredictRequest.fromObject({
  endpoint,
  instances,
  parameters,
});

const [predictResult] = await predictClient.predict(predictRequest);

Hope it's help

[goughjo02]

@ryuzaki01 where are you importing PredictRequest from?

[ryuzaki01]

PredictRequest

import {google, google as googleAi} from "@google-cloud/aiplatform/build/protos/protos"; import PredictRequest = googleAi.cloud.aiplatform.v1.PredictRequest;