obrienlabs / magellan

Magellan - world packet navigation
1 stars 2 forks source link

Add GCP PSC endpoint googleapis.com REST call to BigQuery or Vertex/GenAI #26

Open obriensystems opened 1 year ago

obriensystems commented 1 year ago

Add to https://github.com/obrienlabs/magellan

fmichaelobrien commented 1 year ago

Add code around https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/562 via https://cloud.google.com/generative-ai-app-builder/docs/locations#specify_a_multi-region_for_your_data_store

Add new spring boot endpoint https://github.com/obrienlabs/magellan/commit/a2d45f7a8e627b0405c898f6d5ea9a22413d8941

obriensystems commented 1 year ago

Macbook pro 16 M1 running springsource tool suite 4.13.1 - JDK 17.0.5

Add stub endpoint to spring boot app https://github.com/obrienlabs/magellan/commit/a2d45f7a8e627b0405c898f6d5ea9a22413d8941

Follow https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key

Setup a service account on the genai project, set the GOOGLE_APPLICATION_CREDENTIALS env variable

Screenshot 2023-10-16 at 11 32 05

or via gcloud https://cloud.google.com/iam/docs/keys-create-delete#creating

First reauthenticate gcloud locally

michaelobrien@mbp7 magellan % gcloud auth login

https://cloud.google.com/sdk/auth_success

michaelobrien@mbp7 magellan % gcloud config set project gen-ai-old
WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.

To update your Application Default Credentials quota project, use the `gcloud auth application-default set-quota-project` command.
Updated property [core/project].

set quota project as i was using ADC https://cloud.google.com/docs/authentication/provide-credentials-adc

michaelobrien@mbp7 magellan % gcloud auth application-default set-quota-project gen-ai-old
API [cloudresourcemanager.googleapis.com] not enabled on project [gen-ai-old]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  y

Enabling service [cloudresourcemanager.googleapis.com] on project [gen-ai-old]...
Operation "operations/acat.p2-310840400174-3ad62c19-2186-4147-9445-4ac0817a0483" finished successfully.

Credentials saved to file: [/Users/michaelobrien/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "gen-ai-old" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.
To revert your CLI to the previously installed version, you may run:
  $ gcloud components update --version 419.0.0

michaelobrien@mbp7 magellan % gcloud components update

Create Service Account

https://cloud.google.com/iam/docs/service-accounts-create

michaelobrien@mbp7 magellan % gcloud iam service-accounts create gen-ai-old-sa --description="gen-ai-old-sa" --display-name="gen-ai-old-sa"                       
Created service account [gen-ai-old-sa].

Add roles to SA

gcloud projects add-iam-policy-binding gen-ai-old --member="serviceAccount:gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com" --role="roles/owner"

Create Service Account Key

https://cloud.google.com/iam/docs/keys-create-delete#creating

michaelobrien@mbp7 magellan % gcloud iam service-accounts keys create ~/keys/gcp-gen-ai-old-sa.json --iam-account=gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com
created key [02f9b5c04a76987e10f1e67467b326be53122c7e] of type [json] as [/Users/michaelobrien/keys/gcp-gen-ai-old-sa.json] for [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com]

michaelobrien@mbp7 magellan % cat ~/keys/gcp-gen-ai-old-sa.json 
{
  "type": "service_account",
  "project_id": "gen-ai-old",
  "private_key_id": "02f9b.....2c7e",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkq.....YFedw=\n-----END PRIVATE KEY-----\n",
  "client_email": "gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com",
  "client_id": "108...90",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gen-ai-old-sa%40gen-ai-old.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Export key

michaelobrien@mbp7 magellan % export GOOGLE_APPLICATION_CREDENTIALS="~/keys/gcp-gen-ai-old-sa.json"

Add service account impersonation

https://cloud.google.com/docs/authentication/use-service-account-impersonation

IAM is enabled on the project

serviceAccountTokenCreator is on your user roles

suspect I need to add to the SA list - my user - I do https://console.cloud.google.com/iam-admin/serviceaccounts/details/108536977121828646390/permissions?project=gen-ai-old

check access for user

michaelobrien@mbp7 magellan % gcloud storage buckets list

creation_time: 2023-10-16T16:02:24+0000
default_storage_class: STANDARD
location: US
location_type: multi-region
metageneration: 1
name: gen-ai-old
public_access_prevention: enforced
rpo: DEFAULT
storage_url: gs://gen-ai-old/
uniform_bucket_level_access: true
update_time: 2023-10-16T16:02:24+0000

check SA

michaelobrien@mbp7 magellan % gcloud storage buckets list --impersonate-service-account=gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com
WARNING: This command is using service account impersonation. All API calls will be executed as [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com].
---
creation_time: 2023-10-16T16:02:24+0000
default_storage_class: STANDARD
location: US
location_type: multi-region
metageneration: 1
name: gen-ai-old
public_access_prevention: enforced
rpo: DEFAULT
storage_url: gs://gen-ai-old/
uniform_bucket_level_access: true
update_time: 2023-10-16T16:02:24+0000
obriensystems commented 1 year ago

Use ADC credentials via SA

https://cloud.google.com/docs/authentication/client-libraries https://cloud.google.com/docs/authentication/client-libraries#java

set maven https://cloud.google.com/storage/docs/reference/libraries

add to pom.xml as usual

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>26.24.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-storage</artifactId>
  </dependency>

Run a GCP GCS example


@Service
public class ApplicationService implements ApplicationServiceLocal {

    @Override
    public String health() {
        return "OK";
    }

    @Override
    public String gcp() {
        try {
            authenticateImplicitWithAdc("gen-ai-old");
        } catch (IOException io) {
            System.out.println(io.getMessage());
        }
        return "gcp";
    }

    // https://cloud.google.com/docs/authentication/client-libraries#java
    private void authenticateImplicitWithAdc(String project) throws IOException {

            // *NOTE*: Replace the client created below with the client required for your application.
            // Note that the credentials are not specified when constructing the client.
            // Hence, the client library will look for credentials using ADC.
            //
            // Initialize client that will be used to send requests. This client only needs to be created
            // once, and can be reused for multiple requests.
            Storage storage = StorageOptions.newBuilder().setProjectId(project).build().getService();

            System.out.println("Buckets:");
            Page<Bucket> buckets = storage.list();
            for (Bucket bucket : buckets.iterateAll()) {
              System.out.println(bucket.toString());
            }
            System.out.println("Listed all storage buckets.");
          }

    @Override
    public String forward() {
        // TODO Auto-generated method stub
        return "OK";
    }
}

2023-10-16 12:46:34.327 DEBUG 61449 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : GET "/nbi/app/gcp", parameters={}
2023-10-16 12:46:34.328 DEBUG 61449 --- [nio-8080-exec-8] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ApplicationServiceController#getGCP()
Buckets:
Bucket{name=gen-ai-old}
Listed all storage buckets.
2023-10-16 12:46:34.992 DEBUG 61449 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2023-10-16 12:46:34.992 DEBUG 61449 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Writing ["gcp"]
2023-10-16 12:46:34.993 DEBUG 61449 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2023-10-16 12:46:35.013 DEBUG 61449 --- [nio-8080-exec-4] o.s.web.servlet.DispatcherServlet        : GET "/nbi/webjars/springfox-swagger-ui/fonts/titillium-web-v6-latin-600.woff2", parameters={}
2023-10-16 12:46:35.014 DEBUG 61449 --- [nio-8080-exec-4] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/webjars/"]
2023-10-16 12:46:35.016 DEBUG 61449 --- [nio-8080-exec-4] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
obriensystems commented 1 year ago

Working ADC GCP for GCS example from above https://github.com/obrienlabs/magellan/commit/554164ff0fd09e59abcc174eaf5b834b75bd3579

Screenshot 2023-10-16 at 12 58 26
obriensystems commented 1 year ago

Adding alternate direct File classloader (no ENV variable) - requested by customer (tomcat war) - but this example is spring boot jar with embedded tomcat

https://github.com/obrienlabs/magellan/commit/52641990082c32b22f32137f8316382d88b55264


    private String authenticateFileWithAdc(String project) throws IOException {

        String lastBucket = "none";

        // *NOTE*: Replace the client created below with the client required for your application.
        // Note that the credentials are not specified when constructing the client.
        // Hence, the client library will look for credentials using ADC.
        //
        // Initialize client that will be used to send requests. This client only needs to be created
        // once, and can be reused for multiple requests.

        // move inside the resource path - for jav/war deployment
        //File credentialsPath = new File("~/keys/gcp-gen-ai-old-sa.json");
        File credentialsPath = ResourceUtils.getFile("classpath:gcp-gen-ai-old-sa.json");

        // Load credentials from JSON key file. If you can't set the GOOGLE_APPLICATION_CREDENTIALS
        // environment variable, you can explicitly load the credentials file to construct the
        // credentials.
        GoogleCredentials credentials;
        try (FileInputStream serviceAccountStream = new FileInputStream(credentialsPath)) {
          credentials = ServiceAccountCredentials.fromStream(serviceAccountStream);
        }
        Storage storage = StorageOptions
                .newBuilder()
                .setCredentials(credentials)
                .setProjectId(project).build().getService();

        logger.info("Buckets:");
        Page<Bucket> buckets = storage.list();
        for (Bucket bucket : buckets.iterateAll()) {
            logger.info(bucket.toString());
          lastBucket = bucket.toString();
        }
        logger.info("Listed all storage buckets.");
        return lastBucket;
    }

    023-10-18 12:08:31.579 DEBUG 73958 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : GET "/nbi/app/gcpViaFile", parameters={}
2023-10-18 12:08:31.580 DEBUG 73958 --- [nio-8080-exec-8] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ApplicationServiceController#getGCPViaFile()
2023-10-18 12:08:31.651  INFO 73958 --- [nio-8080-exec-8] g.p.magellan.service.ApplicationService  : Buckets:
2023-10-18 12:08:32.173  INFO 73958 --- [nio-8080-exec-8] g.p.magellan.service.ApplicationService  : Bucket{name=gen-ai-old}
2023-10-18 12:08:32.173  INFO 73958 --- [nio-8080-exec-8] g.p.magellan.service.ApplicationService  : Listed all storage buckets.
2023-10-18 12:08:32.174 DEBUG 73958 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2023-10-18 12:08:32.174 DEBUG 73958 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Writing ["Bucket{name=gen-ai-old}"]
2023-10-18 12:08:32.174 DEBUG 73958 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
michaelobrien@mbp7 magellan % git add magellan-nbi/src/main/java/
michaelobrien@mbp7 magellan % git status
On branch master
Your branch is up to date with 'origin/master'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    modified:   magellan-nbi/pom.xml
    modified:   magellan-nbi/src/main/java/global/packet/magellan/controller/ApplicationServiceController.java
    modified:   magellan-nbi/src/main/java/global/packet/magellan/service/ApplicationService.java
    modified:   magellan-nbi/src/main/java/global/packet/magellan/service/ApplicationServiceLocal.java

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    KEY_FILE
    magellan-nbi/src/main/resources/gcp-gen-ai-old-sa.json

https://cloud.google.com/bigquery/docs/authentication/service-account-file https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/AuthSnippets.java