obriensystems commented 1 year ago

Several client requests for development environment setup beyond the default google cloud shell We have a containerized Dockerfile dev environment referenced in gcp-tools https://github.com/ssc-spc-ccoe-cei/gcp-tools/tree/main/devcontainer We need some initial documentation on creating a minimal dev environment locally outside of a full containerized version

wiki entry

Include 1 or more specific examples of calling google apis to BigQuery for example via python or java client APIs

BigQuery Example API call from a local M1 Macbook pro

BigQuery API call from https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494#issuecomment-1715822826

get auth token https://cloud.google.com/bigquery/docs/authorization https://cloud.google.com/docs/authentication/provide-credentials-adc#how-to https://github.com/GoogleCloudPlatform/shell-samples/blob/HEAD/bigquery/authorization/snippets.sh

gcloud auth application-default login

after we work with

gcloud auth application-default print-access-token
ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"

create a default big query table - in this case one of my GPS data exports 201611185_gps_distinct_rollerblad.csv

biometric:benchmark michaelobrien$ export GOOGLE_CLOUD_PROJECT=bigquery-ol
biometric:benchmark michaelobrien$ export GOOGLE_CLOUD_PROJECT=bigquery-ol

biometric:benchmark michaelobrien$ curl -H "Authorization: Bearer $ACCESS_TOKEN" \
>   "https://www.googleapis.com/bigquery/v2/projects/$GOOGLE_CLOUD_PROJECT/datasets"
{
  "kind": "bigquery#datasetList",
  "etag": "irRvAdMvQcDWywbnuMPyBg==",
  "datasets": [
    {
      "kind": "bigquery#dataset",
      "id": "bigquery-ol:rollerblade",
      "datasetReference": {
        "datasetId": "rollerblade",
        "projectId": "bigquery-ol"
      },
      "location": "northamerica-northeast1"
    }
  ]
}
curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade
{
  "kind": "bigquery#dataset",
  "etag": "YGr4j2cVE8o77F0DFWySFg==",
  "id": "bigquery-ol:rollerblade",
  "selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade",
  "datasetReference": {
    "datasetId": "rollerblade",
    "projectId": "bigquery-ol"
  },
  ...
}

biometric:benchmark michaelobrien$ curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
  "kind": "bigquery#tableDataList",
  "etag": "wVKEep3xEdb51S3g1w7o3w==",
  "totalRows": "4331",
  "pageToken": "BHGUBPMJRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAWYTJM5YXKZLSPEWW63ARQX7PZB65AAAAAEQLOJXWY3DFOJRGYYLEMUNCINLEHE2GIYTEHEWWMMJUMUWTIM3GMQWTQZJQMYWTEMJVGFRWEOJTME3DMMQSBNZG63DMMVZGE3DBMRSRUJBYG5RTQNZUGM3S2YZZGBSC2NBWGVTC2OJYGA4C2ZJXGY4WGMBTGRSGEMLE",
  "rows": [
    {
      "f": [
        {
          "v": "45.424712"
        },
        {
          "v": "-75.698209"

Developer Guide

obriensystems commented 1 year ago

Local GCP SDK docs - using a separate ubuntu VM on AWS

create a VM (with ssh key)
create EIP
associate EIP
ssh from local laptop


michaelobrien@mbp7 gcp-tools % dig bastion.obrienlabs.cloud
bastion.obrienlabs.cloud. 212   IN  A   54.167.180.91

michaelobrien@mbp7 gcp-tools % ssh ubuntu@bastion.obrienlabs.cloud

follow the linux (not the debian/ubuntu) version - even for ubuntu (via curl/tar)
https://cloud.google.com/sdk/docs/install#linux

prep

Install the gcloud CLI

curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-450.0.0-linux-x86_64.tar.gz

will take about a min

tar -xf google-cloud-cli-450.0.0-linux-x86_64.tar.gz

ubuntu@ip-172-31-46-142:~$ ./google-cloud-sdk/install.sh Welcome to the Google Cloud CLI!

To help improve the quality of this product, we collect anonymized usage data and anonymized stacktraces when crashes are encountered; additional information is available at https://cloud.google.com/sdk/usage-statistics. This data is handled in accordance with our privacy policy https://cloud.google.com/terms/cloud-privacy-notice. You may choose to opt in this collection now (by choosing 'Y' at the below prompt), or at any time in the future by running the following command:

gcloud config set disable_usage_reporting false

Do you want to help improve the Google Cloud CLI (y/N)? y

Your current Google Cloud CLI version is: 450.0.0 The latest available version is: 450.0.0

┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Components │ ├───────────────┬──────────────────────────────────────────────────────┬──────────────────────────────┬───────────┤ │ Status │ Name │ ID │ Size │ ├───────────────┼──────────────────────────────────────────────────────┼──────────────────────────────┼───────────┤ │ Not Installed │ App Engine Go Extensions │ app-engine-go │ 4.5 MiB │ │ Not Installed │ Appctl │ appctl │ 21.0 MiB │ │ Not Installed │ Artifact Registry Go Module Package Helper │ package-go-module │ < 1 MiB │ │ Not Installed │ Cloud Bigtable Command Line Tool │ cbt │ 16.1 MiB │ │ Not Installed │ Cloud Bigtable Emulator │ bigtable │ 7.1 MiB │ │ Not Installed │ Cloud Datastore Emulator │ cloud-datastore-emulator │ 36.2 MiB │ │ Not Installed │ Cloud Firestore Emulator │ cloud-firestore-emulator │ 42.8 MiB │ │ Not Installed │ Cloud Pub/Sub Emulator │ pubsub-emulator │ 61.4 MiB │ │ Not Installed │ Cloud Run Proxy │ cloud-run-proxy │ 13.3 MiB │ │ Not Installed │ Cloud SQL Proxy │ cloud_sql_proxy │ 7.8 MiB │ │ Not Installed │ Cloud Spanner Emulator │ cloud-spanner-emulator │ 35.2 MiB │ │ Not Installed │ Cloud Spanner Migration Tool │ harbourbridge │ 20.9 MiB │ │ Not Installed │ Google Container Registry's Docker credential helper │ docker-credential-gcr │ 1.8 MiB │ │ Not Installed │ Kustomize │ kustomize │ 4.3 MiB │ │ Not Installed │ Log Streaming │ log-streaming │ 13.9 MiB │ │ Not Installed │ Minikube │ minikube │ 34.6 MiB │ │ Not Installed │ Nomos CLI │ nomos │ 28.6 MiB │ │ Not Installed │ On-Demand Scanning API extraction helper │ local-extract │ 14.4 MiB │ │ Not Installed │ Skaffold │ skaffold │ 23.3 MiB │ │ Not Installed │ Spanner migration tool │ spanner-migration-tool │ 20.9 MiB │ │ Not Installed │ Terraform Tools │ terraform-tools │ 66.1 MiB │ │ Not Installed │ anthos-auth │ anthos-auth │ 20.4 MiB │ │ Not Installed │ config-connector │ config-connector │ 56.7 MiB │ │ Not Installed │ enterprise-certificate-proxy │ enterprise-certificate-proxy │ 8.2 MiB │ │ Not Installed │ gcloud Alpha Commands │ alpha │ < 1 MiB │ │ Not Installed │ gcloud Beta Commands │ beta │ < 1 MiB │ │ Not Installed │ gcloud app Java Extensions │ app-engine-java │ 123.7 MiB │ │ Not Installed │ gcloud app Python Extensions │ app-engine-python │ 8.4 MiB │ │ Not Installed │ gcloud app Python Extensions (Extra Libraries) │ app-engine-python-extras │ 31.5 MiB │ │ Not Installed │ gke-gcloud-auth-plugin │ gke-gcloud-auth-plugin │ 7.9 MiB │ │ Not Installed │ kpt │ kpt │ 14.4 MiB │ │ Not Installed │ kubectl │ kubectl │ < 1 MiB │ │ Not Installed │ kubectl-oidc │ kubectl-oidc │ 20.4 MiB │ │ Not Installed │ pkg │ pkg │ │ │ Installed │ BigQuery Command Line Tool │ bq │ 1.6 MiB │ │ Installed │ Bundled Python 3.9 │ bundled-python3-unix │ 63.6 MiB │ │ Installed │ Cloud Storage Command Line Tool │ gsutil │ 11.3 MiB │ │ Installed │ Google Cloud CLI Core Libraries │ core │ 21.9 MiB │ │ Installed │ Google Cloud CRC32C Hash Tool │ gcloud-crc32c │ 1.2 MiB │ └───────────────┴──────────────────────────────────────────────────────┴──────────────────────────────┴───────────┘ To install or remove components at your current SDK version [450.0.0], run: $ gcloud components install COMPONENT_ID $ gcloud components remove COMPONENT_ID

To update your SDK installation to the latest version [450.0.0], run: $ gcloud components update

Modify profile to update your $PATH and enable shell command completion?

Do you want to continue (Y/n)? y

The Google Cloud SDK installer will now prompt you to update an rc file to bring the Google Cloud CLIs into your environment.

Enter a path to an rc file to update, or leave blank to use [/home/ubuntu/.bashrc]:
Backing up [/home/ubuntu/.bashrc] to [/home/ubuntu/.bashrc.backup]. [/home/ubuntu/.bashrc] has been updated.

==> Start a new shell for the changes to take effect. For more information on how to get started, please visit: https://cloud.google.com/sdk/docs/quickstarts

ubuntu@ip-172-31-46-142:~$ source ~/.bashrc ubuntu@ip-172-31-46-142:~$ gcloud version Google Cloud SDK 450.0.0 bq 2.0.98 bundled-python3-unix 3.9.16 core 2023.10.06 gcloud-crc32c 1.0.0 gsutil 5.26

obriensystems commented 1 year ago

For now I have put up some wiki content on the LZ where I'll continue to post details on SDK calls into the googleapis.com API. The auth I have run so far - that is off the GCP side (local and another CSP jump box) - uses a bearer access token. However in a non-VPN/interconnect public environment - routing may be affected by any firewall rules or a firewall between the *.googleapis.com endpoint - will look into reproducing what you have posted.

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#local-to-gcp-api-calls-over-public-ips

The tracking issue where details are worked out is below https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/559

I had some recent work calling for example BigQuery from on-prem through both the public API and privately through a PSC private connection that I will return to. https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494#issuecomment-1715822826

There is also an older project https://github.com/CloudLandingZone/gcp-landing-zone where I started some java based SDK code in a maven project. This particular project - i only used the gcp shell for auth but will adapt it over the weekend for STS (eclipse)/local dev to reverify token based authentication. It currently uses the two import com.google.auth.http.HttpCredentialsAdapter; import com.google.auth.oauth2.GoogleCredentials;

in https://github.com/cloud-quickstart/gcp-landing-zone/blob/main/gcp-landing-zone-deploy/src/main/java/zone/gcp/landing/Cli.java#L83

JAVA GCP SDK

Install java either via apt-get or from http://java.sun.com

see https://github.com/cloud-quickstart/gcp-landing-zone#authenticate Verify the credential API outside of the GCP shell - a local and other CSP VM https://github.com/cloud-quickstart/gcp-landing-zone/blob/main/gcp-landing-zone-deploy/src/main/java/zone/gcp/landing/Cli.java#L83

sudo apt-get install openjdk-17-jdk

Install Spring Source Tool Suite for Eclipse

https://spring.io/tools https://github.com/spring-projects/sts4/wiki/Previous-Versions

extract the jar in powershell

checkout code

https://github.com/cloud-quickstart/gcp-landing-zone


micha@carbon MINGW64 /c/wse_github/cloud-quickstart/gcp-landing-zone (main)
$ eval $(ssh-agent -s)
$ ssh-add ~/.ssh/obrienlabs_org_github
# git pull

mvn build code

mvn clean install -U

[INFO] Building jar: C:\wse_github\cloud-quickstart\gcp-landing-zone\gcp-landing-zone-deploy\target\gcp-landing-zone-deploy-0.0.1-SNAPSHOT-jar-with-dependencies.jar
[INFO] gcp-landing-zone-root .............................. SUCCESS [  0.204 s]
[INFO] gcp-landing-zone-deploy ............................ SUCCESS [01:37 min]

Import existing maven project into eclipse

20231014: test authentication

By default on a new machine - no GCP authentication credentials yet

A run is expected to fail - OK

initializing
Unable to initialize service: 
The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.[Ljava.lang.StackTraceElement;@5025a98f
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "com.google.api.services.cloudresourcemanager.v3.CloudResourceManager.projects()" because "crmService" is null
    at zone.gcp.landing.Cli.getPolicy(Cli.java:149)
    at zone.gcp.landing.Cli.addBinding(Cli.java:93)
    at zone.gcp.landing.Cli.main(Cli.java:44)

Authenticate on X1 carbon gen 9

open a google browser with the chrome profile set to your gcloud account

in ming64 bash
micha@carbon MINGW64 /c/wse_github/cloud-quickstart/gcp-landing-zone (main)
$ gcloud auth application-default login
Your browser has been opened to visit:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client

Select account

accept permissions

Credentials saved to file: [C:\Users\micha\AppData\Roaming\gcloud\application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "duet-ai-old" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.

navigate to suggested https://cloud.google.com/sdk/auth_success

Eclipse must be set for cloud library access not ADC (in the shell)

we are set for the wrong project
micha@carbon MINGW64 /c/wse_github/cloud-quickstart/gcp-landing-zone (main)
$ gcloud config get project
duet-ai-old

in eclipse...
initializing
Oct. 14, 2023 4:12:20 P.M. com.google.auth.oauth2.DefaultCredentialsProvider warnAboutProblematicCredentials
WARNING: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.
Exception in thread "main" Unable to get policy: 
403 Forbidden
POST https://cloudresourcemanager.googleapis.com/v3/projects/lz-stg:getIamPolicy
{
  "code" : 403,
  "details" : [ {
    "@type" : "type.googleapis.com/google.rpc.Help"
  }, {
    "@type" : "type.googleapis.com/google.rpc.ErrorInfo",
    "reason" : "SERVICE_DISABLED"
  } ],
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Cloud Resource Manager API has not been used in project duet-ai-old before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=duet-ai-old then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
    "reason" : "accessNotConfigured",
    "extendedHelp" : "https://console.developers.google.com"
  } ],
  "message" : "Cloud Resource Manager API has not been used in project duet-ai-old before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=duet-ai-old then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
  "status" : "PERMISSION_DENIED"
}[Ljava.lang.StackTraceElement;@2609b277
java.lang.NullPointerException: Cannot invoke "com.google.api.services.cloudresourcemanager.v3.model.Policy.getBindings()" because "policy" is null
    at zone.gcp.landing.Cli.addBinding(Cli.java:97)
    at zone.gcp.landing.Cli.main(Cli.java:44)

Lets setup eclipse with the proper environment variables

We are not using the following as it only supports up to Java 11 (we are using 17) https://cloud.google.com/eclipse/docs/quickstart

Follow https://cloud.google.com/java/docs/setup

Setup a service account for authentication and impersonate it

        GoogleCredentials credential =
            GoogleCredentials.getApplicationDefault()
                .createScoped(Collections.singleton(IamScopes.CLOUD_PLATFORM));

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494#issuecomment-1717905153

Macbook pro 16 M1 running springsource tool suite 4.13.1 - JDK 17.0.5

Add stub endpoint to spring boot app https://github.com/obrienlabs/magellan/commit/a2d45f7a8e627b0405c898f6d5ea9a22413d8941

Follow https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key

Setup a service account on the genai project, set the GOOGLE_APPLICATION_CREDENTIALS env variable

or via gcloud https://cloud.google.com/iam/docs/keys-create-delete#creating

First reauthenticate gcloud locally

michaelobrien@mbp7 magellan % gcloud auth login

https://cloud.google.com/sdk/auth_success

michaelobrien@mbp7 magellan % gcloud config set project gen-ai-old
WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.

To update your Application Default Credentials quota project, use the `gcloud auth application-default set-quota-project` command.
Updated property [core/project].

set quota project as i was using ADC https://cloud.google.com/docs/authentication/provide-credentials-adc

michaelobrien@mbp7 magellan % gcloud auth application-default set-quota-project gen-ai-old
API [cloudresourcemanager.googleapis.com] not enabled on project [gen-ai-old]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  y

Enabling service [cloudresourcemanager.googleapis.com] on project [gen-ai-old]...
Operation "operations/acat.p2-310840400174-3ad62c19-2186-4147-9445-4ac0817a0483" finished successfully.

Credentials saved to file: [/Users/michaelobrien/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "gen-ai-old" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.

To revert your CLI to the previously installed version, you may run:
  $ gcloud components update --version 419.0.0

michaelobrien@mbp7 magellan % gcloud components update

Create Service Account

https://cloud.google.com/iam/docs/service-accounts-create

michaelobrien@mbp7 magellan % gcloud iam service-accounts create gen-ai-old-sa --description="gen-ai-old-sa" --display-name="gen-ai-old-sa"                       
Created service account [gen-ai-old-sa].

Add roles to SA

gcloud projects add-iam-policy-binding gen-ai-old --member="serviceAccount:gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com" --role="roles/owner"

Create Service Account Key

https://cloud.google.com/iam/docs/keys-create-delete#creating

michaelobrien@mbp7 magellan % gcloud iam service-accounts keys create ~/keys/gcp-gen-ai-old-sa.json --iam-account=gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com
created key [02f9b5c04a76987e10f1e67467b326be53122c7e] of type [json] as [/Users/michaelobrien/keys/gcp-gen-ai-old-sa.json] for [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com]

michaelobrien@mbp7 magellan % cat ~/keys/gcp-gen-ai-old-sa.json 
{
  "type": "service_account",
  "project_id": "gen-ai-old",
  "private_key_id": "02f9b.....2c7e",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkq.....YFedw=\n-----END PRIVATE KEY-----\n",
  "client_email": "gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com",
  "client_id": "108...90",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gen-ai-old-sa%40gen-ai-old.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Export key

michaelobrien@mbp7 magellan % export GOOGLE_APPLICATION_CREDENTIALS="~/keys/gcp-gen-ai-old-sa.json"

Add service account impersonation

https://cloud.google.com/docs/authentication/use-service-account-impersonation

IAM is enabled on the project

serviceAccountTokenCreator is on your user roles

suspect I need to add to the SA list - my user - I do https://console.cloud.google.com/iam-admin/serviceaccounts/details/108536977121828646390/permissions?project=gen-ai-old

check access for user

michaelobrien@mbp7 magellan % gcloud storage buckets list

creation_time: 2023-10-16T16:02:24+0000
default_storage_class: STANDARD
location: US
location_type: multi-region
metageneration: 1
name: gen-ai-old
public_access_prevention: enforced
rpo: DEFAULT
storage_url: gs://gen-ai-old/
uniform_bucket_level_access: true
update_time: 2023-10-16T16:02:24+0000

check SA

michaelobrien@mbp7 magellan % gcloud storage buckets list --impersonate-service-account=gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com
WARNING: This command is using service account impersonation. All API calls will be executed as [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com].
---
creation_time: 2023-10-16T16:02:24+0000
default_storage_class: STANDARD
location: US
location_type: multi-region
metageneration: 1
name: gen-ai-old
public_access_prevention: enforced
rpo: DEFAULT
storage_url: gs://gen-ai-old/
uniform_bucket_level_access: true
update_time: 2023-10-16T16:02:24+0000

obriensystems commented 1 year ago

Clean developer machine - Windows11 version - i9-13900k, 192g ram, dual 4090, dual samsung 990

Already has JDK 17, git, git-bash, clean eclipse sts 4.4 but no certs, repos, configs, gcp-configs - currently being used only for VCC22 and cuda

download sts 4.4 https://spring.io/tools

extract jar

michael@13900b MINGW64 /c/sts44
$ java -jar spring-tool-suite-4-4.20.0.RELEASE-e4.29.0-win32.win32.x86_64.self-extracting.jar

move to /c/sts44 create /c/wse_sts44 workspace

create keys http://wiki.obrienlabs.cloud/display/DEV/Developer+Guide#DeveloperGuide-Windows10

michael@13900b MINGW64 /c/sts44
$ eval $(ssh-agent -s)
Agent pid 1104
michael@13900b MINGW64 /c/sts44
$ ssh-add ~/.ssh/obrien...hub
Identity added: /c/Users/michael/.ssh/obri...ub (mich...rg)

clone/adapt landing zone SDK java project

michael@13900b MINGW64 /c/wse_sts44
$ mkdir cloud-quickstart
michael@13900b MINGW64 /c/wse_sts44
$ cd cloud-quickstart/
michael@13900b MINGW64 /c/wse_sts44/cloud-quickstart
$ git clone git@github.com:cloud-quickstart/gcp-landing-zone.git

clone/adapt spring boot endpoint project

michael@13900b MINGW64 /c/wse_sts44
$ mkdir obrienlabs
michael@13900b MINGW64 /c/wse_sts44
$ cd obrienlabs/
michael@13900b MINGW64 /c/wse_sts44/obrienlabs
$ git clone git@github.com:obrienlabs/magellan.git

install maven

build on command line

michael@13900b MINGW64 /c/wse_sts44/obrienlabs/magellan (master)
 mvn clean install -U -dSkipTests=true

build in eclipse

switch mvn and java installs in properties
import existing maven project - run mvn install

Test a commit

michael@13900b MINGW64 /c/wse_sts44/obrienlabs/magellan (master)
$ eval $(ssh-agent -s)
Agent pid 1347
michael@13900b MINGW64 /c/wse_sts44/obrienlabs/magellan (master)
$ ssh-add ~/.ssh/obrienlabs_org_github
Identity added: /c/Users/michael/.ssh/obrienlabs_...bs.org)
michael@13900b MINGW64 /c/wse_sts44/obrienlabs/magellan (master)
$ git push origin master
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 32 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 364 bytes | 364.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
remote:
remote: GitHub found 1 vulnerability on obrienlabs/magellan's default branch (1 high). To find out more, visit:
remote:      https://github.com/obrienlabs/magellan/security/dependabot/1
remote:
To github.com:obrienlabs/magellan.git
   6cc64af..7328e2e  master -> master

Developer machine - M1 Macbook Pro

Test API endpoints

http://localhost:8080/nbi/swagger-ui.html

obriensystems commented 1 year ago

see #562 Review use of ADC with the client https://cloud.google.com/docs/authentication/provide-credentials-adc ADC is used by https://github.com/cloud-quickstart/gcp-landing-zone/blob/main/gcp-landing-zone-deploy/src/main/java/zone/gcp/landing/Cli.java#L74

GoogleCredentials credential =
            GoogleCredentials.getApplicationDefault()
                .createScoped(Collections.singleton(IamScopes.CLOUD_PLATFORM));

obriensystems commented 1 year ago

Use ADC credentials via SA

https://cloud.google.com/docs/authentication/client-libraries https://cloud.google.com/docs/authentication/client-libraries#java

set maven https://cloud.google.com/storage/docs/reference/libraries

add to pom.xml as usual

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>26.24.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-storage</artifactId>
  </dependency>

Run a GCP GCS example


@Service
public class ApplicationService implements ApplicationServiceLocal {

    @Override
    public String health() {
        return "OK";
    }

    @Override
    public String gcp() {
        try {
            authenticateImplicitWithAdc("gen-ai-old");
        } catch (IOException io) {
            System.out.println(io.getMessage());
        }
        return "gcp";
    }

    // https://cloud.google.com/docs/authentication/client-libraries#java
    private void authenticateImplicitWithAdc(String project) throws IOException {

            // *NOTE*: Replace the client created below with the client required for your application.
            // Note that the credentials are not specified when constructing the client.
            // Hence, the client library will look for credentials using ADC.
            //
            // Initialize client that will be used to send requests. This client only needs to be created
            // once, and can be reused for multiple requests.
            Storage storage = StorageOptions.newBuilder().setProjectId(project).build().getService();

            System.out.println("Buckets:");
            Page<Bucket> buckets = storage.list();
            for (Bucket bucket : buckets.iterateAll()) {
              System.out.println(bucket.toString());
            }
            System.out.println("Listed all storage buckets.");
          }

    @Override
    public String forward() {
        // TODO Auto-generated method stub
        return "OK";
    }
}

2023-10-16 12:46:34.327 DEBUG 61449 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : GET "/nbi/app/gcp", parameters={}
2023-10-16 12:46:34.328 DEBUG 61449 --- [nio-8080-exec-8] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ApplicationServiceController#getGCP()
Buckets:
Bucket{name=gen-ai-old}
Listed all storage buckets.
2023-10-16 12:46:34.992 DEBUG 61449 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2023-10-16 12:46:34.992 DEBUG 61449 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Writing ["gcp"]
2023-10-16 12:46:34.993 DEBUG 61449 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2023-10-16 12:46:35.013 DEBUG 61449 --- [nio-8080-exec-4] o.s.web.servlet.DispatcherServlet        : GET "/nbi/webjars/springfox-swagger-ui/fonts/titillium-web-v6-latin-600.woff2", parameters={}
2023-10-16 12:46:35.014 DEBUG 61449 --- [nio-8080-exec-4] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/webjars/"]
2023-10-16 12:46:35.016 DEBUG 61449 --- [nio-8080-exec-4] o.s.web.servlet.DispatcherServlet        : Completed 200 OK

Working ADC GCP for GCS example from above https://github.com/obrienlabs/magellan/commit/554164ff0fd09e59abcc174eaf5b834b75bd3579

GoogleCloudPlatform / pubsec-declarative-toolkit

Developer Docs: Add local dev environment/authentication docs with googleapis.com example #559

wiki entry

BigQuery Example API call from a local M1 Macbook pro

Developer Guide

Local GCP SDK docs - using a separate ubuntu VM on AWS

will take about a min

JAVA GCP SDK

Install Spring Source Tool Suite for Eclipse

checkout code

mvn build code

Import existing maven project into eclipse

By default on a new machine - no GCP authentication credentials yet

Authenticate on X1 carbon gen 9

Eclipse must be set for cloud library access not ADC (in the shell)

Lets setup eclipse with the proper environment variables

Setup a service account for authentication and impersonate it

Create Service Account

Add roles to SA

Create Service Account Key

Export key

Add service account impersonation

Clean developer machine - Windows11 version - i9-13900k, 192g ram, dual 4090, dual samsung 990

install maven

Developer machine - M1 Macbook Pro

Test API endpoints

Use ADC credentials via SA