Public Cloud Services

Unfortunately, all cloud vendors do not provide a friendly API to list all public cloud services and categories, as listed on AWS Products, GCP Products and Azure Services pages.

The idea is to have a unified JSON schema for all cloud services.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "array",
  "items": [
    {
      "type": "object",
      "properties": {
        "id": {
          "type": "string"
        },
        "name": {
          "type": "string"
        },
        "summary": {
          "type": "string"
        },
        "url": {
          "type": "string"
        },
        "categories": {
          "type": "array",
          "items": [
            {
              "type": "object",
              "properties": {
                "id": {
                  "type": "string"
                },
                "name": {
                  "type": "string"
                }
              },
              "required": [
                "id",
                "name"
              ]
            }
          ]
        },
        "tags": {
          "type": "array",
          "items": [
            {
              "type": "string"
            }
          ]
        }
      },
      "required": [
        "id",
        "name",
        "summary",
        "url",
        "categories",
        "tags"
      ]
    }
  ]
}

Scraping AWS Cloud Services

The AWS Products page uses undocumented https://aws.amazon.com/api/dirs/items/search endpoint to fetch paged JSON records for available cloud products.

# download AWS service JSON file and generate data/aws.json
pip install -r requirements.txt
python discovery/aws.py > data/aws.json

Scraping GCP Cloud Services

The GCP Products page is rendered on the server side and all data is embedded into the web page.

# scrap GCP Products page to get all services and generate data/gcp.json
pip install -r requirements.txt
python discovery/gcp.py > data/gcp.json

Scraping Azure Cloud Services

The Azure Services page is rendered on the server side and all data is embedded into the web page.

# scrap Azure Services page to get all services and generate data/azure.json 
pip install -r requirements.txt
python discovery/azure.py > data/azure.json

Microsoft365 Services

Edit the ms365.json file. Use data from this page.

Scraping Google Workspace Services (GSuite)

The page page contains all Google Workspace services.

# scrap Google Workspace page to get all services and generate data/gsuite.json
pip install -r requirements.txt
python discovery/gsuite.py > data/gsuite.json

CMP Services

Edit the cmp.json file. Use the CMP UI and documentation.

Credits

Edit the credits.json file.

Update/merge all tags

Run the tags.sh script to regenerate the tags.json file that contains all platform, category and services tags from all services.

Public static location

Upload all generated json files to the public cloud_tags Cloud Storage bucket.

Focus Areas update process

Focus Areas support specific services and categories based on this repo.
Updates to service/category mappings to Focus Areas are performed using the following process, and then updating the zenrouter-infra repo with the output.

Adding support of a Product to a Focus Area

Editing the ProductToFocusArea mapping file

• Create a feature branch and clone this repo
• Edit ProductToFocusArea.tsv , adding the entry
  • The file has the following columns:
  • product - This is the name of the product, take from the name attribute in the cloud catalog files
  • platform, p_group, focus_area - these values must be match one of the Focus Areas defined in FocusAreas.tsv
  • support_level - This must be PRIMARY or SECONDARY, mapped to ZenRouter skills tier
  • status - only entries with status VERIFIED will be processed by the build process
  • meets_volume_criteria and support_level_desc can be ignored and set to any value, this was used for initial FA mapping
• Here is an example of adding AWS Amplify to AWS DevOps Focus Area as a SECONDARY skill

  product     platform            p_group         focus_area  support_level   status        meets_volume_criteria  support_level_desc
  AWS Amplify AWS                 Infrastructure  DevOps      SECONDARY       VERIFIED      N/A                    N/A

Adding support of a Category to a Focus Area

Editing the CategoryToFocusArea mapping file

• Create a feature branch and clone this repo
• Edit CategoryToFocusArea.tsv , adding the entry
  • The file has the following columns:
  • platform, p_group, focus_area - these values must be match one of the Focus Areas defined in FocusAreas.tsv
  • platform_tag - this is the actual category tag
  • support_level - This must be PRIMARY or SECONDARY, mapped to ZenRouter skills tier
• Here is an example of adding aws/category/migration to AWS Databases Focus Area as a SECONDARY skill

  platform    p_group focus_area  category_tag    support_level
  AWS Data    Databases   aws/category/migration  SECONDARY

Build the Focus Area mapping json file

• Once the desired Product and/or Category changes have been completed, generate the focus areas json mapping file

  # Python 3.12 breaks PySpark due to removal of distutils, as such, Python3.11 is required
  brew install python@3.11
  brew install java
  mkdir -p build
  python3.11 -m venv build/venv
  source build/venv/bin/activate
  cd focus_areas/
  python -m pip install -r requirements.txt
  python ./build_focus_areas.py
  # verify the product was added to the focus area in data/focus_areas/all.json, then commit/push
  git add data/focus_areas/
  git commit -m "Added xyz Product to abc Focus Area"
  git push

• Open a PR into master and add PLs as reviewers

Deploying to BigQuery

Once merged into master, deploy the changes into BigQuery

# Valid ADC required to deploy this, or a configured service account
gcloud auth application-default login
git checkout master
git pull
mkdir -p build
python3 -m venv build/venv
source build/venv/bin/activate
cd focus_areas/
python -m pip install -r requirements.txt
python ./deploy_to_bq.py --build --deploy --project doit-zendesk-analysis

Deploying the changes in ZenRouter

Once merged into master, deploy the changes into ZenRouter Infra

• Generate the HCL for datastore.tf and copy to your clipboard

  git checkout master
  git pull
  mkdir -p build
  python3 -m venv build/venv
  source build/venv/bin/activate
  cd focus_areas/
  python -m pip install -r requirements.txt
  python ./generate_hcl.py  | pbcopy

• Create a feature branch in ZenRouter Infra, clone and checkout the branch
• edit datastore.tf, replacing variable "focus_areas" default attribute

  variable "focus_areas" {
  type = map(object({
  id               = string
  name             = string
  practice_area    = string
  primary_skills   = list(string)
  secondary_skills = list(string)
  }))
  default =  
  <Paste the output from generate_hcly.py here>
  }
  # initialize terraform
  terraform init
  # validate the file has no syntax issues
  terraform validate
  # format the file before commit
  terraform fmt datastore.tf

• commit/push/submit a PR in ZenRouter Infra