search

GoogleCloudPlatform / pubsec-declarative-toolkit

The GCP PubSec Declarative Toolkit is a collection of declarative solutions to help you on your Journey to Google Cloud. Solutions are designed using Config Connector and deployed using Config Controller.
Apache License 2.0
30 stars 26 forks source link

Canary onboarding: SaaS use case: Add KCC/Gcloud deployment options limited roles around BigQuery/AutoML/DocumentAI #220

Open fmichaelobrien opened 1 year ago

fmichaelobrien commented 1 year ago

Code in the following branch/location https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/canary/solutions/document-processing Readme/Architecture https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/canary/solutions/document-processing#readme

Details and automation around onboarding a BigQuery/AutoML/DocumentAI use case for serverless/SaaS

Services

IAM roles/permissions

Reference for scripting https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh

Service Account permissions

bootstrap

export ORG_ID=$(gcloud projects get-ancestors $BOOT_PROJECT_ID --format='get(id)' | tail -1)
export EMAIL=$(gcloud config list --format json | jq .core.account | sed 's/"//g')

service enablement

gcloud services enable cloudbilling.googleapis.com


predefined roles

gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "user:${EMAIL}" --role roles/aiplatform.admin


Document AI Service Account
https://cloud.google.com/document-ai/docs/setup?continue=https%3A%2F%2Fdevelopers.google.com%2Flearn%2Ftopics%2Fdocumentai&utm_source=developers.google.com&utm_medium=referral


Systems
transport.g.z
obriensystems commented 1 year ago

test results: create and immediate delete -d true

root_@cloudshell:~/_current/pubsec-declarative-toolkit/solutions/document-processing/gcloud (pdt-tgz)$ ./deployment.sh -b pdt-tgz -u pdt3 -c true -l false -d true -e res....zone 
Date: Fri 16 Dec 2022 06:41:02 PM UTC
Timestamp: 1671216062
running with: -b pdt-tgz -u pdt3 -c true -l false -d true -e restricted@transport.gcp.zone -p
Updated property [core/project].
Switched back to boot project pdt-tgz
Start: 1671216062
unique string: pdt3
REGION: northamerica-northeast1
NETWORK: pdt-pdt3-vpc
SUBNET: pdt-pdt3-sn
CLUSTER: pdt-pdt3
Creating project: kcc-lz-6300
CC_PROJECT_ID:
BOOT_PROJECT_ID: pdt-tgz
BILLING_ID: 01...85D
ORG_ID: 442178577666
Creating KCC project: kcc-lz-6300
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-lz-6300].
Waiting for [operations/cp.9195833045414069903] to finish...done.    
Enabling service [cloudapis.googleapis.com] on project [kcc-lz-6300]...
Operation "operations/acat.p2-1038023658165-1c361f27-31ec-43a1-a765-839f6baee541" finished successfully.
Updated property [core/project] to [kcc-lz-6300].
Updated property [core/project].
billingAccountName: billingAccounts/011...5D
billingEnabled: true
name: projects/kcc-lz-6300/billingInfo
projectId: kcc-lz-6300
Adding roles to project for user: restr...ne
Updated IAM policy for project [kcc-lz-6300].
Updated IAM policy for project [kcc-lz-6300].
Updated IAM policy for project [kcc-lz-6300].
Updated IAM policy for project [kcc-lz-6300].
API's before
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: datastore.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com
Enabling APIs
Operation "operations/acat.p2-1038023658165-8770c1fb-0d3e-4311-8df1-e8051662001e" finished successfully.
Operation "operations/acf.p2-1038023658165-fc1be01a-75cb-4ef3-9846-1dc140fbd9ea" finished successfully.
Operation "operations/acat.p2-1038023658165-c97312fd-28a4-4140-93f9-bad947073aba" finished successfully.
Operation "operations/acf.p2-1038023658165-8966b2f5-ebb7-4c90-85cf-719ff23a1b40" finished successfully.
Operation "operations/acf.p2-1038023658165-99889578-3ff4-444e-ab6c-6fdfcf88cfb9" finished successfully.
Operation "operations/acf.p2-1038023658165-3024943c-e1d6-4409-8a1e-3e8600fd4825" finished successfully.
Operation "operations/acat.p2-1038023658165-e06c74a3-ad6a-495b-821e-b3002919af15" finished successfully.
API's after
NAME: aiplatform.googleapis.com
NAME: autoscaling.googleapis.com
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudbilling.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: compute.googleapis.com
NAME: container.googleapis.com
NAME: containerfilesystem.googleapis.com
NAME: containerregistry.googleapis.com
NAME: dataflow.googleapis.com
NAME: datastore.googleapis.com
NAME: deploymentmanager.googleapis.com
NAME: documentai.googleapis.com
NAME: healthcare.googleapis.com
NAME: iam.googleapis.com
NAME: iamcredentials.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: notebooks.googleapis.com
NAME: oslogin.googleapis.com
NAME: pubsub.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com
Deleting kcc-lz-6300
disable billing on - and delete kcc-lz-6300
billingAccountName: ''
billingEnabled: false
name: projects/kcc-lz-6300/billingInfo
projectId: kcc-lz-6300
Deleted [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-lz-6300].

You can undo this operation for a limited period by running the command below.
    $ gcloud projects undelete kcc-lz-6300

See https://cloud.google.com/resource-manager/docs/creating-managing-projects for information on shutting down projects.
Updated property [core/project].
Switched back to boot project pdt-tgz
Use the following command to switch to your new project
gcloud config set project kcc-lz-6300
**** Done ****

create only
root_@cloudshell:~/_current/pubsec-declarative-toolkit/solutions/document-processing/gcloud (pdt-tgz)$ ./deployment.sh -b pdt-tgz -u pdt3 -c true -l false -d false -e rest...one
Date: Fri 16 Dec 2022 06:46:45 PM UTC
Timestamp: 1671216405
running with: -b pdt-tgz -u pdt3 -c true -l false -d false -e restricted@transport.gcp.zone -p
Updated property [core/project].
Switched back to boot project pdt-tgz
Start: 1671216406
unique string: pdt3
REGION: northamerica-northeast1
NETWORK: pdt-pdt3-vpc
SUBNET: pdt-pdt3-sn
CLUSTER: pdt-pdt3
Creating project: kcc-lz-3479
CC_PROJECT_ID:
BOOT_PROJECT_ID: pdt-tgz
BILLING_ID: 011...5D
ORG_ID: 442178577666
Creating KCC project: kcc-lz-3479
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/kcc-lz-3479].
Waiting for [operations/cp.6912354144468666393] to finish...done.    
Enabling service [cloudapis.googleapis.com] on project [kcc-lz-3479]...
Operation "operations/acat.p2-374013806670-2e97bc09-84b0-49e9-94b3-45c545cce84e" finished successfully.
Updated property [core/project] to [kcc-lz-3479].
Updated property [core/project].
billingAccountName: billingAccounts/011...
billingEnabled: true
name: projects/kcc-lz-3479/billingInfo
projectId: kcc-lz-3479
Adding roles to project for user: restr...one
Updated IAM policy for project [kcc-lz-3479].
Updated IAM policy for project [kcc-lz-3479].
Updated IAM policy for project [kcc-lz-3479].
Updated IAM policy for project [kcc-lz-3479].
API's before
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: datastore.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com
Enabling APIs
Operation "operations/acat.p2-374013806670-2954f6a2-f36e-4183-b5bc-9be9b1a95172" finished successfully.
Operation "operations/acf.p2-374013806670-2ebdba0c-e94a-4fcc-b320-32fc027105a2" finished successfully.
Operation "operations/acat.p2-374013806670-67affaf9-d93f-434f-a0c8-d2acddc9f68d" finished successfully.
Operation "operations/acf.p2-374013806670-b1882ff1-4bdb-4d55-9ae2-b8735d074a11" finished successfully.
Operation "operations/acf.p2-374013806670-8913ebaa-3bde-4be9-9e41-f60c64ddc2d2" finished successfully.
Operation "operations/acf.p2-374013806670-7cde82ff-3975-4d44-a5eb-1edf273322ca" finished successfully.
Operation "operations/acat.p2-374013806670-d5e6834a-a664-49de-943c-55bbe075c461" finished successfully.
API's after
NAME: aiplatform.googleapis.com
NAME: autoscaling.googleapis.com
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudbilling.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: compute.googleapis.com
NAME: container.googleapis.com
NAME: containerfilesystem.googleapis.com
NAME: containerregistry.googleapis.com
NAME: dataflow.googleapis.com
NAME: datastore.googleapis.com
NAME: deploymentmanager.googleapis.com
NAME: documentai.googleapis.com
NAME: healthcare.googleapis.com
NAME: iam.googleapis.com
NAME: iamcredentials.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: notebooks.googleapis.com
NAME: oslogin.googleapis.com
NAME: pubsub.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com
Updated property [core/project].
Switched back to boot project pdt-tgz
Use the following command to switch to your new project
gcloud config set project kcc-lz-3479
**** Done ****
obriensystems commented 1 year ago

dev user with expected least priv roles

Screen Shot 2022-12-16 at 13 57 34 Screen Shot 2022-12-16 at 14 04 23
obriensystems commented 1 year ago

Cloud build trigger automation for python master app via CSR remote on github remote

Screen Shot 2023-01-08 at 20 26 28
obriensystems commented 1 year ago

Run

michael@cloudshell:~/docproc-old/pubsec-declarative-toolkit/solutions/document-processing/gcloud (docproc-old)$ ./deployment.sh -u pdt -c true -l false -d false
Default USER_EMAIL if set is: mi....ev
USER_EMAIL reset to mi....v
Date: Mon 09 Jan 2023 01:51:33 AM UTC
Timestamp: 1673229093
running with: -b docproc-old -u pdt -c true -l false -d false -e mi....ev -p
Updated property [core/project].
Switched back to boot project docproc-old
Start: 1673229094
unique string: pdt
REGION: us-central1
NETWORK: pdt-pdt-vpc
SUBNET: pdt-pdt-sn
CLUSTER: pdt-pdt
Creating project: docai-gen-6623
CC_PROJECT_ID: docai-gen-6623
passed in KCC_PROJECT_ID:
BOOT_PROJECT_ID: docproc-old
BILLING_ID: 0...
ORG_ID: 5...
Creating KCC project: docai-gen-6623
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/docai-gen-6623].
Waiting for [operations/cp.8658187408847568157] to finish...done.    
Enabling service [cloudapis.googleapis.com] on project [docai-gen-6623]...
Operation "operations/acat.p2-81974734379-1a06f2cf-88ef-4dbc-baf6-580d9ed64d9f" finished successfully.
Updated property [core/project] to [docai-gen-6623].
Updated property [core/project].
billingAccountName: billingAccounts/0...5D
billingEnabled: true
name: projects/docai-gen-6623/billingInfo
projectId: docai-gen-6623
Adding roles to project for user: mi...ev
Updated IAM policy for project [docai-gen-6623].
Updated IAM policy for project [docai-gen-6623].
Enabling APIs
Operation "operations/acat.p2-81974734379-f101eb7e-a0e4-47c2-aafc-aa5dced08792" finished successfully.
Operation "operations/acf.p2-81974734379-6d1de5ca-89ae-4f7e-ba7b-7dd379f6de57" finished successfully.
Operation "operations/acat.p2-81974734379-d280de57-b0b6-449e-9771-adf06835f5b7" finished successfully.
Operation "operations/acf.p2-81974734379-271ececa-b018-49cc-887f-3d59ebf1776d" finished successfully.
Operation "operations/acf.p2-81974734379-ffaf1b82-8747-4c8a-af32-e4e20269c11a" finished successfully.
Operation "operations/acat.p2-81974734379-a10a72d2-cf71-46bf-827a-c5f9840ab157" finished successfully.
Operation "operations/acat.p2-81974734379-4646a690-f8cd-4959-bed3-4951fc9c2e43" finished successfully.
Operation "operations/acf.p2-81974734379-3e62e082-ca39-428d-8066-9d01c2829fcd" finished successfully.
Operation "operations/acf.p2-81974734379-0c498681-e4cb-4551-9dbe-e468627d3dd1" finished successfully.
Operation "operations/acf.p2-81974734379-3de0d800-c292-465a-9d6e-1491fbb8af6a" finished successfully.
Operation "operations/acat.p2-81974734379-84e13435-e2e4-494e-8cd5-9ecc0e86b413" finished successfully.
Operation "operations/acf.p2-81974734379-a5278b81-eaba-4d57-9437-fc4054eace4d" finished successfully.
Operation "operations/acf.p2-81974734379-d1087173-f32d-461e-abf2-c36332d057a6" finished successfully.
Operation "operations/acat.p2-81974734379-b9f15602-13fd-4875-8113-448f63dad594" finished successfully.
Operation "operations/acat.p2-81974734379-20e0539f-23d8-479d-a6f7-ad1fbb4d1479" finished successfully.
Operation "operations/acf.p2-81974734379-a994cbc3-33f1-46de-8756-3fc0cc34b668" finished successfully.
Operation "operations/acat.p2-81974734379-0b38321d-a7d4-4ed9-810f-aef9f2b22806" finished successfully.
Operation "operations/acat.p2-81974734379-e11ecb39-9ae9-464e-999f-ad910f81af93" finished successfully.
Create VPC: pdt-pdt-vpc
Created [https://www.googleapis.com/compute/v1/projects/docai-gen-6623/global/networks/pdt-pdt-vpc].
NAME: pdt-pdt-vpc
SUBNET_MODE: CUSTOM
BGP_ROUTING_MODE: REGIONAL
IPV4_RANGE:
GATEWAY_IPV4:

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

$ gcloud compute firewall-rules create <FIREWALL_NAME> --network pdt-pdt-vpc --allow tcp,udp,icmp --source-ranges <IP_RANGE>
$ gcloud compute firewall-rules create <FIREWALL_NAME> --network pdt-pdt-vpc --allow tcp:22,tcp:3389,icmp

Create subnet pdt-pdt-sn off VPC: pdt-pdt-vpc
Created [https://www.googleapis.com/compute/v1/projects/docai-gen-6623/regions/us-central1/subnetworks/pdt-pdt-sn].
NAME: pdt-pdt-sn
REGION: us-central1
NETWORK: pdt-pdt-vpc
RANGE: 192.168.0.0/16
STACK_TYPE: IPV4_ONLY
IPV6_ACCESS_TYPE:
INTERNAL_IPV6_PREFIX:
EXTERNAL_IPV6_PREFIX:
create backup dir in pr..ud dir
/home/michael/docproc-old/pubsec-declarative-toolkit/solutions/document-processing/gcloud
Create service account: service-account-main@docai-gen-6623.iam.gserviceaccount.com
Created service account [service-account-main].
Email: service-account-main@docai-gen-6623.iam.gserviceaccount.com
Updated IAM policy for project [docai-gen-6623].
Updated IAM policy for project [docai-gen-6623].
Updated IAM policy for project [docai-gen-6623].
Updated IAM policy for project [docai-gen-6623].
Updated IAM policy for project [docai-gen-6623].
Updated IAM policy for project [docai-gen-6623].
Updated IAM policy for project [docai-gen-6623].
Updated IAM policy for project [docai-gen-6623].
Creating CSR
Created [docproc3].
WARNING: You may be billed for this repository. See https://cloud.google.com/source-repositories/docs/pricing for details.
cloning for CSR repo https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
Cloning into 'pubsec-declarative-toolkit'...
remote: Enumerating objects: 6081, done.
remote: Counting objects: 100% (108/108), done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 6081 (delta 46), reused 86 (delta 39), pack-reused 5973
Receiving objects: 100% (6081/6081), 5.09 MiB | 23.60 MiB/s, done.
Resolving deltas: 100% (3825/3825), done.
Enumerating objects: 4471, done.
Counting objects: 100% (4471/4471), done.
Delta compression using up to 4 threads
Compressing objects: 100% (1411/1411), done.
Writing objects: 100% (4471/4471), 1.66 MiB | 9.34 MiB/s, done.
Total 4471 (delta 2759), reused 4470 (delta 2759), pack-reused 0
remote: Resolving deltas: 100% (2759/2759)
To https://source.developers.google.com/p/docai-gen-6623/r/docproc3
 * [new branch]      main -> main
Branch 'canary' set up to track remote branch 'canary' from 'origin'.
Switched to a new branch 'canary'
[canary 18311b1] triple cloud build main
 1 file changed, 31 insertions(+)
 create mode 100644 cloudbuild-prod-main.yaml
Enumerating objects: 276, done.
Counting objects: 100% (268/268), done.
Delta compression using up to 4 threads
Compressing objects: 100% (127/127), done.
Writing objects: 100% (256/256), 61.35 KiB | 20.45 MiB/s, done.
Total 256 (delta 130), reused 246 (delta 122), pack-reused 0
remote: Resolving deltas: 100% (130/130)
To https://source.developers.google.com/p/docai-gen-6623/r/docproc3
 * [new branch]      canary -> canary
create ar
Create request issued for: [docproc3]
Waiting for operation [projects/docai-gen-6623/locations/us-central1/operations/d3833584-41c5-4a78-bcc3-ad126f3db925] to complete...done.     
Created repository [docproc3].
Create Cloud Build - prod
Created [https://cloudbuild.googleapis.com/v1/projects/docai-gen-6623/locations/global/triggers/e5aa7f84-c6f7-441a-a22c-8c74246caacd].
NAME: prod-main
CREATE_TIME: 2023-01-09T01:56:09+00:00
STATUS:
trigger_prod_main_build
Already on 'canary'
Your branch is ahead of 'origin/canary' by 1 commit.
  (use "git push" to publish your local commits)
copy random file empty9193_stub.sh
[canary 9ff52d0] trigger prod with 9193
 1 file changed, 16 insertions(+)
 create mode 100755 empty9193_stub.sh
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 667 bytes | 667.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1)
To https://source.developers.google.com/p/docai-gen-6623/r/docproc3
   18311b1..9ff52d0  canary -> canary
Updated property [core/project].
Switched back to boot project docproc-old
Use the following command to switch to your new project
gcloud config set project docai-gen-6623
**** Done ****
michael@cloudshell:~/docproc-old/pubsec-declarative-toolkit/solutions/document-processing/gcloud (docproc-old)$
obriensystems commented 1 year ago

add bulk resource export to krm

Screen Shot 2023-01-13 at 01 29 35
obriensystems commented 1 year ago

Developer accounts with folder admin and project creation roles use case 1: move project between folders or from organization to folder use case 2: create new sub folder off folder use case 3 variant: cannot create folder off organization uc 4 - locked creating folder or proj off admin-only

After org removal of BAC and ProjectCreator - (moved to super admin account only) Verify - restricted user cannot create folders off org - only off designated root folder - ok "You do not have the required "resourcemanager.folders.create" permission to create folders in this location."

Remaining: restricted user can still create projects anywhere


pre
remove from organization at org root in iam (BillingAccountCreator and ProjectCreator) to one of the org admins

set the org admin with additional FolderAdmin role for the org root
gcloud organizations add-iam-policy-binding 5..5 --member=user:y@landing.systems --role=roles/resourcemanager.folderAdmin

sa creates super folder
gcloud resource-manager folders create --display-name=ch-root --organization=5..
name: 'folders/621660468815'

sa creates project in prep for restricted accounts
ch-restricted-proj1

sa associates restricted accounts as project owner
gcloud projects add-iam-policy-binding ch-restricted-proj1  --member=user:$USER_EMAIL --role=roles/owner

sa adds restricted accounts as owner on super folder
gcloud resource-manager folders add-iam-policy-binding 621660468815 --member=user:restricted@landing.systems --role=roles/resourcemanager.folderAdmin

restricted account creates subfolder
gcloud resource-manager folders create --display-name=ch-root-sub1 --folder=621660468815

 name: 'folders/917483507652'
 parent: 'folders/621660468815'

restricted account creates new projects off subfolder
  gcloud projects create ch-restricted-proj-sub2a --name="restricted-proj-sub2a" --folder 917483507652
  gcloud config set project "${CC_PROJECT_ID}"

restricted user moves projects from org into subfolder

try admin only proj
cloud projects create admin-only2 --name="admin-only2" --folder 694452364990

result
org
+--ch-restricted-proj1 owner
+--ch-root (folder)
      +--ch-root-sub1 (folder)  restricted FolderAadmin
             +--ch-sub-proj1a
fmichaelobrien commented 1 year ago

Thanks Chris - I see I need to switch from beta resource-config to config-connector config-connector bulk-export

https://cloud.google.com/config-connector/docs/how-to/import-export/bulk-export#exporting_an_inventory_with_config-connector

for

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/canary/solutions/document-processing/gcloud/deployment.sh#L588

bulk_resources_export_to_krm() {
    sudo apt-get install google-cloud-sdk-config-connector
    #gcloud services enable cloudasset.googleapis.com
    mkdir ${REPO_TREE_DEPTH_FOR_CD_UP}$RESOURCE_CONFIG_BULK_EXPORT_TO_KRM_SUBDIR
    gcloud beta resource-config bulk-export --path=${REPO_TREE_DEPTH_FOR_CD_UP}$RESOURCE_CONFIG_BULK_EXPORT_TO_KRM_SUBDIR
}