s3 store - Githubissues

Hello,

I have been trying to follow all your steps in order to run this in a lambda but it has been impossible.

I am using s3 as disk as part of the config file: cerbos-aws-lambda\conf.default.yml

storage:
  driver: "blob"
  blob:
    # This section is required only if storage.driver is blob.
    bucket: "s3://cerbos-test-pdp?region=us-west-2" 
    downloadTimeout: 30s # DownloadTimeout specifies the timeout for downloading from cloud storage.
    prefix: policies # Prefix specifies a subdirectory to download.
    requestTimeout: 10s # RequestTimeout specifies the timeout for an HTTP request.
    updatePollInterval: 15s # UpdatePollInterval specifies the interval to poll the cloud storage. Set to 0 to disable.

And just in case, I added the s3 config as part of the sam.yaml file:

Variables:
          BUCKET_URL: "s3://cerbos-test-pdp?region=us-west-2"
          BUCKET_PREFIX: "policies"
          CERBOS_LOG_LEVEL: DEBUG

and the set of permissions I attached to the lambda running the cerbos image:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*",
                "s3-object-lambda:*"
            ],
            "Resource": "arn:aws:s3:::cerbos-test-pdp/*"
        }
    ]
}

ERROR: failed to create store: unknown storage driver [blob]

Please, any help is appreciated. This is driving me nuts.

thanks.

Hmm... this is very odd because the error you're getting suggests that the Cerbos binary is not correctly compiled. Since the Makefile downloads the latest release from GitHub, that can't be the case 🤔

What were the exact steps you followed? Are the contents of conf.default.yaml exactly as you posted above?

hello @charithe ,

I checked the version the Makefile is using and this is what I found: "name": "v0.25.0"

Is it the latest one? I think it is based on this: https://github.com/cerbos/cerbos/releases?q=latest&expanded=true

Actually, I found an issue with the regepx used to find the most recent release, the flags used on it didn't work on my ubuntu installation:

not working the \d part: CURRENT_RELEASE=$$(curl -sH "Accept: application/vnd.github.v3+json" https://api.github.com/repos/cerbos/cerbos/tags | grep -o -E '"name": "v\d+\.\d+.\d+"' | head -1); \
I had to replaced them with the more general [0-9]: CURRENT_RELEASE=$$(curl -sH "Accept: application/vnd.github.v3+json" https://api.github.com/repos/cerbos/cerbos/tags | grep -o -E '"name": "v[0-9]+\.[0-9]+.[0-9]+"' | head -1); \

I have been trying 2 approachs:

Adding the policies files as part of the image, using disk storage
Using an s3 bucket to store the policies.

File structure:

1. Policies as part of the docker image:

conf.default.yml file:

# https://docs.cerbos.dev/cerbos/latest/configuration/auxdata.html
# WARNING: Disabling JWT verification is not recommended because it makes the system
# insecure by forcing Cerbos to evaluate policies using potentially tampered data.
auxData:
  jwt:
    disableVerification: true

storage:
  driver: "disk"
  disk:
    directory: /policies
    watchForChanges: true

Dockerfile:

FROM golang:latest as build

WORKDIR /src

COPY gateway ./gateway
COPY go.mod go.sum main.go ./

RUN go get -d -v ./...
RUN go build -ldflags '-s -w' -o /gw main.go
RUN chmod +x /gw

# Now copy it into our base image.
FROM gcr.io/distroless/base
ARG ARCH=x86_64
COPY --from=build /gw /
COPY .cerbos/Linux_${ARCH}/cerbos /
COPY conf.default.yml /conf.yml
COPY ./policies /policies/

# Uncomment for testing with a disk storage
# define REMOTE_CERBOS_URL pointing to HTTP API (see CerbosServerFunctionAPI in sam.yml)
# then run `make test`
#COPY test/testdata/store /store
#COPY test/testdata/conf.yml test/testdata/verify_key.jwk /
ENTRYPOINT ["/gw"]

2. Policies stored in a s3 bucket:

conf.default.yml file:

# https://docs.cerbos.dev/cerbos/latest/configuration/auxdata.html
# WARNING: Disabling JWT verification is not recommended because it makes the system
# insecure by forcing Cerbos to evaluate policies using potentially tampered data.
auxData:
  jwt:
    disableVerification: true

storage:
  driver: "blob"
  blob:
    # This section is required only if storage.driver is blob.
    bucket: "s3://cerbos-test-pdp?region=us-west-2" # "s3://my-bucket-name?region=us-east-2" Required. Bucket URL (Examples: s3://my-bucket?region=us-west-1 gs://my-bucket azblob://my-container).
    downloadTimeout: 30s # DownloadTimeout specifies the timeout for downloading from cloud storage.
    prefix: policies # Prefix specifies a subdirectory to download.
    requestTimeout: 10s # RequestTimeout specifies the timeout for an HTTP request.
    updatePollInterval: 15s # UpdatePollInterval specifies the interval to poll the cloud storage. Set to 0 to disable.
    workDir: /tmp/workDir

Dockerfile:

FROM golang:latest as build

WORKDIR /src

COPY gateway ./gateway
COPY go.mod go.sum main.go ./

RUN go get -d -v ./...
RUN go build -ldflags '-s -w' -o /gw main.go
RUN chmod +x /gw

# Now copy it into our base image.
FROM gcr.io/distroless/base
ARG ARCH=x86_64
COPY --from=build /gw /
COPY .cerbos/Linux_${ARCH}/cerbos /
COPY conf.default.yml /conf.yml

# Uncomment for testing with a disk storage
# define REMOTE_CERBOS_URL pointing to HTTP API (see CerbosServerFunctionAPI in sam.yml)
# then run `make test`
#COPY test/testdata/store /store
#COPY test/testdata/conf.yml test/testdata/verify_key.jwk /
ENTRYPOINT ["/gw"]

thanks for your help @charithe

@luisgallon Can you undo any changes you made to the repo (git reset --hard) or clone a fresh copy and run:

CERBOS_RELEASE=0.25.0 make image

Then, run this command:

docker run -it -e BUCKET_URL=s3://x -e BUCKET_PREFIX=x --entrypoint=/cerbos cerbos/aws-lambda-gateway:latest server --config=/conf.yml --log-level=debug

It will fail because the bucket URL is a dummy one. But, I am only interested in looking at the log output it produces. Please post that here.

sir, I did as you said and this is the log:

$ CERBOS_RELEASE=0.25.0 make image
Downloading Cerbos binary for Linux
[+] Building 0.8s (17/17) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                   0.0s
 => => transferring dockerfile: 38B                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                        0.0s
 => [internal] load metadata for gcr.io/distroless/base:latest                                                                                         0.2s
 => [internal] load metadata for docker.io/library/golang:latest                                                                                       0.7s
 => [internal] load build context                                                                                                                      0.0s
 => => transferring context: 424B                                                                                                                      0.0s
 => [build 1/7] FROM docker.io/library/golang:latest@sha256:2edf6aab2d57644f3fe7407132a0d1770846867465a39c2083770cf62734b05d                           0.0s
 => [stage-1 1/4] FROM gcr.io/distroless/base@sha256:ccaef5ee2f1850270d453fdf700a5392534f8d1a8ca2acda391fbb6a06b81c86                                  0.0s
 => CACHED [build 2/7] WORKDIR /src                                                                                                                    0.0s
 => CACHED [build 3/7] COPY gateway ./gateway                                                                                                          0.0s
 => CACHED [build 4/7] COPY go.mod go.sum main.go ./                                                                                                   0.0s
 => CACHED [build 5/7] RUN go get -d -v ./...                                                                                                          0.0s
 => CACHED [build 6/7] RUN go build -ldflags '-s -w' -o /gw main.go                                                                                    0.0s
 => CACHED [build 7/7] RUN chmod +x /gw                                                                                                                0.0s
 => CACHED [stage-1 2/4] COPY --from=build /gw /                                                                                                       0.0s
 => CACHED [stage-1 3/4] COPY .cerbos/Linux_x86_64/cerbos /                                                                                            0.0s
 => CACHED [stage-1 4/4] COPY conf.default.yml /conf.yml                                                                                               0.0s
 => exporting to image                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                0.0s
 => => writing image sha256:ef04e2ffa9425e7503e5d213d6d84731a0034ebf688d2adb55f1c11b6d8960e7                                                           0.0s
 => => naming to docker.io/cerbos/aws-lambda-gateway

docker run -it -e BUCKET_URL=s3://x -e BUCKET_PREFIX=x --entrypoint=/cerbos cerbos/aws-lambda-gateway:latest server --config=/conf.yml --log-level=debug
2023-02-27T18:09:51.310Z        INFO    cerbos.server   maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
2023-02-27T18:09:51.310Z        INFO    cerbos.server   Loading configuration from /conf.yml
2023-02-27T18:09:51.310Z        DEBUG   cerbos.blob.cloner      Checkout dir contains (0) files
2023-02-27T18:09:51.310Z        ERROR   cerbos.blob.cloner      Failed to get next item {"error": "blob (code=Unknown): MissingRegion: could not find region configuration", "errorVerbose": "blob (code=Unknown):\n    gocloud.dev/blob.(*ListIterator).Next\n        gocloud.dev@v0.28.0/blob/blob.go:529\n  - MissingRegion: could not find region configuration"}
2023-02-27T18:09:51.310Z        ERROR   cerbos.blob     Failed to clone blob store      {"bucket": "s3://x", "workDir": "/tmp/workDir", "error": "failed to get next object in the bucket: blob (code=Unknown): MissingRegion: could not find region configuration"}
2023-02-27T18:09:51.310Z        ERROR   cerbos.blob     Failed to initialize blob store {"bucket": "s3://x", "workDir": "/tmp/workDir", "error": "failed to get next object in the bucket: blob (code=Unknown): MissingRegion: could not find region configuration"}
2023-02-27T18:09:51.310Z        INFO    cerbos.server   maxprocs: No GOMAXPROCS change to reset
cerbos: error: failed to create store: failed to get next object in the bucket: blob (code=Unknown): MissingRegion: could not find region configuration

OK, so it looks like you have a valid cerbos binary inside the container image now (it is not complaining that blob is an unknown storage driver). Have you tried deploying that image to Lambda?

I just tried and I got these logs from aws lambda:

2023/02/27 18:57:59 cerbos health check: Get "http://127.0.0.1:3592/_cerbos/health": dial tcp 127.0.0.1:3592: connect: connection refused, pid: 13
2023/02/27 18:57:59 Failed to start a process: in 2.026521198s: timeout exceeded starting Cerbos
2023/02/27 18:58:00 cerbos launch timeout: 2s
2023/02/27 18:58:00 health check interval: 50ms
2023/02/27 18:58:02 cerbos health check: Get "http://127.0.0.1:3592/_cerbos/health": dial tcp 127.0.0.1:3592: connect: connection refused, pid: 13
2023/02/27 18:58:02 Failed to start a process: in 2.021239241s: timeout exceeded starting Cerbos
START RequestId: 6d640f2d-c7f8-4ea6-92ee-171c8b4ba12d Version: $LATEST
RequestId: 6d640f2d-c7f8-4ea6-92ee-171c8b4ba12d Error: Runtime exited without providing a reason
Runtime.ExitError
END RequestId: 6d640f2d-c7f8-4ea6-92ee-171c8b4ba12d
REPORT RequestId: 6d640f2d-c7f8-4ea6-92ee-171c8b4ba12d  Duration: 2025.94 ms    Billed Duration: 2026 ms    Memory Size: 1024 MB    Max Memory Used: 23 MB  
2023/02/27 18:58:02 cerbos launch timeout: 2s
2023/02/27 18:58:02 health check interval: 50ms
2023/02/27 18:58:02 cerbos health check: Get "http://127.0.0.1:3592/_cerbos/health": dial tcp 127.0.0.1:3592: connect: connection refused, pid: 13
2023/02/27 18:58:04 cerbos health check: Get "http://127.0.0.1:3592/_cerbos/health": dial tcp 127.0.0.1:3592: connect: connection refused, pid: 13
2023/02/27 18:58:04 Failed to start a process: in 2.031665674s: timeout exceeded starting Cerbos

these are the env variables:

`conf.default.yml` file:

---
# https://docs.cerbos.dev/cerbos/latest/configuration/auxdata.html
# WARNING: Disabling JWT verification is not recommended because it makes the system
# insecure by forcing Cerbos to evaluate policies using potentially tampered data.
auxData:
  jwt:
    disableVerification: true

storage:
  driver: "blob"
  blob:
    bucket: "s3://cerbos-test-pdp?region=us-west-2" # "s3://my-bucket-name?region=us-east-2" Required. Bucket URL (Examples: s3://my-bucket?region=us-west-1 gs://my-bucket azblob://my-container).
    prefix: policies # Prefix specifies a subdirectory to download.
    downloadTimeout: 30s # DownloadTimeout specifies the timeout for downloading from cloud storage.
    requestTimeout: 10s # RequestTimeout specifies the timeout for an HTTP request.
    workDir: ${HOME}/tmp/cerbos/work # WorkDir is the local path to check out policies to.
    updatePollInterval: 15s # UpdatePollInterval specifies the interval to poll the cloud storage. Set to 0 to disable.

s3 bucket including the policies:

I have a question about this config: workDir: ${HOME}/tmp/cerbos/work # WorkDir is the local path to check out policies to.

I am following the default config, is it ok or should I use something different?

I have a question about this config: workDir: ${HOME}/tmp/cerbos/work # WorkDir is the local path to check out policies to.

AFAIK, a lambda can only write to /tmp, other directories are read-only.

Edit: cerbos-aws-lambda default config suggests /tmp/workDir value for workDir.

well, this is what I found in the documentation:

.Configuration keys
* `bucket`: Required. A URL specifying the service (e.g. S3, GCS), the storage bucket and any other configuration parameters required by the provider.
** AWS S3: `s3://my-bucket?region=us-west-1`. Must specify region in the URL.
** Google Cloud Storage: `gs://my-bucket`
** S3-compatible (e.g. Minio): `s3://my-bucket?endpoint=my.minio.local:8080&disableSSL=true&s3ForcePathStyle=true&region=local`. Must specify region in the URL.
* `prefix`: Optional. Look for policies only under this key prefix.
* `workDir`: Optional. Path to the local directory to download the policies to. Defaults to the system cache directory if not specified.
* `updatePollInterval`: Optional. How frequently the blob store should be checked to discover new or updated policies. Defaults to 0 -- which disables polling.
* `requestTimeout`: Optional. HTTP request timeout. It takes an HTTP request to download a policy file. Defaults to 5s.
* `downloadTimeout`: Optional. Timeout to download all policies from the the storage provider. Must be greater than the `requestTimeout`. Defaults to 60s.

So I guess I can skip the workDir config.

I have been trying 2 approaches:

Adding the policies files as part of the image, using disk storage

Using an s3 bucket to store the policies.

Did the first approach work for you? Sorry, if I missed that.

cerbos / cerbos-aws-lambda

s3 store #5

File structure:

1. Policies as part of the docker image:

conf.default.yml file:

Dockerfile:

2. Policies stored in a s3 bucket:

conf.default.yml file:

Dockerfile:

these are the env variables:

`conf.default.yml` file:

s3 bucket including the policies: