cshum / imagor

Fast, secure image processing server and Go library, using libvips
Apache License 2.0
3.39k stars 135 forks source link

GCLOUD results are not being stored on the result bucket #498

Open Freakazoid182 opened 4 days ago

Freakazoid182 commented 4 days ago

This could be a configuration error or misunderstanding on my side, but I cannot seem to find where the issue lies.

Running Imagor version 1.4.14 on Kubernetes.

I'm expecting the raw image to be loaded from the loader bucket, and processed images to be stored on the result bucket in the images/result folder, but I don't see this happening. Also I'm not seeing any errors or logs indicating that this is being attempted.

These are the settings I'm using. Am I missing something?

env:
    - name: IMAGOR_SECRET
      valueFrom:
        secretKeyRef:
          key: ImagorSecret
          name: imagor-secret
    - name: IMAGOR_SIGNER_TYPE
      value: sha256
    - name: IMAGOR_SIGNER_TRUNCATE
      value: "40"
    - name: PORT
      value: "8000"
    - name: GCLOUD_LOADER_BUCKET
      value: my-gcp-bucket
    - name: GCLOUD_LOADER_BASE_DIR
      value: images
    - name: GCLOUD_RESULT_STORAGE_BUCKET
      value: my-gcp-bucket
    - name: GCLOUD_RESULT_STORAGE_BASE_DIR
      value: images/result
    - name: GCLOUD_RESULT_STORAGE_ACL
      value: publicRead
    - name: SERVER_ACCESS_LOG
      value: "true"
    - name: DEBUG
      value: "true"
    - name: HTTP_LOADER_DISABLE
      value: "true"

These are the Imagor server logs

DEBUG    imagor/imagor.go:663    imagor    {"version": "1.4.14", "unsafe": false, "request_timeout": "30s", "load_timeout": "20s", "process_timeout": "20s", "save_timeout": "20s", "process_concurrency": 0, "cache_header_ttl": "168h0m0s", "loaders": ["GCloudStorage"], "storages": ], "result_storages": ["GCloudStorage"], "processors": ["Processor"]}
INFO    vips/processor.go:108    VIPS    {"log": "found /usr/local/lib/vips-modules-8.15"}
INFO    vips/processor.go:108    VIPS    {"log": "using configure-time prefix"}
INFO    vips/processor.go:108    VIPS    {"log": "VIPS_PREFIX = /usr/local"}
INFO    vips/processor.go:108    VIPS    {"log": "VIPS_LIBDIR = /usr/local/lib"}
INFO    vips/processor.go:108    VIPS    {"log": "prefix = /usr/local"}
INFO    vips/processor.go:108    VIPS    {"log": "libdir = /usr/local/lib"}
INFO    vips/processor.go:108    VIPS    {"log": "searching \"/usr/local/lib/vips-modules-8.15\""}
INFO    vips/processor.go:108    VIPS    {"log": "loading \"/usr/local/lib/vips-modules-8.15/vips-poppler.so\""}
INFO    vips/processor.go:108    VIPS    {"log": "loading \"/usr/local/lib/vips-modules-8.15/vips-heif.so\""}
INFO    vips/processor.go:108    VIPS    {"log": "loading \"/usr/local/lib/vips-modules-8.15/vips-openslide.so\""}
INFO    vips/processor.go:108    VIPS    {"log": "searching \"/usr/local/lib/vips-plugins-8.15\""}
INFO    vips/processor.go:108    VIPS    {"log": "im_load_plugins: searching \"/usr/local/lib/vips-8.15\"\n"}
INFO    vips/processor.go:108    VIPS    {"log": "im_load_plugins: searching \"/usr/local/lib\"\n"}
INFO    vips/processor.go:108    vips    {"log": "vips 8.15.3 started with concurrency=1 cache_max_files=0 cache_max_mem=0 cache_max=0"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=jp2k load=true save=true"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=gif load=true save=true"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=heif load=true save=true"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=pdf load=true save=false"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=png load=true save=true"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=svg load=true save=false"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=tiff load=true save=true"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=webp load=true save=true"}
INFO    vips/processor.go:108    vips    {"log": "registered image type=jpeg load=true save=true"}
INFO    server/server.go:113    listen    {"addr": ":8000"}
DEBUG    imagor/imagor.go:610    suppress    {"key": "fit-in/200x200/image-uploads/image1.jpeg"}
INFO    vips/processor.go:108    VIPS    {"log": "thumbnailing source"}
INFO    vips/processor.go:108    VIPS    {"log": "selected loader is VipsForeignLoadJpegSource"}
INFO    vips/processor.go:108    VIPS    {"log": "input size is 1920 x 1080"}
INFO    vips/processor.go:108    VIPS    {"log": "loading with factor 4 pre-shrink"}
INFO    vips/processor.go:108    VIPS    {"log": "pre-shrunk size is 480 x 270"}
INFO    vips/processor.go:108    VIPS    {"log": "converting to processing space srgb"}
INFO    vips/processor.go:108    VIPS    {"log": "residual reducev by 0.416667"}
INFO    vips/processor.go:108    VIPS    {"log": "reducev: 15 point mask"}
INFO    vips/processor.go:108    VIPS    {"log": "reducev sequential line cache"}
INFO    vips/processor.go:108    VIPS    {"log": "residual reduceh by 0.416667"}
INFO    vips/processor.go:108    VIPS    {"log": "reduceh: 15 point mask"}
DEBUG    vips/processor.go:106    vips    {"log": "created imageRef 0xc000635d80"}
DEBUG    vips/process.go:249    image    {"width": 200, "height": 113, "page_height": 113}
INFO    vips/processor.go:108    VIPS    {"log": "threadpool completed with 1 workers"}
DEBUG    vips/processor.go:106    vips    {"log": "closing image 0xc000635d80"}
DEBUG    vips/processor.go:106    vips    {"log": "closing source 0xc00091e900"}
DEBUG    imagor/imagor.go:405    processed    {"params": {"path":"fit-in/200x200/image-uploads/image1.jpeg","image":"image-uploads/image1.jpeg","hash":"YDMgda3UV8-SSV4kfo2tbYYC5ExAIqBqH5QbTmLW","fit_in":true,"width":200,"height":200}}
INFO    server/handler.go:102    access    {"status": 200, "method": "GET", "uri": "/YDMgda3UV8-SSV4kfo2tbYYC5ExAIqBqH5QbTmLW/fit-in/200x200/image-uploads/image1.jpeg", "ip": "185.16.23.237", "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0", "took": "337.335225ms"}
DEBUG    imagor/imagor.go:565    saved    {"key": "fit-in/200x200/image-uploads/image1.jpeg"}
Freakazoid182 commented 3 days ago

Managed to find the cause of the issue myself.

The problem was that I was setting GCLOUD_RESULT_STORAGE_ACL to publicRead, while the GCP service account I was using only had the Storage Object User role. This role isn't allowed setting the object ACL.

For me the solution was simply not setting the environment variable, as I actually don't want ACL configured on the objects.

It would be nice still to see if error handling can be improved on this part, as there was no indication why the object wasn't being stored.