Add GCP Storage/Bucket support

yarcat commented 3 months ago

Previously only S3 buckets export destination was supported. This PR introduces the following changes:

Adds GCP (Google Cloud Platform) storage/bucket destination support.
Adds s3_url and gcp_url to the result object. Note: previous url is also supported for backward compatibility.
Both S3 and GCP configurations are optional. If both buckets are configured, the upload is concurrent.

Testing

Environment

RunPod
5GB container space
/workspace Volume mount
1 x RTX A4000
9 vCPU 50 GB RAM
Datacenter: SE
Image configuration:

Existing extra environment variables
1. WEB_TOKEN=XXX

New environment variables provided to the image

GCP_CREDENTIALS=/workspace/myproj-123456-0123456789abc.json
GCP_PROJECT_ID=myproj-123456
GCP_BUCKET_NAME=sd-output

Workflow request/response

IMPORTANT: Note an empty "gcp": {} which requests GCP upload using default settings. You can pass a project name to ensure it isn't empty and isn't ignored if default env variables are not set.

Request

curl -X 'POST' \
   'https://MYRUNPODID-8188.proxy.runpod.net/ai-dock/api/payload' \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer XXX' \
   -d '{ "input": {
  "handler": "RawWorkflow",
  "gcp": {},
  "modifiers": {},
  "workflow_json": {
    "9": {
      "inputs": {
        "filename_prefix": "ComfyUI",
        "images": [
          "10",
          0
        ]
      },
      "class_type": "SaveImage",
      "_meta": {
        "title": "Save Image"
      }
    },
    "10": {
      "inputs": {
        "image": "https://raw.githubusercontent.com/comfyanonymous/ComfyUI/master/input/example.png",
        "upload": "image"
      },
      "class_type": "LoadImage",
      "_meta": {
        "title": "Load Image"
      }
    }
  }
} }'

Response:

{"id":"cf455cdd-2cb6-400e-a2dd-3a789be429e0","message":"Request accepted","status":"pending","comfyui_response":{},"output":[],"timings":{}}

Validation request/response

Request

curl -X 'GET' \
  'https://MYRUNPODID-8188.proxy.runpod.net/ai-dock/api/result/0e922f75-b65d-4de1-bc40-1b92273b5164' \
  -H 'accept: application/json' -H 'Authorization: Bearer XXX'

Response

{
  "comfyui_response": {
    "meta": {
      "9": {
        "display_node": "9",
        "node_id": "9",
        "parent_node": null,
        "real_node_id": "9"
      }
    },
    "outputs": {
      "9": {
        "images": [
          {
            "filename": "ComfyUI_00023_.png",
            "subfolder": "",
            "type": "output"
          }
        ]
      }
    },
    "prompt": [
      0,
      "affecf1e-f104-46c5-b476-d64b79e7b3e3",
      {
        "10": {
          "_meta": {
            "title": "Load Image"
          },
          "class_type": "LoadImage",
          "inputs": {
            "image": "cc5986222181f3a6f9da38d1b847f965.png",
            "upload": "image"
          }
        },
        "9": {
          "_meta": {
            "title": "Save Image"
          },
          "class_type": "SaveImage",
          "inputs": {
            "filename_prefix": "ComfyUI",
            "images": [
              "10",
              0
            ]
          }
        }
      },
      {
        "client_id": "0e922f75-b65d-4de1-bc40-1b92273b5164"
      },
      [
        "9"
      ]
    ],
    "status": {
      "completed": true,
      "messages": [
        [
          "execution_start",
          {
            "prompt_id": "affecf1e-f104-46c5-b476-d64b79e7b3e3",
            "timestamp": 1723814650281
          }
        ],
        [
          "execution_cached",
          {
            "nodes": [],
            "prompt_id": "affecf1e-f104-46c5-b476-d64b79e7b3e3",
            "timestamp": 1723814650321
          }
        ],
        [
          "execution_success",
          {
            "prompt_id": "affecf1e-f104-46c5-b476-d64b79e7b3e3",
            "timestamp": 1723814650424
          }
        ]
      ],
      "status_str": "success"
    }
  },
  "id": "0e922f75-b65d-4de1-bc40-1b92273b5164",
  "message": "Process complete.",
  "output": [
    {
      "gcp_url": "https://storage.googleapis.com/sd-uploads/0e922f75-b65d-4de1-bc40-1b92273b5164/ComfyUI_00023_.png?X-Goog-Algorithm=GOOG4-RSA-SHA256\u0026X-Goog-Credential=sd-uploader%40aitesting-405910.iam.gserviceaccount.com%2F20240816%2Fauto%2Fstorage%2Fgoog4_request\u0026X-Goog-Date=20240816T132411Z\u0026X-Goog-Expires=604800\u0026X-Goog-SignedHeaders=host\u0026X-Goog-Signature=7cf5ed971765c7ff3250869914ac0b9e47400acfd735646f0669bde5b9ff05fd34486cac5be921bbccf73372e79c38e1d25f973fe7ae6d77fc6c8328a8ba02ce106e33119c748a4023f6b1949aeb170cff10210e8014a0273a220845245442a430dc19b147ad71f3506d4f5ff941e658fe92a33e7edd0dac7b4a5c6deabaa2f027e0e9069db45f9b9d26942faa9db75920665bd793dd5e0264ed556dd11411cadd5377912777fa78caa950a7bf147ecf6da583634007dfbd403278c4a352ae17b77c702ecfcc5314a964cbb3aababfad356fe8b9f657d4b68dc138defea5f977c9853597910bf8cf81332a0b4ebf41f81b168b460c19046bffcae1c23dc85576",
      "local_path": "/opt/ComfyUI/output/0e922f75-b65d-4de1-bc40-1b92273b5164/ComfyUI_00023_.png"
    }
  ],
  "status": "success",
  "timings": {}
}

Error test

Providing unavailable "gcp": { "bucket_name": "abc" }

curl -X 'POST' \
   'https://MYRUNPODID-8188.proxy.runpod.net/ai-dock/api/payload' \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer XXX' \
   -d '{ "input": {
  "handler": "RawWorkflow",
  "gcp": {"bucket_name": "abc"},
  "modifiers": {},
  "workflow_json": {
    "9": {
      "inputs": {
        "filename_prefix": "ComfyUI",
        "images": [
          "10",
          0
        ]
      },
      "class_type": "SaveImage",
      "_meta": {
        "title": "Save Image"
      }
    },
    "10": {
      "inputs": {
        "image": "https://raw.githubusercontent.com/comfyanonymous/ComfyUI/master/input/example.png",
        "upload": "image"
      },
      "class_type": "LoadImage",
      "_meta": {
        "title": "Load Image"
      }
    }
  }
} }'
{"id":"47097118-0e6e-4346-b36f-981d940dff93","message":"Request accepted","status":"pending","comfyui_response":{},"output":[],"timings":{}}%                                                                                                                                                                                curl -X 'GET' \
  'https://MYRUNPODID-8188.proxy.runpod.net/ai-dock/api/result/47097118-0e6e-4346-b36f-981d940dff93' \
  -H 'accept: application/json' -H 'Authorization: Bearer XXX'
{"id":"47097118-0e6e-4346-b36f-981d940dff93","message":"Process complete.","status":"success","comfyui_response":{"prompt":[4,"d59a2ad4-75bd-4a08-968e-6cb580ae5517",{"9":{"inputs":{"filename_prefix":"ComfyUI","images":["10",0]},"class_type":"SaveImage","_meta":{"title":"Save Image"}},"10":{"inputs":{"image":"cc5986222181f3a6f9da38d1b847f965.png","upload":"image"},"class_type":"LoadImage","_meta":{"title":"Load Image"}}},{"client_id":"47097118-0e6e-4346-b36f-981d940dff93"},["9"]],"outputs":{"9":{"images":[{"filename":"ComfyUI_00023_.png","subfolder":"","type":"output"}]}},"status":{"status_str":"success","completed":true,"messages":[["execution_start",{"prompt_id":"d59a2ad4-75bd-4a08-968e-6cb580ae5517","timestamp":1723816625788}],["execution_cached",{"nodes":["9","10"],"prompt_id":"d59a2ad4-75bd-4a08-968e-6cb580ae5517","timestamp":1723816625826}],["execution_success",{"prompt_id":"d59a2ad4-75bd-4a08-968e-6cb580ae5517","timestamp":1723816625827}]]},"meta":{"9":{"node_id":"9","display_node":"9","parent_node":null,"real_node_id":"9"}}},"output":[{"local_path":"/opt/ComfyUI/output/47097118-0e6e-4346-b36f-981d940dff93/ComfyUI_00023_.png"}],"timings":{}}

robballantyne commented 3 months ago

Hi @yarcat thanks for your work on this.

I will test for S3 when you've confirmed it working for GCP - This seems a really useful addition and I'd quite like this API feature finished as soon as possible.

yarcat commented 3 months ago

@robballantyne I've updated the PR with the test information performed. And I've uploaded a credentials initialization fix. Please verify S3.

ai-dock / comfyui