search

hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.31k stars 1.73k forks source link

Add ability to upload from local directory for cloud functions #3085

Open kardiff18 opened 5 years ago

kardiff18 commented 5 years ago

Community Note

Description

Cloud Functions lets you deploy via local directory, but currently terraform only supports uploads from GCS or Cloud Source Repos

New or Affected Resource(s)

References

b/274818624

paddycarver commented 5 years ago

I'm not seeing any functionality for direct upload in the API. The closest I see is https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions/generateUploadUrl, which just uploads to cloud storage and then you use that URL as the upload URL when creating the function.

Can you link me to documentation for what you're describing, or can you describe what this would enable that you cannot currently do?

kardiff18 commented 5 years ago

I currently do it via this quickstart https://cloud.google.com/functions/docs/quickstart#deploy_a_function, you just have to do it within the directory. It then supports local upload (and will let you upload multiple files at once.)

On Tue, Feb 19, 2019 at 6:20 PM Paddy notifications@github.com wrote:

I'm not seeing any functionality for direct upload in the API. The closest I see is https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions/generateUploadUrl, which just uploads to cloud storage and then you use that URL as the upload URL when creating the function.

Can you link me to documentation for what you're describing, or can you describe what this would enable that you cannot currently do?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/terraform-providers/terraform-provider-google/issues/3085#issuecomment-465352577, or mute the thread https://github.com/notifications/unsubscribe-auth/AVfYL7meEqHCceBTst73RYO5JcvGHSObks5vPIa5gaJpZM4bD9ie .

--

Kaitlin Ardiff

kardiff@google.com

Strategic Cloud Engineer

paddycarver commented 5 years ago

Can you run the relevant gcloud command and add --log-http with it, and then share the output that's written to stderr here?

jessefoster commented 5 years ago

@paddycarver The gcloud cli does use generateUploadUrl API underneath. I included the http log, minus some secrets and polling.

gcloud beta functions deploy BatchRecord --log-http --trigger-http --memory 128MB --runtime go111 --region us-east1 --env-vars-file=.env.yaml --timeout=540
=======================
==== request start ====
uri: https://www.googleapis.com/oauth2/v4/token
method: POST
== headers start ==
content-type: application/x-www-form-urlencoded
user-agent: google-cloud-sdk gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/e7fe42280b5040b0a1cbe1f912008cf6 environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
== headers end ==
== body start ==
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this request.
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
cache-control: private
content-length: 1330
content-type: application/json; charset=utf-8
date: Wed, 20 Mar 2019 21:25:53 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this response.
-- body end --
total round trip time (request+response): 0.198 secs
---- response end ----
----------------------
=======================

==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/.../locations/us-east1/functions/BatchRecord?alt=json
method: GET
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 0
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/3c497f20696643178a7645b965165dba environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
== headers end ==
== body start ==
== body end ==
==== request end ====

---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
cache-control: private
content-length: 1836
content-location: https://cloudfunctions.googleapis.com/v1/projects/.../locations/us-east1/functions/BatchRecord?alt=json
content-type: application/json; charset=UTF-8
date: Wed, 20 Mar 2019 21:25:54 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
  "name": "projects/.../locations/us-east1/functions/BatchRecord",
  "httpsTrigger": {
    "url": "https://us-east1-....cloudfunctions.net/BatchRecord"
  },
  "status": "ACTIVE",
  "entryPoint": "BatchRecord",
  "timeout": "540s",
  "availableMemoryMb": 128,
  "updateTime": "2019-03-06T20:47:51Z",
  "versionId": "1",
  "labels": {
    "deployment-tool": "cli-gcloud"
  },
  "sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/6308bfa8-153c-438b-8953-e0816e1ea423.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1551906830&Signature=B%2FCGQ2VyGKKrTQ%2BdY%2BnRwW8OO%2FFPL6byy44%2FuBN5QfYsFk8u1k8ezUwJA9FcHlLgFJINzlGcsjLNF7zKNpviWJnWaVlcQ%2BV0WwxyDPD3jkzMeCYnNzObe%2FOzWIMmw9s2lfmUi6Z1%2Fadj1qHmS8%2FSo6jv3VOESLP%2BXvG%2FPWaUjR%2Fh1uNHhGvPaplg7EyXiuIDRi61nNNcsW3Vdj88XZYnifBIVjBz9vIidb52r2Y3RUBWk%2FgzWXWVP11E7Mn%2BHS5D2gSrkXCr8SnRzUhVXEMhj2rDWChA0TnC4VIUPSvW5aoQ7AEdHKhv%2FH8g8a1cUuiFhd4XuBcFb6sYGcXsNuqxfQ%3D%3D",
  "environmentVariables": {
    ...
  },
  "runtime": "go111"
}

-- body end --
total round trip time (request+response): 0.487 secs
---- response end ----
----------------------
=======================

==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/.../locations/us-east1/functions:generateUploadUrl?alt=json
method: POST
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 2
content-type: application/json
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/8a79e2c03c9f4b398fda1418102c0e06 environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
== headers end ==
== body start ==
{}
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
cache-control: private
content-length: 626
content-type: application/json; charset=UTF-8
date: Wed, 20 Mar 2019 21:25:54 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
  "uploadUrl": "https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/fe8f3095-39f8-405d-af51-afb1f36cff2c.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1553118954&Signature=Slj9Ci2C1IMEntxXK44uQNRegAwVsdbYvG0e1ptLsgqNxBEfgr9gH60YZcTxv8YrUkldXa2nzLeKPVd8tKpmG6KmArhCXPKT3upoezvOAyQs2ENbsOwA2r3VhUEaEaMsREv8oR1urNvKjna8RNo%2Bt%2FPNgWw24QM%2BADaliU1jVawele9xH4YS2%2Bpf%2BJL4ObaIW4NOgWpc2Hgl4jJRa6YQJXO2b%2Flz05S1UhTbd6grm%2BqJU3wdNjv6JHVILXhfjca94MJz7SXgql36YSkFej%2BdPuF8%2FdQKkBFjAgmkJTuy3C1QoDE1lH5v09e%2BGBxXWTgT%2F5lxDQhhzqVYf0oapCNb5Q%3D%3D"
}

-- body end --
total round trip time (request+response): 0.534 secs
---- response end ----
----------------------

==== request start ====
uri: https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/fe8f3095-39f8-405d-af51-afb1f36cff2c.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1553118954&Signature=Slj9Ci2C1IMEntxXK44uQNRegAwVsdbYvG0e1ptLsgqNxBEfgr9gH60YZcTxv8YrUkldXa2nzLeKPVd8tKpmG6KmArhCXPKT3upoezvOAyQs2ENbsOwA2r3VhUEaEaMsREv8oR1urNvKjna8RNo%2Bt%2FPNgWw24QM%2BADaliU1jVawele9xH4YS2%2Bpf%2BJL4ObaIW4NOgWpc2Hgl4jJRa6YQJXO2b%2Flz05S1UhTbd6grm%2BqJU3wdNjv6JHVILXhfjca94MJz7SXgql36YSkFej%2BdPuF8%2FdQKkBFjAgmkJTuy3C1QoDE1lH5v09e%2BGBxXWTgT%2F5lxDQhhzqVYf0oapCNb5Q%3D%3D
method: PUT
== headers start ==
Content-Length: 191836
content-length: 191836
content-type: application/zip
user-agent: gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/8818819fd74f4b86b793b4ab079573ec environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
x-goog-content-length-range: 0,104857600
== headers end ==
== body start ==
== body end ==
==== request end ====
---- response start ----
-- headers start --
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
content-length: 0
content-type: text/html; charset=UTF-8
date: Wed, 20 Mar 2019 21:25:55 GMT
etag: "3b85385d5de7a3534bf164465ff2e58a"
server: UploadServer
status: 200
vary: Origin
x-goog-generation: 1553117155199029
x-goog-hash: crc32c=gRZIaQ==, md5=O4U4XV3no1NL8WRGX/Llig==
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 191836
x-guploader-uploadid: AEnB2UrgLdMA0KdEgMEdFPK1ax9D5IsbMdVVzQ4FDq6DROEhO25l263wLH3oOFlThUEAIrJoMXEEyEVuJTWArRzN6d3aPVnmdw
-- headers end --
-- body start --
....
-- body end --
total round trip time (request+response): 0.343 secs
---- response end ----
----------------------
=======================

==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/.../locations/us-east1/functions/BatchRecord?alt=json&updateMask=availableMemoryMb%2CeventTrigger%2ChttpsTrigger%2Cruntime%2CsourceUploadUrl%2Ctimeout
method: PATCH
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 1669
content-type: application/json
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/236.0.0 command/gcloud.beta.functions.deploy invocation-id/cb2831c92bd44bdfb1db8fa7e2b45f57 environment/None environment-version/None interactive/True from-script/False python/2.7.10 term/xterm-256color (Macintosh; Intel Mac OS X 17.7.0)
== headers end ==
== body start ==
{"availableMemoryMb": 128, "entryPoint": "BatchRecord", "environmentVariables": {}, "httpsTrigger": {}, "labels": {"deployment-tool": "cli-gcloud"}, "name": "projects/.../locations/us-east1/functions/BatchRecord", "runtime": "go111", "serviceAccountEmail": "", "sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/fe8f3095-39f8-405d-af51-afb1f36cff2c.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1553118954&Signature=Slj9Ci2C1IMEntxXK44uQNRegAwVsdbYvG0e1ptLsgqNxBEfgr9gH60YZcTxv8YrUkldXa2nzLeKPVd8tKpmG6KmArhCXPKT3upoezvOAyQs2ENbsOwA2r3VhUEaEaMsREv8oR1urNvKjna8RNo%2Bt%2FPNgWw24QM%2BADaliU1jVawele9xH4YS2%2Bpf%2BJL4ObaIW4NOgWpc2Hgl4jJRa6YQJXO2b%2Flz05S1UhTbd6grm%2BqJU3wdNjv6JHVILXhfjca94MJz7SXgql36YSkFej%2BdPuF8%2FdQKkBFjAgmkJTuy3C1QoDE1lH5v09e%2BGBxXWTgT%2F5lxDQhhzqVYf0oapCNb5Q%3D%3D", "status": "ACTIVE", "timeout": "540s", "updateTime": "2019-03-06T20:47:51Z", "versionId": "1"}
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
cache-control: private
content-length: 2324
content-type: application/json; charset=UTF-8
date: Wed, 20 Mar 2019 21:25:55 GMT
server: ESF
status: 200
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
-- headers end --
-- body start --
{
  "name": "operations/ZG9tYWluLW5hbWUtc3lzdGVtLTIyMTcyMS91cy1lYXN0MS9CYXRjaFJlY29yZC9LcUdqdlM0RmU5SQ",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.functions.v1.OperationMetadataV1",
    "target": "projects/.../locations/us-east1/functions/BatchRecord",
    "type": "UPDATE_FUNCTION",
    "request": {
      "@type": "type.googleapis.com/google.cloud.functions.v1.CloudFunction",
      "name": "projects/.../locations/us-east1/functions/BatchRecord",
      "httpsTrigger": {},
      "status": "ACTIVE",
      "entryPoint": "BatchRecord",
      "timeout": "540s",
      "availableMemoryMb": 128,
      "updateTime": "2019-03-06T20:47:51Z",
      "versionId": "1",
      "labels": {
        "deployment-tool": "cli-gcloud"
      },
      "sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-us-east1-3141732f-f672-4768-81db-16c9bab9ab51/fe8f3095-39f8-405d-af51-afb1f36cff2c.zip?GoogleAccessId=service-452490176416@gcf-admin-robot.iam.gserviceaccount.com&Expires=1553118954&Signature=Slj9Ci2C1IMEntxXK44uQNRegAwVsdbYvG0e1ptLsgqNxBEfgr9gH60YZcTxv8YrUkldXa2nzLeKPVd8tKpmG6KmArhCXPKT3upoezvOAyQs2ENbsOwA2r3VhUEaEaMsREv8oR1urNvKjna8RNo%2Bt%2FPNgWw24QM%2BADaliU1jVawele9xH4YS2%2Bpf%2BJL4ObaIW4NOgWpc2Hgl4jJRa6YQJXO2b%2Flz05S1UhTbd6grm%2BqJU3wdNjv6JHVILXhfjca94MJz7SXgql36YSkFej%2BdPuF8%2FdQKkBFjAgmkJTuy3C1QoDE1lH5v09e%2BGBxXWTgT%2F5lxDQhhzqVYf0oapCNb5Q%3D%3D",
      "environmentVariables": {
        ...
      },
      "runtime": "go111"
    },
    "versionId": "2",
    "updateTime": "2019-03-20T21:25:55Z"
  }
}

-- body end --
total round trip time (request+response): 0.676 secs
---- response end ----
michaelbannister commented 4 years ago

I've been using Terraform's archive_file resource to achieve this but I'm now having trouble because GCF's python38 build process runs Python's compileall module, which has this bug/limitation https://bugs.python.org/issue34990 which is triggered by the 2049 modification date that archive_file chooses for the files in the zip.

https://github.com/hashicorp/terraform-provider-archive/issues/40 is a related issue on the archive_file provider; a change there could also provide a solution.

jdiazdiv commented 3 years ago

Any updates about this issue? I may try to fix it.

rileykarson commented 3 years ago

No updates- as highlighted above, it's possible to perform this workflow already using archive_file and the google_storage_bucket_object resource. The documentation examples showcase this using a preexisting zip, but interpolating an archive_file based one should be simple.

Closing this issue out would mean an improvement to be able to do so entirely in band within the resource, which is not currently planned work for the provider development team.

melinath commented 3 months ago

Note from triage: Moving this out of the backlog since it could be reasonable to implement if the product team determines it is a priority.