gcloud preview app deploy fails

GoogleCodeExporter commented 8 years ago

What steps will reproduce the problem?

I'm running build.sh / deploy.sh out of 
https://github.com/ocadotechnology/codeforlife-deploy-appengine

What is the expected output? What do you see instead?

I expect it to deploy to appengine.
I get:
Updating module [dev-new]...\DEBUG: Operation 
[apps/decent-digit-629/operations/9b4485cb-f85d-4101-8090-bd9bd5d00d3d] 
complete. Result: {
    "metadata": {
        "target": "apps/decent-digit-629/modules/dev-new/versions/mike-test-2", 
        "insertTime": "2016-01-07T17:56:53.495Z", 
        "method": "google.appengine.v1beta4.Versions.CreateVersion", 
        "user": "snap-ci@decent-digit-629.iam.gserviceaccount.com", 
        "operationType": "create_version", 
        "endTime": "2016-01-07T17:59:02.699Z", 
        "@type": "type.googleapis.com/google.appengine.v1beta4.OperationMetadata"
    }, 
    "done": true, 
    "name": "apps/decent-digit-629/operations/9b4485cb-f85d-4101-8090-bd9bd5d00d3d", 
    "error": {
        "code": 13
    }
}

What is the output of 'gcloud info'?
Google Cloud SDK [91.0.1]

Platform: [Linux, x86_64]
Python Version: [2.7.6 (default, Jun 22 2015, 17:58:13)  [GCC 4.8.2]]
Python Location: [/usr/bin/python2]
Site Packages: [Disabled]

Installation Root: [/tmp/snap/google-cloud-sdk]
Installed Components:
  core: [2015.12.16]
  core-nix: [2015.12.16]
  gcloud: []
  gsutil-nix: [4.16]
  gsutil: [4.16]
  bq: [2.0.18]
  bq-nix: [2.0.18]
System PATH: 
[/home/m/google-cloud-sdk/bin:/home/m/tmp/AWS-ElasticBeanstalk-CLI-2.6.3/eb/linu
x/python2.7:/home/m/.gem/ruby/1.9.1/bin:/home/m/bin:/opt/freeware/bin:/home/m/bi
n:/opt/freeware/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
:/usr/games:/usr/local/games]
Cloud SDK on PATH: [False]

WARNING: There are old versions of the Google Cloud Platform tools on your 
system PATH.
  /home/m/google-cloud-sdk/bin/git-credential-gcloud.sh
  /home/m/google-cloud-sdk/bin/bq
  /home/m/google-cloud-sdk/bin/gsutil
  /home/m/google-cloud-sdk/bin/dev_appserver.py
  /home/m/google-cloud-sdk/bin/gcloud

Installation Properties: [/tmp/snap/google-cloud-sdk/properties]
User Config Directory: [/home/m/.config/gcloud]
User Properties: [/home/m/.config/gcloud/properties]
Current Workspace: [None]
Workspace Config Directory: [None]
Workspace Properties: [None]

Account: [snap-ci@decent-digit-629.iam.gserviceaccount.com]
Project: [None]

Current Properties:
  [core]
    account: [snap-ci@decent-digit-629.iam.gserviceaccount.com]
    disable_usage_reporting: [True]
  [app]
    use_appengine_api: [True]
    suppress_change_warning: [true]

Logs Directory: [/home/m/.config/gcloud/logs]
Last Log File: [/home/m/.config/gcloud/logs/2016.01.07/17.54.16.923848.log]

Please provide any additional information below.
This is with use_appengine_api True

(With = False, I get version in GET parameter not equal to version in app.yaml 
- but I have no version: in app.yaml, as otherwise gcloud errors immediately)

Original issue reported on code.google.com by mike.bry...@ocado.com on 7 Jan 2016 at 6:02

GoogleCodeExporter commented 8 years ago

This is an issue on our end.

We're rolling out some changes that should address the root cause of this error.

Can you provide more details on the failure mode you see under 
"use_appengine_api False"?

Thanks for the feedback!

Original comment by lball...@google.com on 7 Jan 2016 at 8:48

GoogleCodeExporter commented 8 years ago

Ah, good to hear. I'll wait for that to be resolved then.

Here's the debug log:
DEBUG: Send: /api/appversion/create, params={'version': 'mike-test-5', 
'app_id': 'decent-digit-629', 'module': 'flibble'}
DEBUG: _Authenticate configuring auth; needs_auth=False
DEBUG: Sending request to 
https://appengine.google.com/api/appversion/create?app_id=decent-digit-629&modul
e=flibble&version=mike-test-5 headers={'X-appcfg-api-version': '1', 
'content-length': '2107', 'Content-Type': 'application/octet-stream'} 
body=api_version: '1'
auto_id_policy: default
builtins:
- appstats: 'on'
- default: 'on'
derived_file_type:
- python_precompiled
env_variables:
REDACTED
handlers:
- script: google.appengine.ext.appstats.ui.app
  secure: optional
  url: /_ah/stats.*
- secure: optional
  static_files: static/portal/img/favicon.ico
  upload: static/portal/img/favicon\.ico
  url: /favicon\.ico
- secure: optional
  static_files: static/robots.txt
  upload: static/robots\.txt
  url: /robots\.txt
- secure: optional
  static_files: static/apple-touch-icon.png
  upload: static/apple-touch-icon\.png
  url: /apple-touch-icon\.png
- secure: optional
  static_files: static/apple-touch-icon-76x76.png
  upload: static/apple-touch-icon-76x76\.png
  url: /apple-touch-icon-76x76\.png
- secure: optional
  static_files: static/apple-touch-icon-120x120.png
  upload: static/apple-touch-icon-120x120\.png
  url: /apple-touch-icon-120x120\.png
- secure: optional
  static_files: static/apple-touch-icon-152x152.png
  upload: static/apple-touch-icon-152x152\.png
  url: /apple-touch-icon-152x152\.png
- secure: optional
  static_files: static/apple-touch-icon-180x180.png
  upload: static/apple-touch-icon-180x180\.png
  url: /apple-touch-icon-180x180\.png
- application_readable: true
  secure: always
  static_dir: __static__/static
  url: /static
- script: google.appengine.ext.appstats.ui.app
  secure: always
  url: /_ah/stats.*
- script: webapp.wsgi.application
  secure: always
  url: /.*
instance_class: F4
libraries:
- name: MySQLdb
  version: 1.2.4b4
- name: ssl
  version: '2.7'
- name: PIL
  version: 1.1.7
module: flibble
runtime: python27
skip_files: 
^(.*/)?.*\.scss$|^(.*/)?#.*#$|^(.*/)?.*~$|^(.*/)?.*\.py[co]$|^(.*/)?.*/RCS/.*$|^
(.*/)?\..*$|^lib\/.*info$|^lib\/PIL|^lib\/setuptools\/script.*dev.*.tmpl|README.
md|LICENSE
threadsafe: true
vm_settings:
  module_yaml_path: app.yaml

Updating module [flibble]...\DEBUG: Got http error 400.
DEBUG: Unexpected results: {'status': '400', 'alternate-protocol': 
'443:quic,p=1', 'content-length': '90', 'expires': 'Mon, 01 Jan 1990 00:00:00 
GMT', 'server': 'Google Frontend', 'cache-control': 'no-cache', 'date': 'Thu, 
07 Jan 2016 20:56:23 GMT', 'alt-svc': 'quic=":443"; ma=604800; 
v="30,29,28,27,26,25"', 'content-type': 'text/plain'}
Updating module [flibble]...failed.
DEBUG: (gcloud.preview.app.deploy) Server responded with code [400]:
  Bad Request Unexpected HTTP status 400.
  The version field in the app.yaml file must match the version provided as a GET parameter.
Traceback (most recent call last):
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 609, in Execute
    result = args.cmd_func(cli=self, args=args)
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 1228, in Run
    resources = command_instance.Run(args)
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/surface/preview/app/deploy.py", line 288, in Run
    gae_client.DeployModule(module, version, info.parsed, info.file)
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_client.py", line 309, in DeployModule
    return appversion.DoUpload()
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_deployments.py", line 1026, in DoUpload
    missing_files = self.Begin()
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_deployments.py", line 559, in Begin
    payload=config_copy.ToYAML())
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/util.py", line 293, in Send
    result = self.rpcserver.Send(url, payload=payload, **kwargs)
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/util.py", line 358, in Send
    response = self._server.Send(*args, **kwargs)
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/third_party/appengine/tools/appengine_rpc_httplib2.py", line 286, in Send
    'Unexpected HTTP status %s' % status)
  File "/tmp/snap/google-cloud-sdk/lib/googlecloudsdk/third_party/appengine/tools/appengine_rpc_httplib2.py", line 69, in RaiseHttpError
    raise urllib2.HTTPError(url, response_info.status, msg, response_info, stream)
RPCError: Server responded with code [400]:
  Bad Request Unexpected HTTP status 400.
  The version field in the app.yaml file must match the version provided as a GET parameter.
ERROR: (gcloud.preview.app.deploy) Server responded with code [400]:
  Bad Request Unexpected HTTP status 400.
  The version field in the app.yaml file must match the version provided as a GET parameter.
michael@stump:~/git/codeforlife-deploy-appengine$

Original comment by m...@mikebryant.me.uk on 7 Jan 2016 at 9:00

GoogleCodeExporter commented 8 years ago

(Is there an eta on when appengine deployments will be available again?)

Original comment by mike.bry...@ocado.com on 8 Jan 2016 at 11:25

GoogleCodeExporter commented 8 years ago

The planned rolled for the fix is this week.

Original comment by che...@google.com on 11 Jan 2016 at 4:05

Changed state: Accepted

GoogleCodeExporter commented 8 years ago

I was told the fix has been rolled out. Can you try and let us know if you 
still experience any issues.

Original comment by che...@google.com on 11 Jan 2016 at 10:43

Changed state: NeedsMoreInfo

GoogleCodeExporter commented 8 years ago

I just tried again - got the same error code 13

Original comment by mike.bry...@ocado.com on 11 Jan 2016 at 11:21

GoogleCodeExporter commented 8 years ago

Our rollout is complete, would you mind posting your deployment output from a 
new attempt?

Original comment by lball...@google.com on 13 Jan 2016 at 12:18

GoogleCodeExporter commented 8 years ago

Tried again this morning, and it's working now.

Thanks!

Original comment by mike.bry...@ocado.com on 13 Jan 2016 at 12:57

GoogleCodeExporter commented 8 years ago

Well, it completes - however it's currently failing to upload all of the files.

I can see from the debug output, when it gives the manifest:
...
  "webapp/__init__.py": {
    "sha1Sum": "c0c0afc4e762ed53e81593c705d6d5ea8acdcbbf", 
    "sourceUrl": "https://storage.googleapis.com/staging.decent-digit-629.appspot.com/c0c0afc4e762ed53e81593c705d6d5ea8acdcbbf.py"
  }, 
  "webapp/lib/memcache.py": {
    "sha1Sum": "f8048885d5a9f5c3bc53ceab8bcada959e07767b", 
    "sourceUrl": "https://storage.googleapis.com/staging.decent-digit-629.appspot.com/f8048885d5a9f5c3bc53ceab8bcada959e07767b.py"
  }, 
  "webapp/settings.py": {
    "sha1Sum": "01bec231df86ac85a6f574925864a44cc0487bfb", 
    "sourceUrl": "https://storage.googleapis.com/staging.decent-digit-629.appspot.com/01bec231df86ac85a6f574925864a44cc0487bfb.py"
  }, 
  "webapp/urls.py": {
    "sha1Sum": "1af504ec5f4c6fed7a8fbdd53814b2fa88162fc8", 
    "sourceUrl": "https://storage.googleapis.com/staging.decent-digit-629.appspot.com/1af504ec5f4c6fed7a8fbdd53814b2fa88162fc8.py"
  }, 
  "webapp/wsgi.py": {
    "sha1Sum": "9c18fd528c42d0498efacf25a80f9240264a5342", 
    "sourceUrl": "https://storage.googleapis.com/staging.decent-digit-629.appspot.com/9c18fd528c42d0498efacf25a80f9240264a5342.py"
  }

But in the appengine logs, from doing a directory listing:
13:01:46.404
ls: ['snap-ci-console.log', 'main.py', 'deploy.sh', 'app.yaml', 
'app.yaml.tmpl', 'README.md', 'client-key.pem', 'build.sh', 'manage.py', 
'static', 'LICENSE', 'appengine_config.py', 'server-ca.pem', 'client-cert.pem', 
'lib']

Is this part of the same general failure to work issue, or should I open this 
separately?

Original comment by mike.bry...@ocado.com on 13 Jan 2016 at 1:08

GoogleCodeExporter commented 8 years ago

I've opened a new issue for this behaviour in #467, as it seems unrelated

Original comment by mike.bry...@ocado.com on 13 Jan 2016 at 2:53

GoogleCodeExporter commented 8 years ago

Unfortunately, while it worked this morning - it's now stopped again, and 
returning the same error as before:

/var/go/google-cloud-sdk/bin/gcloud preview app --quiet deploy app.yaml 
--project decent-digit-629 --version snap-65 --no-promote
Loading user based configuration file: [/var/go/.config/gcloud/properties].
User based configuration files are deprecated and will not be read in a future 
gcloud release.
WARNING: The `gcloud preview app` surface is rapidly improving. Look out for
changing flags and new commands before the transition out of the `preview`
component. These changes will be documented in the Cloud SDK release notes
<https://goo.gl/X8apDJ> and via deprecation notices for changing commands.

If you would like to avoid changing behavior, please pin to a fixed version of
the Google Cloud SDK as described under the "Alternative Methods" section of the
Cloud SDK web site: <https://cloud.google.com/sdk/#alternative>.

You are about to deploy the following modules:
 - decent-digit-629/staging (from [/var/snap-ci/repo/app.yaml])
     Deployed URL: [https://snap-65-dot-staging-dot-decent-digit-629.appspot.com]
     (add --promote if you also want to make this module available from
     [https://staging-dot-decent-digit-629.appspot.com])

Beginning deployment...
Copying files to Google Cloud Storage...
Synchronizing files to [gs://staging.decent-digit-629.appspot.com/].

Updating module [staging]...
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging].../
Updating module [staging]...-
Updating module [staging]...\
Updating module [staging]...|
Updating module [staging]...failed.
ERROR: (gcloud.preview.app.deploy) Error Response: [13]

Original comment by mike.bry...@ocado.com on 13 Jan 2016 at 8:08

GoogleCodeExporter commented 8 years ago

Debug output:
Updating module [staging].../DEBUG: Operation 
[apps/decent-digit-629/operations/ab01522f-d65e-45d2-b52f-3c787af17443] 
complete. Result: {
    "metadata": {
        "target": "apps/decent-digit-629/modules/staging/versions/snap-66", 
        "operationType": "create_version", 
        "user": "snap-ci@decent-digit-629.iam.gserviceaccount.com", 
        "insertTime": "2016-01-13T20:14:25.733Z", 
        "@type": "type.googleapis.com/google.appengine.v1beta4.OperationMetadata", 
        "endTime": "2016-01-13T20:16:38.425Z", 
        "method": "google.appengine.v1beta4.Versions.CreateVersion"
    }, 
    "done": true, 
    "name": "apps/decent-digit-629/operations/ab01522f-d65e-45d2-b52f-3c787af17443", 
    "error": {
        "code": 13
    }
}

Updating module [staging]...failed.
DEBUG: (gcloud.preview.app.deploy) Error Response: [13]  
Traceback (most recent call last):
  File "/var/go/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 643, in Execute
    result = args.cmd_func(cli=self, args=args)
  File "/var/go/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 1401, in Run
    resources = command_instance.Run(args)
  File "/var/go/google-cloud-sdk/lib/surface/preview/app/deploy.py", line 387, in Run
    images.get(module))
  File "/var/go/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 90, in DeployModule
    return operations.WaitForOperation(self.client.apps_operations, operation)
  File "/var/go/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/api/operations.py", line 66, in WaitForOperation
    encoding.MessageToPyValue(completed_operation.error)))
OperationError: Error Response: [13] 
ERROR: (gcloud.preview.app.deploy) Error Response: [13]

Original comment by mike.bry...@ocado.com on 13 Jan 2016 at 8:55

GoogleCodeExporter commented 8 years ago

Thanks for checking in. I was told by the team that the fix was rolled back due 
to some issue but will be rolling out again soon.

Original comment by che...@google.com on 13 Jan 2016 at 9:10

GoogleCodeExporter commented 8 years ago

Is there any update on this?

Original comment by mike.bry...@ocado.com on 20 Jan 2016 at 5:40

GoogleCodeExporter commented 8 years ago

Hi, have you tried again. I believe the fix is finally out. Let us know if you 
get this or new error, if you do.

Original comment by che...@google.com on 20 Jan 2016 at 5:48

GoogleCodeExporter commented 8 years ago

It's still breaking with the same error: 
https://snap-ci.com/ocadotechnology/codeforlife-deploy-appengine/branch/master/l
ogs/defaultPipeline/66/Deploy-Staging?back_to=build_history

Original comment by mike.bry...@ocado.com on 20 Jan 2016 at 7:12

GoogleCodeExporter commented 8 years ago

Is there any more information required?

This is the output today:
Updating module [staging]...\DEBUG: Operation 
[apps/decent-digit-629/operations/509df670-eb49-44e5-8059-94d99e053c30] 
complete. Result: {
    "metadata": {
        "target": "apps/decent-digit-629/modules/staging/versions/snap-66", 
        "operationType": "create_version", 
        "user": "snap-ci@decent-digit-629.iam.gserviceaccount.com", 
        "insertTime": "2016-01-22T14:42:48.554Z", 
        "@type": "type.googleapis.com/google.appengine.v1beta4.OperationMetadata", 
        "endTime": "2016-01-22T14:45:07.565Z", 
        "method": "google.appengine.v1beta4.Versions.CreateVersion"
    }, 
    "done": true, 
    "name": "apps/decent-digit-629/operations/509df670-eb49-44e5-8059-94d99e053c30", 
    "error": {
        "code": 13
    }
}

Original comment by mike.bry...@ocado.com on 22 Jan 2016 at 2:51

GoogleCodeExporter commented 8 years ago

Is there anything I can do to work around this?

Original comment by mike.bry...@ocado.com on 26 Jan 2016 at 12:04

GoogleCodeExporter commented 8 years ago

Hello Mike, thanks for reporting the issue. Code 13 usually means there was 
internal backend error. The backend team tried some things thinking it will 
address the issue they have not yet been able to look deeper into this due to 
other constrains. We are tracking this internally and if there updates will let 
you know.

Original comment by che...@google.com on 26 Jan 2016 at 1:36

Changed state: Accepted

GoogleCodeExporter commented 8 years ago

We've rolled out an improvement to our deployment infrastructure.  Please try 
this again, and sorry for the delay!

Original comment by lball...@google.com on 28 Jan 2016 at 6:26

GoogleCodeExporter commented 8 years ago

That works :)
I managed to deploy successfully.

Original comment by mike.bry...@ocado.com on 28 Jan 2016 at 6:48

GoogleCodeExporter commented 8 years ago

Glad to hear that it worked.

Original comment by che...@google.com on 29 Jan 2016 at 12:56

Changed state: Fixed

modxevil / google-cloud-sdk

gcloud preview app deploy fails #462