Support for Google App Engine (GAE) deployment

[jeanEpek]

We would like to add Google App Engine (GAE) as a supported provider in the deploy section.

We currently use an after_success section but it's not ideal.

[royalsflush]

+1

[BanzaiMan]

@jeanEpek What does the after_success section look like?

[jeanEpek]

after_success:

• echo "PR# $TRAVIS_PULL_REQUEST"
• if [[ $TRAVIS_PULL_REQUEST == 'false' ]]; then
• head -n 6 $APP_DIR/app.yaml
• set -e
• python $GAE_DIR/appcfg.py --oauth2_refresh_token=$GAE_OAUTH update $API_DIR/app
• set +e
• STATUS=$?
• echo $STATUS
• if [[ $STATUS == 0 ]]; then
• echo "Version $API_VERSION deployed IN $ENV"
• if [[ $ENV == 'SANDBOX' ]]; then
• echo "Run tests in sandbox"
• python $BUILD_DIR/setup/run_tests.py $GAE_DIR tests/sandbox/ 1

  - if [[ $? -ne 0 ]]; then exit -1; fi

• fi
• else
• echo "Deployment of version $API_VERSION failed"
• fi
• python $BUILD_DIR/setup/notify.py $STATUS deploy.log
• else
• echo "Pull Request $TRAVIS_PULL_REQUEST testing run: No deployement performed"
• fi

[BanzaiMan]

@jeanEpek Hmm. The formatting was lost, unfortunately. Could you try wrapping it inside the YAML block?

```YAML
YOUR STUFF
```

[jeanEpek]

I'm afraid there is not more formatting in the raw thing:

after_success:
- echo "PR# $TRAVIS_PULL_REQUEST"
- if [[ $TRAVIS_PULL_REQUEST == 'false' ]]; then
- head -n 6 $APP_DIR/app.yaml
- set -e
- python $GAE_DIR/appcfg.py --oauth2_refresh_token=$GAE_OAUTH update $API_DIR/app
- set +e
- STATUS=$?
- echo $STATUS
- if [[ $STATUS == 0 ]]; then 
- echo "Version $API_VERSION deployed IN $ENV"
- if [[ $ENV == 'SANDBOX' ]]; then
- echo "Run tests in sandbox"
- python $BUILD_DIR/setup/run_tests.py $GAE_DIR tests/sandbox/ 1
- fi
- else 
- echo "Deployment of version $API_VERSION *failed*"
- fi
- python $BUILD_DIR/setup/notify.py $STATUS deploy.log
- else
- echo "Pull Request $TRAVIS_PULL_REQUEST testing run: No deployment performed"
- fi

Indented, it should show somehow like that:

after_success:
- echo "PR# $TRAVIS_PULL_REQUEST"
- if [[ $TRAVIS_PULL_REQUEST == 'false' ]]; then
    - head -n 6 $APP_DIR/app.yaml
    - set -e
    - python $GAE_DIR/appcfg.py --oauth2_refresh_token=$GAE_OAUTH update $API_DIR/app
    - set +e
    - STATUS=$?
    - echo $STATUS
    - if [[ $STATUS == 0 ]]; then 
        - echo "Version $API_VERSION deployed IN $ENV"
        - if [[ $ENV == 'SANDBOX' ]]; then
            - echo "Run tests in sandbox"
            - python $BUILD_DIR/setup/run_tests.py $GAE_DIR tests/sandbox/ 1
        - fi
    - else 
        - echo "Deployment of version $API_VERSION *failed*"
    - fi
    - python $BUILD_DIR/setup/notify.py $STATUS deploy.log
- else
    - echo "Pull Request $TRAVIS_PULL_REQUEST testing run: No deployement performed"
- fi

[BanzaiMan]

Thanks. It appears pretty straightforward to turn into a dpl provider, probably like PyPI.

[BanzaiMan]

OK, so if anyone wants to test this provider (even before it's merged to the master branch), add this to your .travis.yml:

before_deploy:
  - git clone -b ha-feature-gae https://github.com/travis-ci/dpl.git
  - pushd dpl && gem build dpl.gemspec
  - rvm 1.9.3 do ruby -S gem install dpl-*.gem
  - popd

and the deploy section should look like:

deploy:
  provider: gae
  oauth2_refresh_token:
    secure: STUFFFFFFFFFFFF
  skip_cleanup: true
  app_dir: foo # defaults to $TRAVIS_BUILD_DIR

[BanzaiMan]

Has anyone tried this? I'll be traveling early next week, so if there is any feedback, I'd appreciate it before that. Thanks.

[jeanEpek]

Thanks Banzai, Ill try it quickly.

[jeanEpek]

I pushed a branch (epek SSOAPI-132). TRhe build was successful but I don't see any deployment...

[BanzaiMan]

@jeanEpek Which repository is this?

[jeanEpek]

https://github.com/epek/sso branch SSOAPI-132

[BanzaiMan]

@jeanEpek Strange. For some reason, it did not trigger deployment. I'll take a deeper look later. Thank you.

[jeanEpek]

np. good luck

[BanzaiMan]

I went ahead and merged the feature branch to master, so that before_deploy is not needed. Adding edge: true in deploy (https://github.com/BanzaiMan/travis_production_test/blob/e9160bcf123021cbf6e66b87219489a99301d02a/.travis.yml#L12-L17) should do the job.

https://travis-ci.org/BanzaiMan/travis_production_test/builds/28642046 shows my feeble attempt; it should be executing appcfg.py with correct parameters, but there is no output. (It may be the case that there are other setup that needs to happen for GAE.)

The token is now specified by oauth_refresh_token (without 2).

[jeanEpek]

As far as I see, epek/sso build#419 still doesnt trigger deploy (branch SSOAPI-132). I created a pull request (build #420) and merged with the default branch (develop): that's build #421 and still no mention of deploy in the log, like if the deploy section didn't exist.

[BanzaiMan]

@jeanEpek There won't be any deployment for pull requests. Still not clear SSOAPI-132 isn't deploying; could you try adding on.all_branches: true?

[jeanEpek]

I know. I created a pull request in order to merge and trigger a build when the merged was committed: that's build # 421. Build #420 is the one triggered by the creation of the pull request. I added on.all_branches: true, committed and pushed branch SSOAPI-132 but no deploy (Build # 422). No sure if it's related, but the script still has an after_success and an after_script section.

[jeanEpek]

Hi @BanzaiMan, it seems we receive messages from the deploy section now, but there is an error ("no action") reported by appcfg. It looks like "update" keyword is missing in: python appcfg.py --oauth2_refresh_token=xxx update $APP_DIR

[BanzaiMan]

@jeanEpek Nice catch! The update will be available shortly.

[jeanEpek]

Cool! it passed.

I have a side question for you: is it possible to run scripts (test) automatically after deploy? It seems that the after_script: section is executed after the deploy: section but I wanted to be sure it's a feature.

Thanks!

[BanzaiMan]

Good to hear!

Indeed, after_script is a supported phase in our build lifecycle: http://docs.travis-ci.com/user/build-lifecycle/

[jeanEpek]

Looking at this doc page, the after_script: section should execute before after_success/failure sections. In my experience, it is executed after. But I found the after_deploy: section which is what I was looking for. I there a way to get a $TRAVIS_DEPLOY_RESULT similar to $TRAVIS_TEST_RESULT? Sometimes, the deployment fails and the python script returns a status != 0. On an ancillary note, this GAE deploy is the python version; there is also a java, golang, and php version which are possible. Thanks!

[BanzaiMan]

There is no $TRAVIS_DEPLOY_RESULT at the moment. That merits a separate ticket at https://github.com/travis-ci/travis-ci/issues/new.

And, yes, you can use after_deploy. I think you can consult $? to see if the deployment was successful in the meantime.

Does Java, Go, or PHP version give any more benefit over the python version?

[jeanEpek]

Ok, I'll create a ticket. Concerning the other languages, it's more if we want to generalize to all gae cases. The principle is the same but the tool is different, e.g. ./appengine-java-sdk/bin/appcfg.sh for java. It just stroke me that we would be specific to python with appcfg.py. Also, the sdk directory location ($GAE_DIR=$TRAVIS_BUILD/google_appengine in my script), which is used for testing and deploying, might also change since created by the user, so you might want to take it into account if you want to generalize this gae deploy option.

[BanzaiMan]

@jeanEpek

Thanks for opening the ticket.

The application directory is specified with app_dir option, defaulting to $TRAVIS_BUILD_DIR, if it, too, is absent.

As long as there is no feature missing in the Python client, there is no reason to switch to other clients (while performance may vary, I tend to think the difference would be negligible).

I'm going to close this issue now.

[jeanEpek]

I understood app_dir is the app directory (where the app code resides), not the SDK. The SDK is separate directory provided by google with libraries and tools (like appcfg.py) to access their platform.

Also, the python client can only be used with a python app. If I want to develop an app using java, and deploy it to gae, I must use a java sdk and java specific deploy tools. That's different from heroku for example which have the same client whatever the language used.

Note also that there is a new way of deploying to gae, through a github repo, called Pipelines/Push-to-Deploy, but it's still in preview and doesn't support all languages yet, i believe.

[BanzaiMan]

Oh, really? Each language requires a separate client to deploy? That is a serious feature that I have not considered. (And I wonder why.)

Hmmm. Not sure how we can support multiple languages within a single provider. :-/

[jeanEpek]

Yeah... you may have to create an additional parameter, e.g. language: or something alike... You may also want to have a parameter for the sdk repo, or just ask for the deploy tool to be in the path.

Otherwise you could still create gae_py, gae_java,... providers but that's less elegant!

[hello-josh]

Using the gcloud tool you can deploy any app regardless of language. It also supports service accounts for auth via gcloud auth activate-service-account passing a json key file instead of having to use a user's refresh token.

This is how I implement it:

install:
  - if [[ ! -f google-cloud-sdk/path.bash.inc ]]; then
      echo "Downloading google cloud sdk";
      wget https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.zip;
      unzip -qq google-cloud-sdk.zip;
      google-cloud-sdk/install.sh --usage-reporting false --path-update false --rc-path=~/.bashrc --bash-completion false --override-components=app;
    fi;
  - google-cloud-sdk/bin/gcloud components update preview app --quiet

.... snip

after_success:
- ./travis-ci-deploy.sh

Where travis-ci-deploy.sh boils down to the following:

google-cloud-sdk/bin/gcloud auth activate-service-account \
    --key-file "${GAE_CLIENT_KEY_JSON_FILE}"
google-cloud-sdk/bin/gcloud \
    --project "${GAE_PROJECT_ID}" \
    preview app deploy \
    --version "${GAE_VERSION_LABEL}" \
    --quiet \
    "app.yaml"

gcloud preview app deploy reference.

[lorenzleutgeb]

Hi!

I'm using an after_success similar to @Trii and would love to help improving support for GAE.

Disclaimer: I only use GAE for Go and a bit of Python, also I mainly use managed VMs. The other stuff I do not know very well.

As it seems to me there was a little confusion about Google's tools for GAE. Let me add my portion to this confusion: There is appcfg (also appcfg.py) which is old (AFAIK) and language agnostic. It is wrapped by other tools (such as goapp). I might even suspect that it will fade some day. Different language runtimes have added their flavours to it, like @jeanEpek indicated (the Java runtime has the most differences AFAIK).

Also, there is the gcloud which is not only relevant for GAE but also GCE (they are both part of "Google Cloud"). This is the tool I mostly use for administrating App Engine. It also wraps appcfg and friends with it's gcloud preview app ... commands. If you read closely, that page says:

These commands replace their equivalents in the appcfg tool.

Hmm ... This is why I suggest dropping appcfg altogether and adopting gcloud preview app. As the GAE provider is in some non-stable state anyway, this sounds reasonable. Also, I found that Google is quite responsive over at their issue tracker (way more than they normally are, at least), and we might be able to influence the tooling a bit.

I'd be happy to write up a provider that takes something like the Application ID, location of the JSON secret (with reasonable default), module, version ... and whatever comes to my mind. Other CI Providers require you to give away your personal credentials by the way: Circle CI, Shippable, Codeship (at least via OAuth). That makes me sad, I think Service Accounts are made for CI, and Travis should support that.

Also, a note on Managed VMs on GAE (the thingy I am using): They require a Docker host to prepare the container for deployment. We would probably have to wait for this to happen.

Off topic: I think getting in touch with gcloud might even be valuable thinking towards a GCE provider.

BTW, this issue lacks proper labeling.

[hello-josh]

FWIW this is how I do it now https://github.com/Trii/appengine-buildout-template/blob/master/travis-ci-deploy.sh

[lorenzleutgeb]

Just found this post by Andrew Jessup to app-engine-managed-vms@googlegroups.com (restricted), from 2015-07-08 that support the rumors:

Hi folks,

We're making an important change to the Google Cloud SDK to simplify how App Engine developers can access and work with the SDK.

As of the next release, 0.9.68, of the Google Cloud SDK - we will no longer be distributing the stand alone App Engine SDKs.

The gcloud preview app deploy and gcloud preview app run commands that most developers use within gcloud will continue to work, users should not notice any obvious change.

For users who were directly using appcfg.py, appcfg.sh, dev_appserver.py or dev_appserver.sh scripts, or the GUI launchers that were bundled with the Google Cloud SDK, you have two options:

You can continue to use the latest versions of these tools used by installing them from the stand-alone App Engine SDKs that can be found here: https://cloud.google.com/appengine/downloads.

In the short term, if you need to prevent this from happening, you can pin a version of gcloud SDK with the following commands:

gcloud config set --scope=installation component_manager/fixed_sdk_version 0.9.67 gcloud components update

As part of this change, the gae-java, gae-python, gae-php, and gae-go components no longer exist in the component manager. The gcloud preview app run command will install language specific runtime libraries on demand, as needed by your application.

If you have questions or concerns, please let us know.

Thanks,

Andrew, on behalf of the Cloud SDK team

Andrew Jessup | Product Manager, Google Cloud Platform | jes...@google.com

[hello-josh]

I spoke with some people in the #gcloud channel on freenode and they said that this does not affect using gcloud for deployment. It just means that you have to download the google app engine sdk in order to run your app locally since they will no longer bundle tools like dev_appserver.py and appcfg.py

[lorenzleutgeb]

Yeah, but I see that all these steps as a plan to deprecate dev_appserver.py and appcfg.py in the long term. And even if it's not true it still means that a dpl provider should use gcloud for deployment.

[ajessup]

Hey folks - gcloud preview app is definitely the right way forward here for GAE and GCE. It is a single toolchain that supports multiple languages and deployment targets, including Managed VMs. gcloud preview app run is effectively bundling the dev_appserver.py process and gcloud preview app deploy is a fork of appcfg.py.

I work pretty closely with the gcloud team and would be happy to facilitate a Travis integration. Drop me a line at jessup AT google.com and let me know what you need.

[waprin]

@flowlo Managed VMs no longer require Docker to deploy, currently the default tries to use Docker locally but falls back to a remote Docker build (using a GCE instance). Soon the remote Docker deploy will be the default. Until then if you want to avoid the fallback behavior, you can force it to use the remote Docker deploy with --docker-build=remote .

[waprin]

Here is an example of a deploy to Managed VMs from Travis:

https://github.com/GoogleCloudPlatform/continuous-deployment-demo/blob/managed_vms/.travis.yml

Note the line

ssh-keygen -q -N "" -f ~/.ssh/google_compute_engine

which generates keys to copy to the remote GCE instance to do the Docker build. Otherwise it's similar to the other App Engine examples like the one Trii posted. Would love to see this builtin to Travis though.

[BanzaiMan]

GAE provider now uses gcloud. Can this issue be closed?

[BanzaiMan]

Closing this.

[aubm]

For the record, deployment to GAE is not working for my Go app, this definitely has something to do with those issues:

• https://code.google.com/p/googleappengine/issues/detail?id=13199
• https://groups.google.com/forum/#!topic/google-appengine-go/eoSSKKN_VmI
• https://groups.google.com/forum/#!topic/google-cloud-sdk/x8Rf9n_U55M
• https://code.google.com/p/googleappengine/issues/detail?id=12554

[lorenzleutgeb]

@aubm please check whether #498 is the issue you are experiencing and maybe comment about your situation there and/or upvote it. If not, please file a new issue with a more complete description so that I can take a detailed look. Thank you very much!