Custom image keep on pulling

[StefanAgus]

I've been using custom docker image to speed up the build time for around a week. But instead of making it faster, it become way slower because of the time it needs to pull the custom image. Is there any way to cache the custom docker image? here is our .shippable.yml ` build_environment: Ubuntu 14.04 language: node_js node_js:

• '4.6.2' services:
• redis # on port 6379 by default

• mongodb # on port 27017 by default env: global:

  • ROBOT_USERNAME=cermati-robot
  • ROBOT_EMAIL=robot@cermati.com
  • PROJECT="cermati web app"

  encrypted ROBOT_PASSWORD

  • secure: "hr+gy2HN/ndfUYB4bSbzuzEt+Eq1ROklPfF7Dnne583F7BzA6TUG94xvcwjmJl5QTAYAGRPSyu6rG3j48qVTX9THcrOI9fdHYpRBZ3CDWDhZdDekFSzd0wWikbdLgRke0sAbuofRF7Uqah6l+mgySty7RDQvZNLfgtUGrGyOSI3u+fXz/HwPIEofjp73MvLXy9KKH1RuouV5HJv/Gc69JHAKwa+ujkIH3FiYdfrAEk19+Gl7yYnFbL9KLw/r4MM5aFcdVqecdEJAiqX3wJlQaqcrP7vxBq2+ASPWsFUMw1E71dtOHP5jPGAOW5StYeTbaBSvQhduytPkPAHIwljF8Q=="

  encrypted SLACK_URL

  • secure: "Wm6zQzkDM9Oj3v25ZOWh7qnyZBUbF0K1xCCDFVpMR0KYq6aDg2vCFKUtpVyqXWqYQitCtzl8fpJbHw3vFS2XLpKas+Oc1DAehW7e4WKUJghLqmXx/IAzdi+qvvoFxkK23HP/jNvjXCwClxsVCKntpONHxDDyoigeLzOXkFmsEy6NIv3cbHPiGDNNcxgq0ecFn/lzxOX5AWYRfhCK23yxxy5Hii5+OUZVfaI4kevvTGDSdwYDXd6DVTc6iULSX6onvLKxR8FPI29qZhAWOpqT9G3m9XJAREB6F7bHia9pmRq7XHh3w1KhDT4vRA+IgqczWJoe0gMK7Ab18pGY7YzHGw=="

  encrypted S3 access_key_id

  • secure: "rM/b/tj1+kTgT81P6uDYv3lFjtl8rGZ985J0oU2O+PuC4r2qLqKg+eX8WGrNBS7v8FopqD0dc3gaZGcNt2jdjSKfPWmTjuyBKKOzXdaViEkp4wVebWGhPL5jAwDIhxua2cAq/R3WSHqIVT2ofenfIidVCW/n4XDsAHDa+jbq6Trr2S6M00KsXPB+ZwwGPnL34+zTiCG0XxIXAj5PcyoaJ32UYXc7wfQiUByg6LLfARnBC89q5TDsBeGSLtCciPNYyVPU2Nzyh374qOLAczyeTH44o0T57HdxhEBiZcl+1oRQQaVthuvVECtZ59jz+cOu9PrDGKz4zXf/k1e2UitCAw=="

  encrypted S3 secret_access_key

  • secure: "YCY+71aPz+cHeos1WSRKVAma5+8TG6vqzqLdVRbFTkHMqC/TW21K3YPvGQwRUFYrRl3NpKsr3z+VXf/W/Zpyyy7Wm2M8ht6bOJpKCENoVZSPdTRru5Gk7OHm+Er6rrhD90eaAW/rJhtQ8+OuJi368nXrajmKfc/sRSZUZVmndjSm2/PVI1NLJpGADCspnAX8/QI1wNtK4BcEyzwm6PRSWbkZ7+oG8LOVoWqJjf2TxMD3V5PH10Vy3HY4hoWXz5Ob71Bi1hW0ljade7OZ4Y7UIjaX0mqu6eWlq6bUKcn7oW5slBlQBrr2Em/BWuj64gEt6xFvIAUUh3gSg06c7N5vbg=="

  encrypted PRODUCTION_MONGO_URI

  • secure: "VIhd6lFTRLN3NGHJKbZ93yquP9/Ar0GAWnkx37ai6CUwLfGROLkaOp/b1lyFs+A78DiaX1rLTDCaTJ3lovBTehXmXDmgP1Y2vxVW9eh4rPvX00PjRk4rSYU6PZpmMAfQ/VIyUFisvo/oEVR7mKBk+LN/l2Q0a8iejr5UqgVSpVioMarnNzNOcD79BLwa57WpnN7AEgE5rjGdGO/DdDUO0hiM8NGGDWnw6U31WPVvScF4omxCcpgJKcE6KNg2hnF0PyCR5WSkEEBq+D/LVFemAI4ji0OG/NuXNmNgDKcIjMU67+vGbT/oyJpSlfC5jfhNf7IS+lZimlJ81nXaSzgH1A==" branches: only:
  • master
  • staging
  • production build: pre_ci_boot: image_name: sharyono/nodedocker image_tag: prod pull: true options: "-e HOME=/root"

  ci:

  Manual nvm Setup

  • . /root/.nvm/nvm.sh

    Starting redis server

  • sudo su -c "/usr/bin/redis-server > /dev/null 2>&1 &"

    Starting mongo server

  • sudo su -c "/usr/bin/mongod -f /etc/mongod.conf --smallfiles > /dev/null 2>&1 &"

  if there's string 'SKIPTEST' inside the commit message then skip integration test

  • export SKIP_TEST=git log --format=%s -n 1 $COMMIT | grep -q SKIPTEST ; echo $?

  if there's string 'SKIPTEST' then SKIP_TEST will be 0 otherwise 1

  • echo $SKIP_TEST
  • export SKIP_REVERT=git log --format=email -n 1 | grep -q 'Shippable Build' ; echo $?

  if the committer is Shippable Build then SKIP_REVERT will be 0 otherwise 1

  • echo $SKIP_REVERT

  • if [[ "$SKIP_TEST" == 1 && "$SKIP_REVERT" == 1 ]]; then export SHOULD_REVERT_IF_FAIL=1; else export SHOULD_REVERT_IF_FAIL=0; fi

  • echo $SHOULD_REVERT_IF_FAIL

  Already done in the docker image

  Dependencies to install yarn

  curl -sS http://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -

  echo "deb http://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

  Already done in the docker image

  install dependencies for node package canvas for Image Quality Assessor

  reference: https://github.com/Automattic/node-canvas

  sudo apt-get update

  sudo apt-get install -y libcairo2-dev libjpeg8-dev libpango1.0-dev libgif-dev build-essential g++

  Explicitly set NODE_ENV=development to force npm to install all dependencies

  • NODE_ENV=development

    Run the scripts to do yarn install and package installation

  • ./scripts/package_install.sh
  • cp .env.shippable .env

  Explicitly append NODE_ENV with current branch because .env.shippable does not set NODE_ENV

  • echo "NODE_ENV=$BRANCH" >> .env

  Explicitly append API_URL based on current branch

  • if [[ "$BRANCH" == "staging" ]]; then echo "API_KEY=https://stg1.cermati.com" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "API_KEY=https://www.cermati.com" >> .env; fi
  • if [[ "$BRANCH" == "master" ]]; then echo "API_KEY=http://localhost:3000" >> .env; fi

  Explicitly append BASE_URL based on current branch

  • if [[ "$BRANCH" == "staging" ]]; then echo "BASE_URL=https://stg1.cermati.com" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "BASE_URL=https://www.cermati.com" >> .env; fi
  • if [[ "$BRANCH" == "master" ]]; then echo "BASE_URL=http://localhost:3000" >> .env; fi

  Explicitly append OLX_ADIRA_PRODUCT_ID, OLX_ADIRA_SHARIA_PRODUCT_ID, OLX_ADIRA_USED_CAR_LOAN_PRODUCT_ID based on current branch

  • if [[ "$BRANCH" == "staging" ]]; then echo "OLX_ADIRA_PRODUCT_ID=577332f038562e8b39c065a5" >> .env; fi
  • if [[ "$BRANCH" == "staging" ]]; then echo "OLX_ADIRA_USED_CAR_LOAN_PRODUCT_ID=58b53df19620e17c7280ee72" >> .env; fi
  • if [[ "$BRANCH" == "staging" ]]; then echo "OLX_ADIRA_SHARIA_PRODUCT_ID=5773331338562e8b39c065a6" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "OLX_ADIRA_PRODUCT_ID=57733362cfc5c7436dae3b40" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "OLX_ADIRA_USED_CAR_LOAN_PRODUCT_ID=58b71cd22101e47f727f45c7" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "OLX_ADIRA_SHARIA_PRODUCT_ID=5773337dcfc5c7436dae3b41" >> .env; fi

  Explicitly append OLX_CIMB_MORTGAGE_PRODUCT_ID based on current branch

  • if [[ "$BRANCH" == "staging" ]]; then echo "OLX_CIMB_MORTGAGE_PRODUCT_ID=57d8f9cff6df8af012683353" >> .env; fi
  • if [[ "$BRANCH" == "staging" ]]; then echo "CIMB_MORTGAGE_LANDING_PAGE_PRODUCT_ID=57dbb8bed9ef8d8330fb08dd" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "OLX_CIMB_MORTGAGE_PRODUCT_ID=57d8fa3f8db2346b37eae5c7" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "CIMB_MORTGAGE_LANDING_PAGE_PRODUCT_ID=57dbd143e59979cb387f1b1f" >> .env; fi

  Explicitly append Gatekeeper secret key and partner on current branch

  • if [[ "$BRANCH" == "staging" ]]; then echo "API_PARTNER=cermati" >> .env; fi
  • if [[ "$BRANCH" == "staging" ]]; then echo "API_SECRET_KEY=vhGbtIM0BkxS8Wa68V8wS8UeibCe5zOT" >> .env; fi
  • if [[ "$BRANCH" == "staging" ]]; then echo "ENABLE_API_GATEKEEPER=true" >> .env; fi
  • if [[ "$BRANCH" == "staging" ]]; then echo "ENABLE_SKIP_APPLICATION_STEP=true" >> .env; fi
  • if [[ "$BRANCH" == "staging" ]]; then echo "CERMATI_TOKEN_URL=https://d29hoxuvzc9tjm.cloudfront.net/static/dist/js/cermati_token.js" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "API_PARTNER=cermati" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "API_SECRET_KEY=shMlboP4bAA7wt4lPntAPiixbSdyJ43v" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "ENABLE_API_GATEKEEPER=true" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "ENABLE_SKIP_APPLICATION_STEP=false" >> .env; fi
  • if [[ "$BRANCH" == "production" ]]; then echo "CERMATI_TOKEN_URL=https://dgxyivnisjpn1.cloudfront.net/static/dist/js/cermati_token.js" >> .env; fi

  Explicitly append OLX_DANAMON_MORTGAGE_PRODUCT_ID based on current branch

  • if [[ "$BRANCH" == "staging" ]]; then echo "OLX_DANAMON_MORTGAGE_PRODUCT_ID=5865d37b4abbed72768a411d" >> .env; fi

  • if [[ "$BRANCH" == "production" ]]; then echo "OLX_DANAMON_MORTGAGE_PRODUCT_ID=5865d54709b490843c8a923f" >> .env; fi

  • if [ "$BRANCH" != "master" ]; then ./node_modules/.bin/gulp build.production; fi

  • if [ "$BRANCH" = "master" ]; then ./node_modules/.bin/gulp build.development; fi

  • ./node_modules/.bin/gulp --gulpfile partnershipPage/olx-adira/gulpfile.js

  • ./node_modules/.bin/gulp --gulpfile partnershipPage/olx-cimb-mortgage/gulpfile.js

  • ./node_modules/.bin/gulp --gulpfile partnershipPage/cimb/mortgage/gulpfile.js

  • ./node_modules/.bin/gulp --gulpfile partnershipPage/olx/mortgage/danamon/gulpfile.js

  • ./node_modules/.bin/gulp --gulpfile partnershipPage/olx/used-car-loan/adira/gulpfile.js

  copy shippable env to test env

  • if [[ "$BRANCH" == "master" && "$SHOULD_REVERT_IF_FAIL" == 1 ]]; then cp .env.shippable ./tests/.env; fi
  • if [[ "$BRANCH" == "master" && "$SHOULD_REVERT_IF_FAIL" == 1 ]]; then npm run test-sitemap; fi
  • if [[ "$BRANCH" == "master" && "$SHOULD_REVERT_IF_FAIL" == 1 ]]; then npm run test; fi

  Use node version v5.7.1 because jsdom (in test-react script) needs node version > 4.x

  install Node Version Manager

  • curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | bash
  • nvm install v5.7.1
  • nvm use v5.7.1
  • npm run test-react

  on_success:

  • export ROOT_DIRECTORY=pwd

    Create new release if not Master

  • if [ "$BRANCH" != "master" ]; then ./node_modules/.bin/gulp ci.push-tag; fi
  • if [ "$BRANCH" != "master" ]; then cp public/dist/styles/*.min.css public/dist/; fi
  • if [ "$BRANCH" != "master" ]; then BUILD_DATE=$(git log -n1 --pretty=format:'%ci' | cut -d' ' -f1); fi

  install s3cmd tool for upload build artifact to s3

  • if [ "$BRANCH" != "master" ]; then sudo apt-get install s3cmd; fi
  • if [ "$BRANCH" = "staging" ]; then export DEPLOY_TO="com.cermati.build.artifacts/staging/shippable-build/$BUILD_DATE/$COMMIT/"; fi
  • if [ "$BRANCH" = "production" ]; then export DEPLOY_TO="com.cermati.build.artifacts/production/shippable-build/$BUILD_DATE/$COMMIT/"; fi
  • if [ "$BRANCH" != "master" ]; then echo -e "[default]\naccess_key=$access_key_id\nsecret_key=$secret_access_key\nbucket-location=ap-southeast-1\n" > /root/.s3cfg; fi
  • if [ "$BRANCH" != "master" ]; then cd public/dist; fi
  • if [ "$BRANCH" != "master" ]; then s3cmd sync . s3://$DEPLOY_TO; fi
  • cd $ROOT_DIRECTORY && python slack_notifier.py --slack-url $SLACK_URL --message success

  clean up

  • rm .env
  • if [ -f ./tests/.env ]; then rm ./tests/.env; fi

  on_failure:

  • if [[ "$BRANCH" == "master" && "$SHOULD_REVERT_IF_FAIL" == 1 ]]; then ./node_modules/.bin/gulp ci.revert-last-commit; fi
  • python slack_notifier.py --slack-url $SLACK_URL --message failure

  clean up

  • rm .env
  • if [ -f ./tests/.env ]; then rm ./tests/.env; fi

  cache: true cache_dir_list:

  • $SHIPPABLE_BUILD_DIR/node_modules `

[ambarish2012]

@StefanAgus to cache your own image, you will need to build on your own dedicated node. On the free plan, nodes gets dynamically assigned which prevents caching.

Please look here for more information - http://docs.shippable.com/getting-started/byon-overview/#advantages-of-byon.