search

Financial-Times / origami-image-service

Optimises and resizes images
https://www.ft.com/__origami/service/image
0 stars 0 forks source link
origami service

readme

Origami Image Service

Optimises and resize images. See the production service for API information.

MIT licensed

Table Of Contents

Requirements

Running Origami Image Service requires Node.js, npm and git-lfs.

Running Locally

Before we can run the application, we'll need to install dependencies:

npm install

Run the application in development mode with

make run-dev

Now you can access the app over HTTP on port 8080: http://localhost:8080/

Configuration

We configure Origami Image Service using environment variables. In local development, configurations are set in a .env file. In production, these are set through Doppler project that will sync secrets to Heroku as well

Add secrets locally

Origami stores their secrets on Doppler to get them on your local development environment you will need to install the Doppler CLI, login in Doppler and run the following command to setup Doppler within the repo:

doppler setup

Setup will ask you to select the project you want to use, select origami-image-service-v2 and then select the local environment. Once setup is complete you can download the secrets to your local environment by running:

doppler secrets download --no-file --format env-no-quotes > .env

NOTE: You might need to request contributor access to the Doppler project from the Origami team.

Required everywhere

Required in Heroku

Open Telemetry API configuration:

TODO: The options below are required at the moment, but are duplicates of other options above. This will be addressed once all services are using Origami Makefile.

Required locally

Headers

The service can also be configured by sending HTTP headers, these would normally be set in your CDN config:

Adding Images

The Origami Image Service fetches and transforms images from external hosts, such as FT APIs or any given URL. It also hosts a number of image sets directly.

To add, edit, or remove an image in one of these image sets see the image-sets directory. Some image sets have their own readme.md or contribution.md with further guidance specific to the image set. For example see the fticon contributing guide.

Removing an image from an image set is considered a major change. To remove an image a new major version of the Origami Image Service API must be released. Therefore it's typical to deprecate images first, and remove multiple deprecated images later as a batch. To manage this each image set directory has a deprecated.json file containing a list of images in the set which are deprecated and should be removed in the next major version of the Origami Image Service. Deprecated images are hidden on the Origami Image Service image sets page.

Testing

The tests are split into unit tests and integration tests. To run tests on your machine you'll need to install Node.js and run npm install. Then you can run the following commands:

make test              # run all the tests
make test-unit         # run the unit tests
make test-integration  # run the integration tests

You can run the unit tests with coverage reporting, which expects 90% coverage or more:

make test-unit-coverage verify-coverage

The code will also need to pass linting on CI, you can run the linter locally with:

make verify

We run the tests and linter on CI, you can view [results on CI][ci]. make test and make lint must pass before we merge a pull request.

You can run the integration tests against a URL by setting a HOST environment variable to the URL you want to test. This is useful for testing a Heroku application after it is deployed, which we do on CI.

HOST="https://www.example.com" make test-integration

Deployment

The production (EU/US) and QA applications run on Heroku. We deploy continuously to QA via [CI][ci], you should never need to deploy to QA manually. We use a Heroku pipeline to promote QA deployments to production.

You can promote either through the Heroku interface, or by running the following command locally:

make promote

Alternatively you can use labels on pull requests to promote to production. If you add the label release:major, release:minor or release:patch to a pull request, it will be promoted to production when it is merged. This is useful for small changes, or if you don't have access to the Heroku pipeline. But if changes are large, or you want to test them on QA first, you should merge PR without a label and test on QA.

Creating release manually from github will also promote to production.

Dependency management (Dependabot)

This repository uses Dependabot for automated dependency management. Dependabot regularly checks for outdated dependencies in the project and automatically creates pull requests to update them. This helps ensure that the project stays up-to-date with the latest versions, improving security and functionality.

By leveraging Dependabot, we can:

How to manage open Dependabot PRs?

  1. Identify a PR you want to tackle and re-run the status checks. If they all pass it will be automatically merged and deployed to QA app

    • If this fails on one of the status checks: Review the changes proposed by Dependabot to ensure they are compatible with the project. This involves checking out the branch and running locally and fixing where appropiate.

  2. Review the QA app to check the app is running normally

  3. Promote to production

Scheduled Tasks

The Origami Image Service uses a Heroku Schedule to run scripts/delete-old-images-from-cloudinary.js daily. This is setup only for the origami-image-service-eu app. It removes transformations for past images, reducing the number of images we store via Cloudinary.

Monitoring

Trouble-Shooting

We've outlined some common issues that can occur in the running of the Image Service:

Requesting a PNG but being returned a JPG, why is that?

When a png image is requested, and the requested image has no alpha channel (no transparency in the image), a jpg is instead returned because it will have a smaller filesize.

I need to purge an image

Please read the purging documentation on the website.

I need to purge all images, is this possible?

Please contact origami.support@ft.com - There is a way to purge all images, but this will incur a large cost.

What do I do if memory usage is high?

For now, restart the Heroku dynos:

heroku restart --app origami-image-service-eu
heroku restart --app origami-image-service-us

If this doesn't help, then a temporary measure could be to add more dynos to the production applications, or switch the existing ones to higher performance dynos.

What if I need to deploy manually?

If you really need to deploy manually, you should only do so to QA. Production deploys should always be a promotion from QA.

You'll need to provide an API key for change request logging. You can get this from the Origami LastPass folder in the note named Change Request API Keys. Now deploy to QA using the following:

make deploy

How do I reduce Cloudinary storage use?

Cloudinary stores images and their transformations (format, size, etc.). This is not always necessary as:

  1. Our CDN caches images, for up to a year depending on its origin.
  2. Images are less likely to be viewed after a period of time. Some images are replaced, and won't be accessed again.
  3. The image size apps request and the formats browsers support change – serving jpg is less common now.

To combat ever increasing storage requirements, a scheduled task periodically deletes stored transformations. If Cloudinary storage climbs high, ensure this script is working as expected. Alternatively, consider deleting all transforms for un-accessed assets using Cloudinary's bulk deletion tool.

SVGs don't work when running locally

When an SVG image is requested we rewrite the URL to go route back through the Image Service, this is to sanatize the SVG of any cross-site-scripting attack vectors and to tint the SVG if tinting has been requested. It looks something like this:

When you're running locally this won't work because Cloudinary cannot access your localhost. The flow would look like this:

So Cloudinary responds with a 404, and you may see an error like connect ECONNREFUSED 127.0.0.1:443. You can get around this by manually specifying a hostname in your configuration. You'll need to tell the service to rely on the QA instance for SVG tinting. Add the following to your .env file:

HOSTNAME=origami-image-service-qa.herokuapp.com

License

The Financial Times has published this software under the MIT license.