search

DestinyItemManager / dim-api

Destiny Item Manager API Service
MIT License
34 stars 15 forks source link

readme

The Destiny Item Manager API (DIM Sync)

Destiny Item Manager (DIM) uses the Bungie.net API to read information about Destiny game state, and to move or change items. However, DIM offers other features that Bungie.net's API does not support: tags and notes for items, saved loadouts, etc. To allow users to save this data and sync it between different clients (mobile, desktop, etc), we built our own API, which is branded as "DIM Sync" in our application. While this API was developed with DIM in mind, it is not exclusive to DIM. We designed it to be used by other Destiny community tools, and we welcome them to use it. Today, these other applications make use of DIM Sync:

What this isn't

This API is not the Bungie.net API! You cannot load Destiny inventory or move items with this API - that's the Bungie.net API. You also cannot remote-control DIM with this - it does not support applying DIM loadouts, or initiating DIM's Smart Moves. This API is limited to saving and loading information about tags, notes, saved DIM loadouts, and other information that is not part of the Bungie.net API.

API Types

The types for all API requests and responses are written out as TypeScript in the api/shapes folder. You can use the npm package @destinyitemmanager/dim-api-types to reference these types in your own code. This file in DIM's source code also lists out all the API endpoints, and shows examples of how to call them.

Updating the package version in dim-api-types/package.json and pushing to master will automatically publish to NPM.

Get an API key

To use the DIM API, you will need a DIM API key. Anyone can get an API key for localhost development using the instructions below.

Before you can interact with the DIM API you must set up your application to talk to the Bungie.net API. DIM's API piggybacks on Bungie.net's authentication: you need to have your users logged in via the Bungie.net API first, and then that token is used to get a token to talk to the DIM API. You will need to be ready with the Bungie.net API information for your development application.

Once your application is set up at Bungie.net, you can generate a DIM API key for localhost development. The payload is JSON, and requires three pieces of information:

Using your preferred method of making HTTP calls, perform a request to the DIM API's new_app endpoint:

curl 'https://api.destinyitemmanager.com/new_app' \
-X 'POST' \
-H 'Content-Type: application/json' \
--data-binary '{"id":"myusername-dev","bungieApiKey":"my-bungie-api-key","origin":"https://localhost:8080"}'

The response will contain a dimApiKey field - that's your DIM API key. If you ever forget it, make the request above again, and the same key will be returned to you.

Alternatively, you can set up DIM for local development — the "Enter API Credentials" step brings you to a developer page, which will have a button to fetch a DIM API key as well.

Once you have finished developing your app locally and are ready to push it to a production domain, join the DIM Discord and message bhollis to get a production token. Be sure to explain what you're building and how it will use the DIM API - if you don't, your request will be ignored.

Authenticating

As mentioned above, DIM's API piggybacks on Bungie.net's authentication: it knows a user is who they claim to be, because DIM confirms that their credentials check out with Bungie.net.

To authenticate a user of your application with the DIM Sync API, you will exchange their Bungie.net auth token for a DIM auth token. Just like the Bungie.net authentication system, the DIM auth token has an expiration time, after which you'll have to get a new one. Unlike the Bungie.net API, there's no refresh token — you just make the same request you made for the initial token. Here's how DIM does it.

To issue or reissue the DIM API token, you'll need your user's Bungie.net membership ID, an unexpired and valid Bungie.net access token, and your DIM API key. The DIM API will query the Bungie.net API using the user's access token, to verify that it belongs to the given membership ID — we don't store the token or use it for anything else.

curl 'https://api.destinyitemmanager.com/auth/token'
-X 'POST' \
-H 'Content-Type: application/json'
-H 'X-API-Key: dimApiKey'
{
  bungieAccessToken: 'foo',
  membershipId: '1234',
}

The returned token has an expiration - do not use the token after that expiration. For all subsequent DIM API calls, you'll include the DIM API token and the DIM API key as HTTP headers, like so:

Authorization: Bearer xxxThisUsersDimAccesTokenxxx
X-API-Key: xxxYourDimApiKeyxxx

Reading profile data

The DIM API is meant to be familiar to users of the Bungie.net API — there is a single, central read API: GET https://api.destinyitemmanager.com/profile?platformMembershipId=1234&components=tags,loadouts

The platform membership ID is required and should correspond to the Bungie.net platform you're loading for. DIM API data is stored per-platform. You should specify which components you want returned in the result. The current available components are:

Updating profile data

DIM API is meant to be usable "offline". A client can collect a sequence of change operations called updates, and then send them to be applied them in bulk, when the client is ready to make a request to the DIM API. By sending deltas in this way, the DIM API can effectively keep synchronized between different clients even if they're out of date, since you only send updates for what has changed.

As such, there's a single, central update endpoint: POST https://api.destinyitemmanager.com/profile

The body is a JSON object containing the destinyVersion (DIM supports D1 still!), the platformMembershipId, and a list of updates. Each update is one of the update types. So you can flush the local queue, and apply a series of changes consisting of different tag updates, loadouts, etc. — all interleaved together.

Loadout Shares

You can create shared loadouts, which are not part of a user's profile - these are globally accessible through dim.gg URLs. To create one, you can POST https://api.destinyitemmanager.com/loadout_share with a loadout document, and it will return the loadout's ID. If you GET https://api.destinyitemmanager.com/loadout_share?shareId=XXXXX you can retrieve a shared loadout based on its ID. It's best to read the DIM code related to shared loadouts to understand how to use them.

Cloud Architecture

The DIM API is a NodeJS server application that is deployed in the DigitalOcean cloud via Kubernetes.

Local Development

Local Postgres

The API server requires a Postgres database for storage. In production we use a hosted Postgres, but in development you need to run your own. You can install Postgres however you'd like, but one way is through Docker Desktop's local Kubernetes support:

  1. Go to hub.docker.com, sign up for an account, and download Docker Desktop.
  2. Open Docker preferences, go to the Kubernetes tab, and enable Kubernetes.
  3. Run kubectl apply -f kubernetes/postgres-configmap.yaml - this contains our development password and configures Postgres.
  4. Run kubectl apply -f kubernetes/postgres-deployment.yaml - this actually runs Postgres on port 31744. It will pull the latest Postgres image from Docker.

You can now connect to postgres on port 31744 with the username and password from the configmap file. Next, install the schema using a migration command.

DB Migrations

We use db-migrate to manage the schema in the database. Migrations allow you to make versioned changes to the schema and then apply them — first locally, and then to the production database.

Running the server

  1. Run corepack enable to enable NodeJS to find pnpm. With Homebrew, you need to brew install corepack.
  2. Run pnpm to install packages.
  3. Run pnpm start to run a development server. It'll need a Postgres server running on port 31744 that has been migrated to the latest schema for the server to work.

If you want to develop on dim.gg instead of api.destinyitemmanager.com, edit .env and set VHOST=dim.gg.

Running tests

  1. Run pnpm test. You'll need a Postgres server running on port 31744 that has been migrated to the latest schema for the tests to work.

Production Deployment and Administration

To access the production Postgres DB (either to run commands via the postgres shell or something like PSequel), you need to log into the DigitalOcean control panel, and copy the "Public Network" connection info for the dim-api-db database. You also need to add your home IP address to the "Trusted Sources" list.

You may also want to edit two files to include that information. These changes must never be checked in!

kubectl access

To be able to inspect or edit Kubernetes resources, or deploy, you will need to have your kubectl context set up. Log in to the DigitalOcean control panel, navigate to the Kubernetes cluster, and follow the instructions next to "Download Config File". It's easiest to install the doctl tool and ask it to create the kubectl context for you.

Deploying

Before deploying, run cd api && npx db-migrate up -e prod if you've created any new database migrations. If you forget to do this, the app will likely crash as it will try to do things with an outdated schema.

Deployment is done by running pnpm deploy. This will build the application into a Docker image, push that image to the global Docker container registry, and then apply an updated deployment config that points to the new image. You can run kubectl get pods to watch the pods cycle over to the new version, which takes a minute.

Handy Commands

Metrics

You can visit our metrics dashboard at https://grafana.destinyitemmanager.com to see how the API is doing. Access is granted through GitHub to DIM core team members.