This is the standalone version of our Asset Bundle Conversion tool that is actually present at the unity-renderer The intent of this repository is to decouple the conversion tool to have fewer dependencies and more maintainability.
To improve the performance of the WebGL build, in the past, we decided to convert all scenes into Asset Bundles. So this tool loads every scene asset, loads and re-imports all gltf's to turn them into AssetBundles just for the CI to upload them into the content servers.
git submodule update --init and git submodule update .) Unity 2021.3.20f1.Decentraland > Convert Scene menu.Start.AssetBundles folder located at the root of this repository.This tool is exposed as a standalone project and as a Docker-based service. The code of the service lives in the consumer-server folder, and runs commands locally by calling the project asset-bundle-converter of this same repository.
To build the image locally, Docker must be used. The recommended command is:
docker build -t ab-converter .And to run the server locally, the minimum command is the following:
docker run -p 5001:5000 ab-converterAfter it starts, you should be albe to hit http://localhost:5001/metrics to check the server is live.
The service uploads the results of the conversion to a S3 bucket defined in the CDN_BUCKET env var.
The structure of the bucket will look like this:
(root)
├──/manifest (manifests of the converted entities)
│ ├── entityId1.json
│ └── entityId2.json
├──/v4 (files of the v4 of the converter)
│ └── ...
└──/v5 (files of the v5 of the converter)
└── ... ENV AB_VERSION v1 in the Dockerfile)/manifest/:entity_id.json. Manifests should have a TTL of 1 hour in the edge and CDN cache./ab/:AB_VERSION/:CID. Enabling using the same CDN different versions at a time. Converted assets should have a TTL of 1year in the CDN and edge servers./ab/:AB_VERSION/:CID, /ab/:AB_VERSION/:CID.gz and /ab/:AB_VERSION/:CID.br being to enable network optimizations.Logs of conversions are stored in a different bucket for safety reasons, the bucket is defined with the environment variable LOGS_BUCKET.
The logs for each conversion are stored in the path logs/:AB_VERSION/:entityId/:DATE_AND_TIME.log
To schedule a manual conversion, there is an special with custom authentication at /queue-task. It sends a job to the queue, the job will be consumed by any available worker.
curl -XPOST -H 'Authorization: <TOKEN>' https://asset-bundle-converter.decentraland.org/queue-task -d '{"entityId": "bafyadsaljsdlkas", "contentServerUrl": "https://peer.decentraland.org/content"}' This converter leverages versioning for the assets, the version is changed by the AB_VERSION env var in the Dockerfile.
This differs from the previous asset bundles in an impactful way: not all assets are stored at root level anymore. This is due to incompatibilities across versions of the shaders/materials and unity itself.
Prior to this converter, the renderer used to look for the asset bundles of all models and textures, and fallback to the original asset if a 404 was returned.
Now the assets need to be resolved based on a manifest including the list of converted files. If the files are not in the list, we can fallback directly to the original asset without waiting for the 404 and thus, optimizing network roundtrips and loading times.
Here is some pseudocode to illustrate the asset resolution process for an entity:
// this is the manifest file, it is uploaded after the entire entity was uplodaded.
type Manifest = {
version: string
files: string[] // list of converted files
exitCode: number // not used
}
const assetBundleCdn = 'https://ab-cdn.decentraland.zone' // .org
/**
* This function returns the manifest of a converted scene
* @param entityId - EntityID used for the conversion
* @param assetBundleCdn - baseUrl of the asset bundle CDN.
*/
async function getSceneManifest(entityId: string, assetBundleCdn: string): Manifest | null = {
const manifest = await fetch(`${assetBundleCdn}/manifest/${entityId}.json`)
if (manifest.statusCode == 404) return null
return manifest.json()
}
/**
* This function resolves the final URL of the asset bundle or returns null if it was not converted
* @param manifest - Manifest of the converted entity
* @param cid - content identifier of the asset being converted
* @param assetBundleCdn - baseUrl of the asset bundle CDN.
*/
async function resolveAssetBundle(
manifest: Manifest,
cid: string,
assetBundleCdn: string
): string | null {
if (manifest.files.includes(cid)) {
if (UNITY_WEBGL)
// brotli compressed asset bundles for WebGL, the browser will
// uncompress and cache this asset in a different thread
// NOTICE: if the asset bundles support range requests, they
// won't work with the .br postfix!
return `${assetBundleCdn}/${manifest.version}/${cid}.br`
else
// raw asset bundles for the rest of the platforms
return `${assetBundleCdn}/${manifest.version}/${cid}`
}
return null
}As an exercise for the reader, here is a URL to resolve assets manually:
https://ab-cdn.decentraland.zone/manifest/bafkreie7b36aggssaerg7nvj5s56op5zqyxcqjtkq4q4kjrfhnkljhshgy.json
This repository has continous delivery to the goerli (decentraland.zone) network.
To deploy to production, you must first select the full commit hash from the version you whish to deploy. Then check it exists as a tag in https://quay.io/repository/decentraland/asset-bundle-converter?tab=tags and lastly execute the workflow "Manual Deploy" selecting the target environment and the docker tag (commit hash).
NOTICE: Please do not use latest as tag for the "Manual deploy", the pipeline of deployments runs on deltas, and if there was a "latest" and now we try to deploy "latest", it will detect no changes and finish without deploying. Always use the commit hash.
This repository is protected with a standard Apache 2 license. See the terms and conditions in the LICENSE file.