Correctly use the bungie manifest

[GeekyDeaks]

This is a huge task, but one that could be significantly useful. Currently there are some hardcoded hacks for class / gender and this does not work for race. Taking a look at the destiny manifest. There are bunch of links for some (fairly large) SQLite DB's, which contain a wealth of info, including some real gems like the activity hash, so I think we can report what an on-line character is doing. e.g.

sqlite> select * from DestinyActivityDefinition limit 1;
-2143553567|{"activityHash":2151413729,"activityName":"Tower","activityDescription":"Home of the Guardians, where you can regroup, rearm, and form new alliances before venturing beyond.","icon":"/common/destiny_content/icons/4cbdf020c412d4425aae660b8385f5f5.png","releaseIcon":"/img/misc/missing_icon.png","releaseTime":0,"activityLevel":1,"completionFlagHash":0,"activityPower":0.0,"minParty":1,"maxParty":6,"maxPlayers":16,"destinationHash":2612139675,"placeHash":3747705955,"activityTypeHash":1589650888,"tier":-1,"pgcrImage":"/img/theme/destiny/bgs/pgcrs/default.jpg","rewards":[],"skulls":[],"isPlaylist":false,"isMatchmade":true,"hash":2151413729,"index":0,"redacted":false}

The race is also in there:

sqlite> select * from DestinyRaceDefinition limit 1;
-1491684358|{"raceHash":2803282938,"raceType":1,"raceName":"Awoken","raceNameMale":"Awoken Male","raceNameFemale":"Awoken Female","raceDescription":"The Awoken survived the Collapse in deep space. But they were forever changed.","hash":2803282938,"index":0,"redacted":false}

The really nice thing is that there are DB's for a bunch of different languages, so we can render them in the persons native language! nice... especially since I am with a bunch of europeans... although I have to confess many of them seem to know english better than I... which is a little worrying :)

[unisys12]

We should write this where it updates automagically every other Tuesday. This is when the API is updates take place. This way, we can keep up with changes dynamically and not proactively.

Not sure if I've shared this with you, but I keep a copy of the manifest files in CSV format in a Drive Folder. I can send you the link to that, which will give you a slight broader view of what all is in there. Using apps such as DbBrowser, Raptr and RazorSQL are nice frontends for SQLite, but make it hard to read most of the data because the meat of the dish is stored in a BLOB format (json object).

Sticking with SQLite will aid us in getting this feature up and going pretty quick. Will start on it now, actually.

[unisys12]

got a little done on this... just in case you haven't peered into my Discord Test Chat. LOL! It's not finished and running into stupid issues. WIth that, I am going to call it a night. If you want to take a look at my progress/failures, checkout https://github.com/unisys12/discord-destinybot/tree/impliment-sqlite. That is the branch I am currently working on.

At present, I am pulling the manifest, but having a really stupid issue/brain-fart, where I cannot seem to retrieve any of the contents of the object. Everything returns "undefined". It's been a long day, so I am probably not thinking clearly. Also, if you look at the commit history, you can see the changes a bit clearer.

There are so many possibilities with this API, it is unreal. In this environment though, we are somewhat limited, but will still be cool to deliver something like this. Anyway, pending catching up on the live stream tomorrow night (will not be able to watch it live) I will jump back on this... unless you want to carry this further or whatever.

Either way... I have to close my eyes now.

[GeekyDeaks]

Looking good - I'll take a look at your manifest code when I get to work. I agree about the auto-update, I noticed that the initial /Manifest/ entry point has the following key:

"version": "51273.16.07.06.1706-15"

So I am guessing we can use that to determine if we need to do a refresh?

You probably already know this, but I eventually realised that the id field in the database is just the hash coerced into a signed 32 bit integer, which we can do easily in JS like so:

var id = hash | 0;

[GeekyDeaks]

Ok, see what is going on. I made the destinyAPI() function generate the URL as follows:

url: config.destiny.url + op

so everything is expected to be relative to https://www.bungie.net/Platform/Destiny. Unfortunately the URL's provided in the manifest i.e.:

"mobileWorldContentPaths": { "en": "/common/destiny_content/sqlite/en/world_sql_content_c7ea5036ed3278fe06a7a8feae87bf94.content",

are all relative to https://www.bungie.net/. You can probably re-use the get thunk like so:

var res = (yield get(uri + "/" + lang))[0];
if (res.statusCode !== 200) {
    logger.error("failed to download mobileWorldContent: %s\n", res.statusCode);
    throw new Error("download failure: "+res.statusMessage);
} else {
    var zip = res.body;
    // unzip the DB
}

Just need to remember to wrap all this in a co(function *() { }) though... They are only about 4 MB compressed, but may want to think about .pipe()'ing it into a temporary file instead of using yield.

[unisys12]

Alright, I got the URI sorted. But I am thinking that inside of doing all that URI processing and API calls within /bot/commands/manifest.js, I should make a separate API method in /destiny/index.js. This will keep all API logic within the same file and cut down on the need to initialize the same dependencies multiple times for the same action.

Will get on that tonight after work. Updated branch with changes so far this morning.

[GeekyDeaks]

Hi Phillip - I copied your code into my modular branch and took a look, after a little bit of head scratching found out the request was corrupting the .zip. I made some quick changes and it now downloads the manifest file and extracts the sqlite db.

[unisys12]

Ok, so all that's left is incorporating a library to handle SQLite interact. Create a manifest, that can be pulled up and down with each update. This content file has all the cmds needed to handle creation of the database, so all we have to do is import this file through the manifest. Then writing all the queries that connect to all the things!

Sorry about not being able to get that finished last night and you having to correct it. Didn't want to make a pull request with partial code in it is all.

[GeekyDeaks]

It's fine - you do all the leg work and I take the glory!

Shall we make life a little easier to begin with a have a static DB file. This way we can focus on the manifest lookups, then deal with the auto-update later?

[unisys12]

yeah, that was sorta what I was going to do. Baby steps. Update and upgrade after things are working. proof of concept, I guess.

[unisys12]

Submitted pull request #10 to address a portion of this.

[GeekyDeaks]

commit ae72bf9 implements a very basic activity lookup. It relies upload the mongoDB being setup and populated correctly. The config will need to be updated e.g.:

modules: {
    db : {
        url: "mongodb://localhost:27017/des"
    },
    destiny: {
        ....
        collection: "destiny.manifest"
    },

Then run node manifest_update.js and keep your fingers crossed.

I have made the .init() function for each module return a promise. The reason for this was to try and allow async operations, such as opening the DB to finish before loading other dependent modules. At the moment, the modules are loaded in the order they are defined in the config, but I am not sure why this is as it's defined as an object and not an array, so I was thinking of maybe changing it to array to be sure.