TrackHAR tracker database schema

[zner0L]

• Can we generate the documentation from the schema "without any further work"?
• The result is a JSON of the data transferred by each tracker, generated by adapters
  • How to format the location of the data with it being so nested and obfuscated?
    • Generate a text from the code we use to extract the data
  • Is it enough evidence that our tool just extracts the data?
  • How to normalize the values?
    • only for human readability
• Publish the request data we use to reverse engineer the tracker requests
  • Can we find a suitable journal in addition to us publishing on our website?

[zner0L]

One idea:

This object would be the result:

{ 
    prepare: { body: 'ungz.pb', query: 'plain' },
    values: {
        location: { value: '0.00000 0.00000', location: 'body.3045', prepare: 'base64' },
        userId: { value: 42, location: 'body.id' },
    }
}

With this JSON as the in between step (having run gunzip and transformed the protobuf schema):

{
    "id": 42,
    "3045": "MC4wMDAwMCAwLjAwMDAw"
}

The full prepare path for the location would be ungz.pb.base64

[baltpeter]

Very rough draft of what an entry in the tracker wiki thingy might look like:

Tracker name: OneSignal

Exodus link? datenanfragen/data link

Endpoint 1

Endpoint URLs:

• https://api.onesignal.com/players
• https://onesignal.com/api/v1/players
• /https:\/\/api.onesignal.com\/players\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}/

TODO: We probably don't want to mention match here?

View recorded requests to this endpoint

Request format (TODO: bad title)

blurb: The bodies are gkgkd

To parse, you:

1. base64_decode(body)
2. JSON.parse(1.b)
3. gunzip(1.a)
4. return { ...2, ...3 }

Contained tracking data

Data type	Source	Path	Reasoning
Location	body	user.device.abc[0].latitude + ' ' + user.device.abc[0].longitude	obvious property name
Location	body	facebook.ads.location	obvious property name
Location	query string	a	obvious values
Network connection type	body	user.device.abc[0].network_type		https://source.tld
Unique ID	header	X-Secret-Value	https://research.tweasel.org/onesignal/x-secret-value

Endpoint 2

…

[zner0L]

• If a value contains more than one distinct information, like a locale property that contains language and country information, we just use this one property as a path information without applying any splitting functions. We only apply further parsing of non-standard formats which are too obfuscated to be easily parsed.
• If information or part of it is contained in multiple locations, all of them are mentioned in the extract object in an array. We now decided to split any compound information following the above principle.

[zner0L]

We have the problem that sometimes data is contained in lots of similar objects in an array, like Google Firebase transmitting a list of events. We then find the same data multiple times in in the array. To document this, we use the asterisk * from JSON path. This leads to the situation, where we can either find the same information in several different places e.g. in the query string or the body, or we find the same information in the same place multiple times. In this case, we combine both value arrays in the return value into one flattened array.

[zner0L]

To link to specific historic versions of an adapter, we generate a permalink from the adapter id and the commit hash.

[baltpeter]

Initial implementation of this: tweaselORG/TrackHAR#1