search

tweaselORG / TrackHAR

Library for detecting tracking data transmissions from traffic in HAR format.
Creative Commons Zero v1.0 Universal
5 stars 0 forks source link

Write adapter for singular.net #16

Closed baltpeter closed 8 months ago

baltpeter commented 1 year ago

https://sdk-api-v1.singular.net/api/v1/start

Observed in the Airbnb app on Android (no interaction):

POST /api/v1/start?a=airbnb_b5aba129&ab=x86_64&aifa=a7d11091-ffdf-42d9-b1c8-04b0fb065fcc&asid_scope=0&asid_timeinterval=0.0&av=23.23&br=google&c=wifi&current_device_time=1686175607578&custom_user_id=7d3e3ffa90ffd44b&ddl_enabled=true&ddl_to=60&de=generic_x86_64_arm64&device_type=phone&device_user_agent=Dalvik%2F2.1.0+%28Linux%3B+U%3B+Android+11%3B+sdk_gphone_x86_64+Build%2FRSR1.210722.013.A2%29&dnt=0&event_index=0&i=com.airbnb.android&install_ref_timeinterval=0.001&install_time=1686175525911&is=true&k=AIFA&lag=0.054&lc=en_US&ma=Google&mo=sdk_gphone_x86_64&n=Airbnb&p=Android&pr=sdk_gphone_x86_64&rt=json&s=1686175607577&sdk=Singular%2Fv12.0.6&singular_install_id=26037c57-5815-4a4d-9b80-0f28b036e2e3&u=a7d11091-ffdf-42d9-b1c8-04b0fb065fcc&update_time=1686175591090&v=11&h=ccafe33a50675394553462a7e3aaae4fd61d81df HTTP/1.1

User-Agent: Singular/SDK-v12.0.6.PROD
Content-Type: application/json
Host: sdk-api-v1.singular.net
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 2
baltpeter commented 1 year ago

Also, in the same run:

POST /api/v1/event?a=airbnb_b5aba129&aifa=a7d11091-ffdf-42d9-b1c8-04b0fb065fcc&av=23.23&c=wifi&custom_user_id=7d3e3ffa90ffd44b&event_index=3&i=com.airbnb.android&k=AIFA&lag=0.006&n=__LicensingStatus&p=Android&rt=json&s=1686175607624&sdk=Singular%2Fv12.0.6&seq=2&singular_install_id=26037c57-5815-4a4d-9b80-0f28b036e2e3&t=25.857&u=a7d11091-ffdf-42d9-b1c8-04b0fb065fcc&h=2962f60ec86a94b53f1344b4e2e1002f9b8f6fcb HTTP/1.1

User-Agent: Singular/SDK-v12.0.6.PROD
Content-Type: application/json
Host: sdk-api-v1.singular.net
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 146
baltpeter commented 10 months ago

Endpoints we currently know:

baltpeter commented 10 months ago

On first look, it seems like we'll only need a single adapter for these. They all transmit the data as query string parameters and/or a JSON body with a single top-level payload property, which is a string of a JSON object holding the actual data.

None of the requests set any cookies and all headers I saw were benign.

baltpeter commented 10 months ago

The av property in the query string always matches the app's version for the requests in our database. A few samples:

It's also likely that "av" is an acronym for "app version".

baltpeter commented 10 months ago

The v property in the query string is the operating system version. These are the values we have observed for this property: 11, 14.8, 14.5.1, 15.6.1, 13

Those exactly match the OS versions we have tested on (Android 11, iOS 14.8, iOS 14.5.1, iOS 15.6.1, Android 13).

baltpeter commented 10 months ago

The custom_user_id property in the query string can be used by the developer to transmit their own first-party ID for the user to Singular. This is already self-evident from the name, but explicitly confirmed in section 2.5 of Singular's own Android SDK integration guide (archived):

The Singular SDK can send a user ID from your app to Singular. This can be a username, email address, randomly generated string, or whichever identifier you use as a user ID. Singular uses the user ID in user-level data exports as well as internal BI postbacks (if you configure such postbacks).

Singular further confirm themselves that the custom_user_id is persistent unless explicitly unset by the developer:

The user ID persists until you unset it using unsetCustomUserId or until the app is uninstalled. Closing/restarting the app does not unset the user ID.

In many cases, you will want to call setCustomUserId when the user logs in to your service and unsetCustomUserId if the user logs out.

Similarly, in their website SDK guide (archived), they say:

The Singular SDK can send a user ID from your website to Singular. This can be a username, email address, randomly generated string, or whichever identifier you use as a user ID. […] The user ID persists until you unset it using the logout method or until the user deletes their local storage.

For reference, these are the values for this property we have observed (remember that we never signed in with an account or similar, so it is to be expected that there are no email addresses, usernames, etc.):

baltpeter commented 10 months ago

Very helpful: Looks like the Server-to-Server (S2S) API Endpoint Reference (archived) contains documentation for just about every parameter!

The server-to-server integration allows developers to manually send tracking data to a REST API, unlike the SDK, which does that automatically for the developer (quote from Server-to-Server (S2S) Integration Guide, archived):

As an alternative to the Singular SDK, which has to be embedded within your apps, Singular also provides a REST API that can be used to build a full integration that runs from your own server.

As such, it makes sense that the docs for that document all parameters.

baltpeter commented 10 months ago

The p property in the query string is the name of the operating system/platform.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
p One of the following Platforms: Android, iOS, PC, Xbox, Playstation, Nintendo, MetaQuest or CTV iOS, Android, PC, Console Android

This lines up with the values we have observed for this property in our request database. We have only seen Android and iOS.

baltpeter commented 10 months ago

The i property in the query string is the app ID.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
i Package Name (Android) or Bundle ID (iOS) of your application. iOS, Android,PC, Console com.singular.app

This lines up with the values we have observed for this property in our request database. Some examples:

baltpeter commented 10 months ago

As the name already implies, the ip property in the query string is the device's IP address.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
ip The IP of the device. iOS, Android,PC, Console 172.58.29.235

This lines up with the values we have observed for this property in our request database:

baltpeter commented 10 months ago

The ve property in the query string is the operating system's version.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
ve OS Version of the device at session time. iOS, Android,PC, Console 9.2

This lines up with the values we have observed for this property in our request database:

Those are some of the iOS (first three) and Android (last one) versions we tested with.

baltpeter commented 10 months ago

As the name implies, the asid property in the query string is the Android App Set ID.

According to Singular's documentation (archived):

Parameter Description Supported Platforms
asid The App Set ID (for Android 12+ devices) Android

See also their Types of Device IDs documentation (archived):

Device ID Type Name Platform Description
ASID App Set ID Android A developer-scoped device ID on Android devices.
baltpeter commented 10 months ago

The ma property in the query string is the device manufacturer.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
ma Make of the device hardware, typically the consumer-facing name (e.g., Samsung, LG, Apple). This parameter must be used with the model parameter. iOS, Android samsung

This lines up with the values we have observed for this property in our request database:

baltpeter commented 10 months ago

The mo property in the query string is the device model.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
mo Model of the device hardware (e.g., iPhone 4S, Galaxy SIII). This parameter must be used with the make parameter. iOS, Android SM-G935F

This lines up with the values we have observed for this property in our request database. Some examples:

baltpeter commented 10 months ago

The lc property in the query string is the device's locale.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
lc The IETF local tag for the device, using two-letter language and country code separated by an underscore. iOS, Android en_US

This lines up with the values we have observed for this property in our request database:

baltpeter commented 10 months ago

The bd property in the query string is the operating system's build number.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
bd Build of the device, URL encoded iOS, Android Build%2F13D15

This lines up with the values we have observed for this property in our request database:

The first three are iOS build numbers (cf. https://tidbits.com/2020/07/08/how-to-decode-apple-version-and-build-numbers/, archived), and the fourth one is the base64-encoded string for 19G82. The last one is an Android build number:

Screenshot_1692286707

baltpeter commented 10 months ago

As the name already implies, the idfa property in the query string is the device's Apple IDFA.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
idfa For iOS apps only. Upper-case raw advertising ID with dashes. iOS DFC5A647-9043-4699-B2A5-76F03A97064B

The one value we have observed for this property in our request database (F04C6D92-D4FA-4CA4-AC62-DE94C8C2DAE7) is indeed the IDFA of one of our test devices.

baltpeter commented 10 months ago

As the name already implies, the idfv property in the query string is the devices Apple IDFV.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
idfv For iOS apps only. Upper-case raw IdentifierForVendor with dashes. iOS 21DB6612-09B3-4ECC-84AC-B353B0AF1334
baltpeter commented 10 months ago

The aifa property in the query string is the device's Google Advertising ID.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
aifa For Android apps only. Lower-case raw advertising ID with dashes. Android 8ecd7512-2864-440c-93f3-a3cabe62525b

The values we have observed for this property in our request database are indeed the GAIDs of our test devices.

baltpeter commented 10 months ago

The andi property in the query string is the devices ANDROID_ID.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
andi For Android apps only. Lower-case raw android ID. Required only when Android Advertising ID is not available on the device. Android fc8d449516de0dfb
baltpeter commented 10 months ago

As the name implies, the app_v property in the query string is the app version.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
app_v App version iOS, Android 1.2.3
baltpeter commented 10 months ago

The install property in the query string signifies whether this is the app's first launch after installation.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
install Install flag. 'true' if the session was the first after installing the app. 'false' otherwise. Required for Reinstall tracking capabilities. iOS, Android,PC, Console false

In our request database, we have only observed the value true for this property, which makes sense considering that we always test on freshly installed apps.

baltpeter commented 10 months ago

As the name implies, the install_time property in the query string is the precise date and time of when the app was installed.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
install_time The time of the first app install as UNIX time. […] iOS, Android 1510040127

The values we have observed for this property in our request database are UNIX timestamps with different precision that line up with the time frames of our test runs. Some examples:

baltpeter commented 10 months ago

The c property in the query string encoded the way the device was connected to the network.

According to Singular's documentation (archived):

Parameter Description Supported Platforms Example
c Connection type 'wifi' or 'carrier'. iOS, Android wifi

We have observed the following values for this property in our request database: wifi, wwan

baltpeter commented 10 months ago

I'm done for now (#24). A few properties that I haven't included yet (mostly because we don't really have a way to express them yet) that we may want to look at in the future: