Explore Telus Insights API

[mikevespi]

We will want to begin a deep dive on the Telus Insights API immediately upon being granted access.

Key questions that need answers:

• what is the structure of the incoming data
• any batch restraints (max query size etc.)
• what aggregation has already taken place

As for the last point, we know that the data is heavily aggregated by Telus Insights, and possibly windowed (?). We need to know as much as we can about that process. This will likely require a conversation with someone from Insights, which we just need to ask for when we're ready.

As is always the case with discovery work, this list is incomplete. Make note of anything uncovered that could be helpful or relevant, and surface those findings with the team.

Be sure to document your findings, as we will likely need to present findings and our intentions in the near to immediate future, and will need the information for upcoming work.

Contact with J or Suhya to set up this meeting. Make sure that J has gotten the list of emails that need access to the API first.

Definition of Done:

• [ ] An understanding of what we have been given access to
• [ ] Determine if any information needed for ClimateTrax is missing (this includes more information on the processes that have been take on the data, not just the specific values needed for emissions calculations).
• [ ] Findings documented
• [ ] Follow up meeting with Insights scheduled if and when necessary

[joshgamache]

Added research below to Notion in the ClimateTrax board.

Insights API research documentation

Destination API data shape

Incoming data structure from /destination API point (sample data response). Parameters supplied to the API are input_params and it's children:

{
  "job_id": "33d2c589-fdb8-43fb-876c-3ff033526635",
  "api": "destination",
  "active": true,
  "input_params": {
    "output_geoid": {
      "study_zone": [
        "2021A00054806021-Airdrie",
        "2021A00054806016-Calgary",
        "2021A00054806012-Okotoks"
      ],
      "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd"
    },
    "input_geoid": {
      "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
      "study_zone": [
        "2021A00054806021-Airdrie",
        "2021A00054806016-Calgary",
        "2021A00054806012-Okotoks"
      ]
    },
    "end_time": "2020-12-31T23:59:59",
    "api": "destination",
    "time_bucket_size": 527040,
    "start_time": "2020-01-01T00:00:00"
  },
  "created_date": "2022-11-24T17:59:09",
  "status": "COMPLETE",
  "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
  "study_zones": [
    {
      "buckets": [
        {
          "timeframe_bucket": "2020-01-01T00:00:00",
          "outputs": [
            {
              "count": 845140,
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
              "output_geoid": "2021A00054806012-Okotoks"
            },
            {
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
              "count": 1473010,
              "output_geoid": "2021A00054806016-Calgary"
            },
            {
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
              "output_geoid": "2021A00054806021-Airdrie",
              "count": 80060
            }
          ]
        }
      ],
      "input_geoid": "2021A00054806012-Okotoks"
    },
    {
      "buckets": [
        {
          "outputs": [
            {
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
              "output_geoid": "2021A00054806012-Okotoks",
              "count": 1484670
            },
            {
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
              "output_geoid": "2021A00054806016-Calgary",
              "count": 4887650
            },
            {
              "count": 3291300,
              "output_geoid": "2021A00054806021-Airdrie",
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd"
            }
          ],
          "timeframe_bucket": "2020-01-01T00:00:00"
        }
      ],
      "input_geoid": "2021A00054806016-Calgary"
    },
    {
      "buckets": [
        {
          "timeframe_bucket": "2020-01-01T00:00:00",
          "outputs": [
            {
              "output_geoid": "2021A00054806012-Okotoks",
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
              "count": 79060
            },
            {
              "count": 3338760,
              "output_geoid": "2021A00054806016-Calgary",
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd"
            },
            {
              "count": 2839490,
              "requestId": "public-canada-csd-2021-ab-48-zip-a2388d8e-8e6a-4e53-9346-f9a12735ffbd",
              "output_geoid": "2021A00054806021-Airdrie"
            }
          ]
        }
      ],
      "input_geoid": "2021A00054806021-Airdrie"
    }
  ]
}

General Insights notes

• The API does not respond directly with the data requested. It is entered into a job queue, then you query against an endpoint in order to determine when the job's status is 'COMPLETE' (see the docs under Best Practices - Wait time before results and Getting started - Run a Job - REST API), which will then have all data.
• Data has already been anonymized and aggregated .
• API restrictions are listed under Introduction - API Integrations.
• We will need to provide our own boundaries and 'study zones', as the defaults do not include hex-tessellated zones. See Getting Started - Shapefiles and Study Zones.
• The endpoint relevant to us is the Destination API. While we have to make assumptions on this, we're able to acquire the data we need from it. See the API response above. Assumptions include:
  • Not knowing the mode of transportation of the devices counted (could be walking, public transit, car, truck). We will need to use some way to approximate breakdown of trips.
  • Not having absolute trip lengths. We won't know the path a trip took, therefore the trips will all be approximate. Trips from one hex to another will only approximate the distance between the hexes not the actual path.
    • We hope to address the above at long distances by using weighted connections between distant (i.e.. city to city) hex clusters.

OD Matrix endpoint

The OD Matrix endpoint was also tested for potential use, but was returning errors (for Josh) when attempting to use it at a year-long scale. Ticket has been submitted to Telus.