cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services
17 stars 2 forks source link

Project: TPW Data Ecosystem Map #12427

Closed rebekkabry closed 8 months ago

rebekkabry commented 1 year ago

Ecosystem Map is an enterprise-wide view of the data sources that support our business dashboards and provide insight into operational performance.

Background

Prior to the TPW merger, PW and ATD respectively have developed a variety of datasets, data source locations, operational tools and dashboards that provide insight into business activities and consideration for business initiatives toward changes. Currently there is no standard data documentation or visibility into the data pipelines that support our major dashboards and tools. The ideal behind the Ecosystem map is to provide documentation to support current tools, provide standardized data mapping documentation standards, and to provide stakeholders with a centralized “themed” platform to access all dashboards, reports, resources related to a specific workgroup or tools.

This map and documentation will be the backbone for current and upcoming projects. Such as: Dash, TPWStat, 311 integrations, etc.

image

Scope & Deliverables

Due to the large, complex nature of this initiative, we are planning to address this initiative in 4 phases:

Phase 1 – Collecting Start to End

Phase 2 – Current State Mapping Dashboard/Tools Level

image image

Phase 3 - High Level Data Relationships between the stakeholders

Phase 4 – Centralized Platform Build Out

Desired Outcomes

  1. Improve access to data and data related resources by providing a centralized platform linked to the appropriate workgroup or work function.
  2. Improve stability of data pipelines by providing insights and documentation into the data paths, especially for dashboards and other reporting tools.
  3. Promote a culture of standardized data documentation by providing data documentation templates/examples.

Timebox

This is an iterative project that will accumulate data from different divisions as data owners become available to walk through the pipeline and variations of data collected.

Concepts and references

What is a data ecosystem map?

A data ecosystem map illustrates the different actors in a data ecosystem, and how value is exchanged across it. A data ecosystem consists of data infrastructure, and the people, communities and organizations that benefit from the value created by it. Data infrastructure is made up of data assets, standards, technologies, policies and the organizations that steward and contribute to them.

Why use the tool of an ecosystem map? Creating a data ecosystem map helps to understand how data creates value. It identifies the data, data stewards and data users; the different roles they play; and the relationships between them. You can use your map as a practical tool to plan and visualize a data ecosystem, or show opportunities for increasing value to particular parts of a data ecosystem. A data ecosystem map can be used to:

Opportunities and Value

Request details

https://atd-dts.gitbook.io/wiki/product-ops/dts-website#images The first image will be used as the project's thumbnail image on the website.


This Github issue represents a project of Austin Transportation's Data & Technology Services team. Project status is documented regularly in the comments below.

rebekkabry commented 1 year ago

We're building a high-level dashboard that synthesizes real-time data to visualize the work and goals of TPW and its divisions.

Milestones

Phase 1: Current State

[] gather ATD data sources [] gather PW data sources [] link to organization divisions, departments and/or programs TPW [] investigate missing areas or gaps

Phase 2: Gap Investigation

[] TDB

Outcome:

[] compiled list of TPW data sources [] diagram that shows all our applications and how data flows between them and the open data portal [] automated updates for https://atd.knack.com/dts#open-data-inventory/

[] find trends, relational concepts, commonality and story value [] list of assets for TPW? Possible github type record and note sharing?

Opportunities

[] build standardized reporting or data sets needs across business function areas [] insight into overlaps and/or gaps [] insight into executive main interests and/or gaps

rebekkabry commented 1 year ago

Considering a survey to send out to all division managers inquiring about data and reporting current details. To include items such as:

Get feedback from Amenity on best approach and /or additional questions to include in survery

rebekkabry commented 1 year ago

Current Compiling Views: image

7/6/23 - Meeting with Amenity to review MVP.

Questions to Answer:

Notes:

rebekkabry commented 1 year ago

Status Notes: Map of applications/products to Data Sources

image image

rebekkabry commented 1 year ago

Working Documents

Miro Board Excel Mapping PPT

rebekkabry commented 1 year ago

Organization Function > Division > Output (in progress)

image

rebekkabry commented 1 year ago

image

rebekkabry commented 1 year ago

DTS | Data and Technology Services Portal Knack review and model build out - in progress.

image

rebekkabry commented 1 year ago

Applications Insights - based on the current "applications form"...

image

Fields / Filters to Consider

  1. Environment: contains Production

    What doesn’t make it to list show up on test or prototype list

  2. Application Class Clean Up/BreakDown

    For OPSDB or 3rd DB; filter Application Class: Enterprise, Database - Other, Database Server, Web Application - Other, Device-Specific Firmware/Software, Enterprise Business System For OPSDB Local Fles; filter Application Class: Computer File For Knack Application Tool OPSDB; filter Application Class: Knack Application For OutputAccess; Filter Application Class: ArcGIS Online Web App, ArcGIS Online Feature Service For OutputAccess: note ArcGIS Online Feature Service has layers URL, how to include, metadata export available?

    1. Note some databases may need to be added manually for instance Hanson - SBO

Results/Impact

  1. Allows set up for Hierarchy/Levels/Nodes based on role

    Roles: Operations Tools (3rd party, DTS Supported, CW), ETL in, Storage, Storage Access/Dataset Details, ETL out, Output, Output Access

  2. Allows clarity at highest view of the role and type of asset
rebekkabry commented 12 months ago

Knack Dataset Insight Deck

Knack Application Insight Deck

rebekkabry commented 11 months ago

Proposal

current Knack connections image

proposed Knack connections image

rebekkabry commented 10 months ago

Current Support of TPW Departments and Programs

Divisions Role
Finance  
Human Resources  
Data & Technology Services  
Performance Management  
Parking Enterprise  
Public Info PM  
Administrative Manager  
Legal Affairs  
Street and Bridge Operations Service
District Maintenance Service
Utilities & Structures Service
Pavement Operations Service
Urban Forestry Service
Transportation Engineering Service
Capital Services/ Project Delivery Service
Active Transportation Engineering Services Service
Vision Zero - Transportation Service
Signs and Markings Service
Arterial Management "Signals" Service
Office of Special Events Service
Right of Way (ROW) Management Service
Office of the City Engineer Service
Sidewalks and Special Projects Service
Community Services Service
Emergency Management Service
Logistics Service
Capital Program Consultant Strategic Projects  
Public Info & Mktng PM Strategic Com & Outreach  
Smart Mobility/ TDM  
Long Range Planning  
Transportation Development Services  
Capital Program Conslt. Program & Project Development  
City Wide  
Divisions   Power BI PW Power BI Knack ArcGIS TPW Stat SD23/Pwstat
Total Count Supported   6   16   17  
Total Count Supported %   19%   50%   53%  
Department Count (all)   32    
Service Departments (only) Count   18    
Service Count Supported   5   7   14  
Service Count Supported %   28%   39%   78%  
amenity commented 10 months ago

Moving this out of "Ongoing" to "In Progress" since it is a project.

rebekkabry commented 10 months ago

Summary Report 1/11 with DTS Management:

rebekkabry commented 9 months ago

Updates: