[Task]: Data Sources Discovery

[acouch]

Summary

This task is to do a time-boxed discovery on the datas sources that we are targeting to use for the project within the next year. The goal is to get a high-level assessment of the types of data sources, the size and storage requirements, and to surface large questions that need to be answered if there are any.

Data sources we will be looking at:

• Google analytics

• Grants program data

  • Grants opportunities - oracle database
  • Applications - S3 bucket
  • XML data

• USA spending data - csv

• Communication platforms

• Slack stats

• Google group stats

• GitHub stats

• Mongo chat logs?

• API application metrics - NewRelic or S3

• API infrastructure metrics - Cloudwatch

Acceptance criteria

• [ ] A public gitbook page that provides more details about each data source with a high-level estimate of the periodicity, data type (csv, json etc), data size, data dictionary if possible, etl considerations
• [ ] A recommendation for data storage, captured in #1506

[widal001]

@AlexanderStephensonUSDS Can potentially help on this! Though he might not have access to most/all of these data sources.

[acouch]

@ebuwa-evbuoma-fike I created this before you started your more-focused discovery process. Should we close this one?

[ebuwa-evbuoma-fike]

@acouch That makes sense, since we are taking an MVP approach to analytics deliverables.