Investigate data storage size, cost and storage alternatives from TransitFeeds

[emmambd]

Describe the problem

As part of adding the historical data on TransitFeeds to the Mobility Database , we need to understand

• The costs associated with the data
• How best to store the data
• Where we should "tier" the data (e.g provide data from 3 years ago self-serve, data earlier than that will require emailing us and manual work to extract it from a less cost intensive storage centre).

We have just under 10Tb of files from TransitFeeds.

Proposed solution

We need to evaluate several different storage options (AWS, GCP, etc) and assess

• Total size and cost of storage
• Size and cost of storage from 2023 (1 year option)
• Size and cost of storage from 2020-2023 (3 year option)
• Size and cost of storage from 2018-2023 (5 year option)

Alternatives you've considered

No response

Additional context

• Discuss updating downloading datasets from 1 week to 3 days (licensing)

No response

[jcpitre]

Here is a summary (and raw data) of what is on Transitfeeds as of 2013-12-08: https://docs.google.com/spreadsheets/d/15UmhrhS-E0sxpzU9w3bQo45PkyBzw7zAJjz7Ug4RZY4

[jcpitre]

See this for report.

[emmambd]

@jcpitre Thanks so much for your super detailed analysis! This is great work and very helpful. Based on your Conclusion and a review of the options, it sounds like

• Based on the current storage needed (2800 GB), there's no major cost differences that make it obvious how many years to use as the cut-off for self-serve access. (The costs are fairly similar).
• I'm thinking there's value going forward in having tiering as our archive grows over time and costs increase. We may be able to roughly estimate the increase of our storage from month to month once the API is launched and we're regularly archiving data.

I propose we revisit this analysis a month after the Mobility Database API v1 is launched (so I'll book a holding time during quarterly planning in March) and we can talk about how to manage costs of our infrastructure overall.

This is assuming there are no urgent reasons why we need to revisit our technical budget. Based on what I've been told, there isn't at present. cc @isabelle-dr @davidgamez

Decision:

• Book meeting at quarterly planning to discuss + plan to act on this insight starting in quarter 2 of the year.

If there are any concerns with this timing or insight I'm missing, please let me know!

[isabelle-dr]

This sounds great, thank you for such a transparent & data-driven approach to making this decision.