Write high-level introduction to MagicBox and how it's built

[jwflory]

Summary

Write one or two page doc to explain MagicBox projects, their relationships, and how it all fits together (complements #15)

Background

There's a lot of repos and they're listed in the project index, but that page isn't helpful. It gives us a bunch of "dots" but not how they're connected. It's confusing for a newcomer to understand how the projects fit together, what each one does, and why they are important.

Details

A one or two page document to explain that pipeline, or flow, of how the repos are connected and how the data is processed will be helpful. I am writing a short document to give that explanation / introduction. The purpose is that we could share this with anyone and they could get a brief understanding of the project and how it fits together.

I plan to use past Medium articles and some of the info on the unicefstories.org site to create this.

Outcome

Easier to understand the project, relationship among repos is more clear, easier to introduce someone to the projects

[jwflory]

I shared a survey about MagicBox with people I met abroad in Europe over the last two weeks. From their feedback, I've accumulated a "question queue" of things people were confused by or were curious about (also found in the documentation project board). I plan to map out a strategy to answer these questions over the next week.

Big picture

• What is MagicBox?
• Is MagicBox a public API?
• How do I access MagicBox?
• What's the two-year vision for MagicBox?

Use cases

• What data does MagicBox provide?
• Who would find MagicBox useful?
  • Who are our current users?
• What are some live examples that use it?
  • What use cases has MagicBox actually been used for? Zika?
  • Are we able to track the Zika virus risk factor now? Or is it manually generated with our current data?

Technical

• How to integrate more countries?
• What are the "core" repos to MagicBox?
  • Which repos play a supporting role to MagicBox, but aren't critical?
  • How do the projects come together?
• Where does the data come from?

Contributing / community

• How do I contribute?
• What actual development work needs to be finished?

[jwflory]

I reviewed many of the Medium blog posts that @mikefab has written and I think a lot of the high-level overview exists throughout those posts. I want to help lift some of those things out of Medium and into documentation proper so they're public and centralized to a single place. This will likely be some of my work for the rest of this week and into next week.

[jwflory]

Today, I took a deep dive into one of the longer, private posts on Medium and did a cross-section with the questions generated above. I grouped my research and questions into content sections listed below. Each section represents a unique article.

About MagicBox

• What MagicBox is for
• How do projects come together

MagicBox data

• Where does data come from?
• How do we get / ingest data?
• Managing large sets of school data
  • Our CSV files are huge!
• Data ingestion rules
• Project Connect
  • Ruby on Rails CRUD (create, read, update, delete) app to upload data
  • PostgreSQL with single table
  • File name dichotomy
  • Data inserted when imported (how we add new columns to school data)
• How to integrate more countries (help us!)

Data validation

• magicbox-latlon-admin-server
  • Why we need to validate data
  • How we validate data today

Serving data

• Schools / mobility served via magicbox-open-api
  • School data is sensitive (e.g. terrorism)
• API has three endpoints
  • Response: country list
  • Response: schools in country
  • Response: school
• API problems in hindsight
• Access control to-dos

Data visualization

• Visualizing data in OSM Leaflets
• Why we map the way we do
• Authentication: Auth0
  • Tokens with API queries

[jwflory]

Documentation progress is in #21, #22, #23, and #24. Two candidate articles remain, one for access control and one for rules. Aiming to finish this by the end of the week.

[jwflory]

Rules are explained in #25 and access control in #26.

Marking ticket as blocked until @mikefab returns and has a chance to review the above PRs. Once he reviews the existing content, we can look at expanding our coverage and writing about some of the missing sections.

[jwflory]

@mikefab reviewed all the open PRs last night and they are now merged in the repository. The initial work of creating some high-level documentation on MagicBox is done. The next steps will be to figure out gaps in our documentation and address them.

We will do this when @mikefab returns from the Colombia visit next week.

For now, check out the awesome docs!

magicbox.readthedocs.io

Closing this as complete.

[thoat]

Each article on the current Docs is wonderful in itself - so detailed, carefully curated and thoughtfully visual. However, as of now, the Docs as a whole is a bit difficult to read because there is no clear pointer as to how each article connects to one another or how should a reader flows from one to another. (If you approach the Docs from the Navigation Menu on the left, you'll get what I mean.)

However, I think the outlines @jwflory has above (on Feb 8 and Feb 21) provide an excellent flow, and they have been broken into tasks here and here. Once the tasks there are largely completed, the Docs will read more intelligibly.