Closed hancush closed 3 years ago
This is high value documentation that I'd like to capture before November. I think a divide and conquer approach will be most expedient. Some questions I would find useful to answer:
I can head up 1 and 5. @fatima3558, do you think you could pick up 2 and 3? And @jeancochrane, could you do 4?
Also, are there other learnings we want to capture?
I would add "Debugging Airflow" to this list. Happy to pick this one up along with 4. I suspect @fatima3558 may not have the context for 3 so I'd be fine picking that one up, although if Fatima can tackle it I'd be glad to see someone else do it too.
I can definitely pick up 2, and I can give 3 a try as well!
Cool! I've updated my original comment to add debugging and assign everyone to their respective sections.
I've given no. 3 a shot, but I doubt what I've written in the google doc is everything we changed from default settings. I'm willing to do additional research about other default settings we changed for the dashboard and fill that section of the google doc further.
I've drafted 1. and 5. in the doc, as well!
Added 4 and 6! I think we're ready for a PR.
Converted the doc to Markdown and submitted a PR: https://github.com/datamade/how-to/pull/128. I think it would be good for the three of us to review the sections we didn't write before we submit this for final review.
I will open a seaprate issue for documenting leanrs from Meytro dashboar dreorg.
Documentation request
We're using Airflow to run the Metro scrapers and subsequent ETL! Let's capture this tool as part of our stack, and document lessons learned and any snake pits.