Friction Log for Quickstart Section

[zkamvar]

On 2024-07-16, I walked through the quickstart document and wrote a friction log (long description of a friction log and this shorter description with clear, but non-trivial example) to highlight how easy it was for me to walk through the quickstart.

📛 Who’s doing this?	Zhian Kamvar
📅 Date of friction log	2024-07-16
💡 What’s the use case?	Quickstart to create a new Forecasting Hub
🖥️ OS System	Ubuntu Linux 22.04
🌈 Logging key	(👍🏽): Awesome, easy (🤔): Hmm, bit tricky but getting through it. (🔥): I want to throw my computer out the window

Steps

These steps come after reading through the Overview section.

• 👍🏽 Go to https://hubverse.io/en/latest/quickstart-hub-admin/intro.html and see that there are 7 concrete steps.
• 🤔 reading through and clicking on the header link in https://hubverse.io/en/latest/quickstart-hub-admin/intro.html#template-hubs, takes me to a new page
• 🤔 there are no obvious differences between the page for "Template Hubs" and "Example Hubs" because GitHub lists the pinned repositories first and I have to scroll down to view the filtered repositories
• 🤔 skipped the rest of the page because it doesn't seem to be moving me forward in creating a hub

Getting Started

• 👍🏽 create GitHub account (I had one already)
• 👍🏽 created template repo
• 👍🏽 clone repo

Setting Up

• 🤔 There is a link to the structure of hub repositories with several other links in that page... I just want to quickly set up a hub and get it running so I skip this.
• 🤔 There is missing step for "Hub Administrative Configuration". The last step says "examine the file", but it does not say to edit it.
• 🤔 There is a purple highlighted word in the GitHub URL for the image that is difficult to read.
• 👍🏽 I edit admin.json with my information

Configuring tasks

• 👍🏽 I can open and view the tasks.json file
• 🤔 It's weird that I have to download the premade tasks file from an example repository. NOTE: not sure what this file actually means yet.
• 🤔 round_id and origin_date: What happens if round_id_from_variable is false? Why is required null and not false? Why are these dates not required?
• 🤔 Setting the "target": What does "target" mean in this context? Is "inc covid hosp" specific to the data? The second bullet second sentence is confusing. Where can we add ["cum covid hosp"]?
• 👍🏽 Setting the "horizon": The definition here is good
• 👍🏽 Setting the "location"
• 🔥 "required" and "optional" elements: It took me a few reads to understand that target, horizon, location, and origin_date were all subsets of a task_id. I think a diagram of the schema relationships is warranted here.
  • Might be a good idea to highlight what we are not setting for these models.
  • task_ids says that the elements "such as" the list of valid target, horizon, location, and origin_date... does this mean there are more of them?
• 🤔 Setting the "mean": The introduction of ["NA"] doesn't make sense to me. Why is it here?
• 👍🏽 Setting the "quantile"
• 🤔 Defining "target_metadata": the highlighting is more distracting than helpful here. Simulating various modes of colorblindness with the browser shows that the highlighting loses semantic meaning.
• 👍🏽 Set up "submissions_due"

Configuring Model Metadata

• 🤔 This section mentions YAML, but then shows JSON schema and never brings up YAML again
• 👍🏽 Uploading changes to GitHub
• 🤔 Validating config files
  • 👍🏽 installed hubAdmin package
  • 🤔 validate_hub_config() fails for metadata schema because some elements need to be arrays
  • 👍🏽 validate_config() works
  • 🤔 validate_config_val_errors() fails with a cryptic "x" not found error
  • 🤔 Try to pass result of validate_hub_config() to validate_config_errors() gives an error

Setting Up Continuous Integration

• 🤔 I was able to get this set up well enough, but it's not clear what GitHub actions are needed.

Using the modelling hub

This information would have been useful to highlight at the beginning of the tutorial.

Thoughts

We need Personas

Overall, I was mostly confused as to what a hub was used for. It was not clear to me who the audience for this particular section was. I now understand that it's for hub administrators, but this is still a bit of a fuzzy definition for me. It would be useful to have a set of personas that describes the people who would want to learn about or use hubs. Greg Wilson often has pretty good examples of personas for his books: https://third-bit.com/sdxpy/intro/#intro-audience

Show the cake first

The quickstart was confusing for me because I was starting from no previous knowledge of these hubs. It was not clear how these hubs would be used. The last chapter helped demystify that. It would help to have an exposition of a valid hub early on, which would help deliver visible results early so that the learners can build a mental model of what components the hub has and how they work in the day-to-day operation of the hub.

The figure in https://hubverse.io/en/latest/user-guide/tasks.html is a really good example of a concept map for the hubs that provides context for who provides what in the hubs. Having this at the beginning of the tutorial with how that maps to a hub folder structure would go a long way.

Definitions should be in context of what they are defining

The key definitions are helpful, but because it's a marching list, it's difficult to remember how they relate to each other. It would be more helpful to define these in context for the persona who will be interact with these definitions.

Define the learning goals for the readers

I think this is somewhat achieved with the list in the introduction, but it could be higher-level and more focused. There were parts of the quickstart that did not feel quick and felt a bit irrelevant. Having learning goals and creating a concept map that will make developing an effective tutorial easier.

[micokoch]

This is great, and very helpful @zkamvar . I like the friction log and it's very clear. I'm not sure how you prefer to address the issues you raised. It seems you already started addressing some of it in this discussion. I am happy to help, but I think there needs to be some coordination to address the various issues. All I mean to say, is let me know if you would like me to help addressing any of this, or otherwise, I will assume you are doing it.

[zkamvar]

Thank you for following up and for patiently reading the log!

I'm not sure how you prefer to address the issues you raised. It seems you already started addressing some of it in this discussion. I am happy to help, but I think there needs to be some coordination to address the various issues.

I want to apologise for opening this as an issue and not a discussion, because a friction log is a way to reveal bumps in the road that can be eventually resolved. It's not so great as a standalone issue because it requires discussion to identify the individual pieces that need to be addressed.

Regarding logistics: Would it be better for me to move this issue to a discussion in https://github.com/orgs/hubverse-org/discussions and close it so that it doesn't overshadow any issues that are more actionable in hubDocs?

---

My strategy here was to tackle the first piece of documentation a user might see from the perspective of someone who has seen the outputs of a hub, but has never actually used the hubverse before (It's like trying to find the supplies for making coffee in a dark and unfamiliar kitchen by the light of your phone).

Part of my hope for this (and future) log is that someone else will see some of the 🤔 or 🔥 points and say "oh yeah, I've run into this before," which will help identify low-hanging fruit (One that comes to mind is setting the optional output type id for mean to "NA", which I later saw was a question from another team member in slack some time ago).

[micokoch]

I think it would be good to leave this post here. I don't look at the hubverse discussion board often, and many (perhaps most) of these issues are ones that Melissa and I (and perhaps you) will try to tackle. We usually look for our next tasks looking at the Documentation board.

I suggest we leave this here, and if one of us wants to tackle a specific item, they can open an issue about it, so we know someone is working on it. For example, if I want to address one of the YAML - JSON confusions you mention, I will make an issue about it. Or if you want to create personas, you could make an issue so that we know you're working on it.

Does that work for you?

[zkamvar]

That works for me!

[annakrystalli]

This is super useful @zkamvar !

I think many of these issues might best be tackled as a pair programming challenge together with someone who can answer all the questions raised as sth like a scheduled documentation sprint.

If @nickreich agrees, I would be happy to schedule 2-3 hrs in the next couple of weeks to work on this together.