Status Updates – Navigator 2.0 Usability Research

[InMatrix]

We'll use this issue to share status updates of the Navigator 2.0 API usability research project.

Status Reports

• 2021-03-25 (Week 12)
• 2021-04-02 (Week 13)
• 2021-04-09 (Week 14)
• 2021-04-16 (Week 15)
• 2021-04-23 (Week 16)
• 2021-04-30 (Week 17)
• 2021-05-06 (Week 18)
• 2021-05-14 (Week 19)
• 2021-05-21 (Week 20)
• 2021-05-28 (Week 21)
• 2021-06-04 (Week 22)
• 2021-06-11 (Week 23)
• 2021-06-18 (Week 24)
• 2021-06-25(Week 25)
• 2021-07-02(Week 26)
• 2021-07-09(Week 27)
• 2021-07-16(Week28)

TODO List

Target completion dates are included in the parenthesis after each item, but they are subject to adjustments as we go.

• [x] Gather scenario code snippets for vrouter, beamer and auto_route (#9, 4/30)
• [x] Complete heuristic evaluations
  • [x] Conduct heuristic evaluation for vrouter (5/7)
  • [x] Conduct heuristic evaluation for beamer (5/7)
  • [x] Conduct heuristic evaluation for auto_route (5/7)
  • [x] Summarize heuristic evaluation findings (5/14)
• [x] Complete API walkthrough study
  • [x] Adapt walkthrough study plans for the 3 packages based on heuristic evaluation results (5/14)
  • [x] Conduct walkthrough studies for 3 packages (6/5)
  • [x] Report findings (6/11)
• [x] Write a final report synthesizing all the usability and technical evaluation results (6/25)

[InMatrix]

2021-03-25 (Week 12)

DONE

• Added code snippets for some routing scenarios
  • Router (Navigator 2.0):
    • Merged: deeplink-pathparam, deeplink-queryparam, dynamic-linking
    • In review: login, skipping stacks
  • VRouter
    • In review: deeplink-pathparam, deeplink-queryparam, dynamic-linking
• Exercised the API usability evaluation guide on Navigator 2.0 using the “deep linking by path parameter” scenario. This allowed us to better operationalize the evaluation process. We’ll post the evaluation results to the Github and make minor revisions to the evaluation guide next week.

NEXT STEPS

• Conduct an end-to-end research pilot: Our immediate next step is to prove out our research methodology by exercising it end-to-end on a single package, vrouter, and using just a couple of scenarios as soon as possible. This pilot will include a heuristic evaluation based on the API usability evaluation guide and a couple of user study sessions. We will make improvements to the study protocol through this pilot and get a better sense of what we might learn from this exercise before applying the same process to more packages and scenarios. We aim to complete this pilot and share what we learned by mid-April.

• Select packages for the full study: this project has surfaced many routing packages from the community. However, our bandwidth won’t allow us to properly study all of them in detail. Our tentative plan is to select 3 packages in the full package comparison study (#7). We outlined a tentative selection criteria in #10. Feel free to advocate for a package and explain why the package meets most, if not all of the criteria. We aim to design the study in a way that a motivated community member can contribute to the analysis for additional packages in the future. We aim to complete the full study by mid-May.

[theweiweiway]

@InMatrix

Just throwing this out there - happy to provide code snippets and such for auto_route, especially since I'm in contact with Milad-sama as well

[InMatrix]

2021-04-02 (Week 13)

DONE

• Added additional code snippets (#9)
  • Router (Navigator 2.0): Merged snippet for the sign-in routing and skipping stacks scenarios. The snippet for nested routing is in review.
  • VRouter: Merged snippets for the deep linking and dynamic linking scenarios. Snippets for the sign-in routing and skipping stacks scenarios are in review.
  • Beamer: Snippets for the deep linking and dynamic linking scenarios are in review.
• Added flow_builder and navi to the comparison table
• Made progress on the research pilot
  • Completed the first user study session on vrouter using its implementation of the deeplinking by path parameter scenario
  • Conducted a heuristic evaluation of vrouter
• Opened a discussion about additional requirements for Nested Routing

NEXT STEPS

• Complete the research pilot
  • Run another user study session
  • Review pilot results and revise the study design
• Select packages for the full study (described in last week’s status report)

[xuanswe]

NEXT STEPS

• Complete the research pilot

  • Run another user study session
  • Review pilot results and revise the study design
• Select packages for the full study (described in last week’s status report)

@InMatrix I need 1 or 2 days to release Navi 0.1.0 with completed code snippets for all scenarios and a clear architecture layers. Is this not too late for NEXT STEPS you mentioned?

[InMatrix]

@theweiweiway Feel free to add code snippets for auto_route to the scenario-code folder. We've also been in touch with @Milad-Akarie.

@nguyenxndaidev It's definitely not too late. However, submitting code snippets doesn't guarantee that we will be able to use them in the full evaluation. We still haven't yet go through the selection process outlined in #10. I hope the snippet writing exercise is worthwhile in itself for the purpose of reflecting on your API design. :-)

[xuanswe]

We still haven't yet go through the selection process outlined in #10.

@InMatrix Thanks for the info! Could you please update your plan, when will you start #10?

[xuanswe]

@InMatrix I released Navi 0.1.0. Refactoring the given code in scenario-code folder and will create a PR with the new files.

[xuanswe]

@InMatrix PR https://github.com/flutter/uxr/pull/38 is created.

[mariamhas]

2021-04-09 (Week 14)

DONE

• Updated API Usability Evaluation Guide based on experience applying it on the Router API
  • Changed rating scales from 3 points to 5 points
  • Added examples for carrying out specific steps
• Pilot study - completed 2 additional user study sessions using snippets from vrouter

IN-PROGRESS

• Started a process to select 3 packages we want to focus on in the next stage of research. More information will be available in issue #10
• Finishing up the evaluation of vrouter as part of the research pilot

NEXT STEPS

• Implement a version with transitions for nested routing scenario using Router based on discussion
• Review code snippet pull requests
• Revise the user study design based on pilot study results
• Share pilot study results
• Align on project schedule/timeline with the goal to complete study by EOQ2

[mariamhas]

2021-04-16 (Week 15)

DONE

• Posted a tentative decision about packages we plan to focus on in the next round of evaluation (#10).
• Posted a public update about API Walkthrough study pilot interviews.
• Added beamer snippets for 3 scenarios
• Added vrouter snippets for 2 scenarios
• Reached a resolution about the nested routing scenario
  • The initial version of the snippet written with Router (https://github.com/flutter/uxr/pull/34) will preserve the tab selection state but will not support transition animations between tabs.
  • There are a few advanced features of the nested routing scenario we’d encourage package authors to consider separately from the “base version” as implemented with the Router API. They include: custom transition animations between tabs, preserving scrolling positions, and preserving text input state. More discussions about those additional requirements are being held in issue #35.

IN-PROGRESS

• Finalize the study plan for the API Walkthrough study.
• Nested routing sample snippet - exploring adding transition animations between tabs

NEXT STEPS

• Get initial version of nested routing sample snippet merged
• Conduct heuristic-based usability evaluations of 3 packages (currently blocked on nested routing snippets)

[InMatrix]

2021-04-23 (Week 16)

DONE

• Merged the Router snippet for the nested routing scenario (#34)
• Discussed preliminary heuristic usability evaluation results for vrouter.

IN-PROGRESS

• Preparing heuristic evaluations for vrouter, beamer, and auto_route. This is currently blocked by #9.
• Working on specifying how to evaluate API viscosity. Please add your thoughts to #41.
• Converting the snippets folder into a demo app (#42)

NEXT STEPS

• Please see the TODO list posted in the top comment.

[mariamhas]

2021-04-30 (Week 17)

DONE

• Updated the instructions about assessing API Viscosity based on this discussion.
• Merged beamer snippets to be used for heuristic evaluations
• AutoRoute snippets also added
• Opened Discussions feature for the repo for open-ended topics

IN-PROGRESS

• Working on walkthrough study plan
• Reviewing the PR from Navi

NEXT STEPS

• Complete heuristic evaluations by 5/7 for VRouter, Beamer, AutoRoute

[InMatrix]

2021-05-06 (Week 18)

DONE

• Snippets for AutoRoute and Navi have been reviewed and merged. Thanks @theweiweiway and @nguyenxndaidev for your contributions.
• Tweaked the research design of the the API Walkthrough study based on discussion in #40.

IN-PROGRESS

• Working on heuristic evaluations of VRouter, Beamer, and AutoRoute following the API Usability Evaluation Guide.
• Recruiting participants for the API Walkthrough study.

NEXT STEPS

• Synthesize heuristic evaluation results and generate package-specific research questions for the walkthrough study.

[mariamhas]

2021-05-14 (Week 19)

DONE

• Completed heuristic evaluations of VRouter, Beamer, and AutoRoute following the API Usability Evaluation Guide.

IN-PROGRESS

• Synthesize heuristic evaluation results and generate package-specific research questions for the walkthrough study.
• Recruiting participants for the API Walkthrough study.

NEXT STEPS

• API Walkthrough user study - starting with VRouter

[InMatrix]

2021-05-21 (Week 20)

DONE

• Conducted 3 sessions for VRouter API walkthrough study
• Consolidated heuristic evaluation results

IN-PROGRESS

• Preparing for Beamer and AutoRoute study sessions

NEXT STEPS

• Complete VRouter API walkthrough study (2 more sessions to go)
• Finalize an outline for the final study report

[creativecreatorormaybenot]

Hi, is there any timeline for the following goal stated in the wiki?

Design or endorse an easy-to-use package for implementing common navigation patterns, especially for use cases on the web.

I am having trouble connecting the status updates to that.

[InMatrix]

@creativecreatorormaybenot Hi, thank you for your question. This project has been taking longer than we initially anticipated. Our current projection is to complete usability studies of the 3 packages on our shortlist by the end of this quarter. We aim to publish a report with our findings and recommendations for choosing routing packages in July. Thank you for your patience.

[mariamhas]

2021-05-28 (Week 21)

DONE

• Completed VRouter Walkthrough study with 4 external participants

IN-PROGRESS

• Preparing Auto-Route Walkthrough study - starting next week

NEXT STEPS

• Beamer Walkthrough study - reviewing quiz questions and recruiting participants

[creativecreatorormaybenot]

Thanks a lot for the answer @InMatrix 🙏 I do not want to escalate the comments in this update thread, so one last clarification on the final status update / end goal:

We aim to publish a report with our findings and recommendations for choosing routing packages in July.

Am I right to assume that this means that the navigation in the Flutter framework will not be changed anymore, i.e. there will not be a Navigator 3.0 or a flutter/packages but rather the final step of the work tracked here is endorsing a community package?

[InMatrix]

@creativecreatorormaybenot As far as I know, there is no active discussion nor work about changing the built-in Navigation API in any significant way right now. It is our intent to provide guidance to Flutter users on choosing from existing navigation and routing APIs. Whether the Flutter team will endorse a single community package for navigation in the manner of endorsing Provider or make a more nuanced, conditional recommendation depends on the results of the ongoing evaluation.

Will there ever be a Navigator 3.0 at some point in the future? I simply don't know, since the timeframe is unspecified.

[mariamhas]

2021-06-04 (Week 22)

DONE

• Completed 3 API walkthrough sessions for AutoRoute
• Summarized a few VRouter sessions

IN-PROGRESS

• Scheduling additional sessions for AutoRoute and Beamer
• Working on a fix of Browser state: https://github.com/flutter/flutter/issues/80546

NEXT STEPS

• Summarize API walkthrough study results
• Write final research report

[mariamhas]

2021-06-11 (Week 23)

DONE

• Completed 4 sessions for VRouter, 2 for AutoRoute and 3 for Beamer
• Fixed, not yet merged Browser state: https://github.com/flutter/flutter/issues/80546

IN-PROGRESS

• Additional sessions for AutoRoute and Beamer next wekk

NEXT STEPS

• Summarize API walkthrough study results
• Write final research report

[InMatrix]

2021-06-18 (Week 24)

DONE

• Completed all 15 API walkthrough study sessions (5 for each of the three packages evaluated). Yay!

IN-PROGRESS

• Summarize API walkthrough study results

NEXT STEPS

• Write final research report
• Plan reviewing results with the Flutter team and package authors

[idkq]

Not sure if this is the right forum but seems like there is bug on the Beamer snippet.

Repro

`flutter run -d Chrome -t lib/deeplink-queryparam/deeplink-queryparam_beamer.dart --web-port=51513` Paste in browser bar and hit enter (TWICE - two times): http://localhost:51513/#/?filter=land http://localhost:51513/#/?filter=land (again) Filters disappears

Doctor

[✓] Flutter (Channel beta, 2.2.2, on macOS 11.4 20F71 darwin-x64, locale en-US) [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3) [✓] Xcode - develop for iOS and macOS [✓] Chrome - develop for the web [✓] Android Studio (version 4.1) [✓] IntelliJ IDEA Ultimate Edition (version 2021.1.2) [✓] Connected device (2 available) • No issues found!

[InMatrix]

@idkq Would you mind opening a new issue in this repo and tag Beamer's author?

[slovnicki]

@idkq it has been solved - #69 (@InMatrix I don't have a permission to close that issue, feel free to close it)

[mariamhas]

2021-06-25 (Week 25)

DONE

• Summarized all 15 API walkthrough study sessions (5 for each of the three packages evaluated). Yay!

IN-PROGRESS

• Write final research report

NEXT STEPS

• Plan reviewing results with the Flutter team and package authors

[mariamhas]

2021-07-02 (Week 26)

DONE

• n/a

IN-PROGRESS

• Writing final research report, targeting to complete in 2 weeks.
• Preparing an update for a regular sync with Flutter GDEs next week

NEXT STEPS

• Plan reviewing results with the Flutter team and package authors

[idkq]

Hi @InMatrix will the community & non-pkg-authors be able to review and contribute to the final report by any means? I assume that the report, when completed, will be published somewhere in the flutter website? I feel that despite the great job, some questions can remain open and shall be addressed in the future, so the community could help ensuring and capturing that in the report - if a draft is provided.

[lulupointu]

@idkq I believe anyone can contribute the the Flutter website by submitting PRs to https://github.com/flutter/website

[mariamhas]

2021-07-09 (Week 27)

DONE

• Presented updates on research to Flutter GDEs

IN-PROGRESS

• Writing final research report, targeting to complete in 1 week.
• Preparing to share draft report with package authors to get feedback

NEXT STEPS

• Finalizing report to share with community

[InMatrix]

@idkq The current plan is to share a mostly complete draft of the research report with the authors of the three packages we evaluated in detail for their comments and corrections first. And then we'll socialize the report internally with stakeholders on the Flutter team to discuss next steps. The final version will be shared with everyone, but we might not amend the report based on public discussions except for obvious factual or writing errors. This report will be an artifact reflecting our understanding about the issue of designing high-level routing APIs at a specific point in time. Nonetheless, responses, counter-arguments, and follow-up research will be welcomed, and they can be posted in the discussion section of this repo or somewhere else.

[rrousselGit]

Late to the party

I have some WIP about a different approach on navigation. Is it too late to hop in the discussion and share some snippets?

[slovnicki]

Welcome @rrousselGit I think the best place to leave your thoughts is #6

[mariamhas]

2021-07-16 (Week 28)

DONE

n/a

IN-PROGRESS

• Writing final research report, draft almost complete and will soon be ready to share with package authors

NEXT STEPS

• Finalizing report to share with community

[ajnaf]

2021-07-16 (Week 28)

DONE

n/a

IN-PROGRESS

• Writing final research report, draft almost complete and will soon be ready to share with package authors

NEXT STEPS

• Finalizing report to share with community

Can you at least give us which package is strongly recommended so we can start using it while waiting for the details

[InMatrix]

@ajnaf We're not ready to make a recommendation yet. Sorry, this has taken a long time. If you need to decide which package to use now, I'd suggest you take a look at the example code in this folder. You will be able to get a sense of how well the scenarios you care about can be implemented by different packages.

[alsoLut]

when will this work be finished?

[InMatrix]

@bogdanmarinescu We're getting close. Last week, we shared a draft of the research report with authors of VRouter, AutoRoute, and Beamer. We're in the process of addressing their comments. We're looking to release the report within a couple of weeks. Thanks for your patience!

[PedroAlvarado]

@InMatrix Short of the full report, it'd be great to take a look at the draft report with the understanding that it may differ from the final report.

[InMatrix]

The research report is available to download. Please use this discussion thread to post any questions or comments you might have. Thank you for your support and patience!