Let's Think over UX of ONE-vscode

[hyunsik-yoon]

So far ONE-vscode has been developed, focusing on functionality of tools we provide.

It's time to think over UX that embraces user's needs.

[hyunsik-yoon]

How about following designing thinking framework? (https://www.justinmind.com/blog/design-thinking-process-ux-design/) The process is as follows:

• step 1. Empathy
  • Collect user activity
• step 2. Define
  • Find problems
• step 3. Ideate
  • Perform brainstorming to solve the problems and select good solution
• step 4. Prototype
  • Make prototype
• step 5. Test
  • Let users use prototype. Collect their feedback.

I am not sure if we could finish step 4 and and step 5. However, thinking over step 1 to 3 would also be very valuable.

[hyunsik-yoon]

Today, we (@hyunsik-yoon, @jyoungyun, Yunjae Hong) discussed 1. Empathy.

1. Empathy

In this process, user's activities towards compiling model need to be collected. The following is the list of who/what/how/why/issue.

Term

• model source code: python code for PyTorch or Tensorflow model
• target code: compiled model for specified backend (CPU, GPU, NPU, etc)
• model file: onnx, tflite, pb, pth file, which can be compiled to target code

end-user

• Developers who has little knowledge about on-device inference
  • having experience with programming language(s)
  • having knowledge of one or two deep learning courses in university
  • having experience with PyTorch and/or Tensorflow to build and train models
  • not having sufficient experience with on-device inference
  • may not know, e.g., what quantization is
  • may have little experience with tflite, ONNX
  • not having sufficient experience with ONE

what/how/why/difficulty

• what: S/he installs ONE-vscode
  • how: on single Windows machine
  • how: or on single Linux machine
  • how: or on his/her machine to run vscode and remotely connect Ubunto ssh server.
  • difficulty: Installation is not seamless. ONE-vscode and onecc should be installed separately.
• what: S/he updates ONE-vscode as well as onecc
  • how: by selecting official version or daily version (automatic update)
  • how: or by selecting a specific version of onecc
• what: S/he compiles a model file into target code, run value test, and run performance/memory check
  • how: with one-step UI(e.g., shortcut key) right after modifying model source code
  • how: or by running Python code, e.g., target_code = one.compile(model); target_code.value_test(); target_code.measure()
  • why: to easily make sure that his/her model code can be compiled into target code without error
  • difficulty: Currently, such mechanism with shortcut key is not provided
  • note: Even ourselves cannot guarantee if compiler produces correct target code. End-user also feels that. Therefore value test is a reasonable activity.
• what: S/he runs value test against target code
  • how: by comparing outputs visually (e.g., expected image vs actual image)
  • how: or by showing some metric that checks difference of actual and expected outputs
  • difficulty: visual comparison is not enough
  • difficulty: or metric should be explained in easy way, e.g, diff ratio >= 3% can be acceptable
• what: S/he measures performance of target code
  • why: because if target code is slow, model source code should be modified
  • difficulty: information should be easy to understand. Compiler or backend-specific terms should be somehow easy to understand.
• what: S/he measures performance of target code running on simulator or real H/W
  • how: by selection simulator or H/W or both (multiple H/W devices can be used, too)
  • why: because running OK on simulator does not mean it's OK on real H/W
  • difficulty:
• what: S/he enhances model source code after measuring performance
  • how: by fixing source code pointed by performance result
  • difficulty: Currently no such feature is provided
• what: S/he measures DRAM consumption
  • how: by checking static DRAM consumption (static = calculated by compiler)
  • why: becauae low consumption of DRAM is necessary for embedded device
  • note: (dynamic DRAM consumption is beyond compiler)
• what: S/he compiles model file with default option (no manual change on compilation option)
  • how: just like $ gcc test.cc works OK
  • why: because this is normally what developer does for the first compilation
• what: S/he enters more compilation options
  • how: by entering O1, O2, ... just like what they do with gcc
  • why: because such options are naturally expected
• what: S/he enters quantization options
  • how: by selecting Q8, Q16, or something else (e.g., mixed)
  • difficulty: End-user may not know about quantization. Should we ask them to study quantization? Isn't it too much?
• what: S/he understands the reason of compilation error
  • how: by reading error message
  • difficulty: Currently, error messages are too complicated to end-users
• what: S/he exports pth or onnx to compile PyTorch model
  • how: by writing source code to export model
  • why: because pth and onnx are acceptable format for onecc
  • difficulty: S/he should be able to know they have to export their model into pth oro onnx format
• what: S/he manages many versions of model files, target codes, data file, compilation options, etc
  • how: by their own way
  • why: to compile model file later
  • difficulty: there is no revision tool for model
• what: S/he runs ONE-vscode in the vscode workspace for model source code
  • note: However, what can ONE-vscode provides with model source??
• what: S/he contacts to us when they cannot solve error
  • how: by posting an issue
  • how: or by sending an email
  • difficulty: Posting an open issue in Github might be psychologically difficult to some developers

[hyunsik-yoon]

2. Define

• after offline discussion with @jyoungyun and @YongseopKim)
• In this process, problem statements are defined.
• For end-user mentioned at https://github.com/Samsung/ONE-vscode/issues/282#issuecomment-1028849954, we found the following problems when end-user uses onecc and ONE-vscode:

Category	Problem	Note
Installation	ONE-vscode and onecc are installed only in Ubuntu device	end-user may use different OS
Installation	ONE-vscode and onecc are installed separately
Update	Upgrading ONE-vscode and onecc is difficult	- Upgrade is not automatic and it may cause version mismatch. - It is questionable if installed ONE-vscode can be used with the latest version of onecc.
target H/W support	ONE-vscode cannot be used with real target H/W	Installing and configuring required S/W on target device is difficult.
Input model	Generation of tflite and ONNX, which are input format of onecc is difficult	- end-user may not know what tflite or ONNX is. - Exporting tflite or ONNX may cause error.
Compile	There is no default option for compilation (like $ gcc test.cc)	Too many parameters are frustrating at the first trial of compilation.
Compile	It is hard to select optimization parameters.	optimization parameters are too complicated. Learning all options seems frustrating.
Compile	Not sure what quantization is	end-users are familiar with f32 models but may not with quantization.
Verify	It is hard to check if compiled target code runs really correctly.	- There is no tool for value test. - It is hard to make input data. - Actual result from target H/W should be value-checked too.
Verify	It is hard to check if compiled target code shows good performance.	- Not sure how to do performance check. - Even though performance test was successful, the result is hard to understand. - With the result, it is hard to pinpoint which line in model source code should be enhanced. - Performance from target H/W should also be presented.
Verify	It is hard to check how much Dram is consumed with the compiled target code.	There is no such tool.
Compilation Step	Steps in compilation are complex and some steps need to be ran separately in CLI.	- There is no way to run whole steps (from compilation to verification) with, e.g., one short-cut key. - All commands related to circle are not familiar and difficult to understand.
file management	It is difficult to manage revision and history of model files, input data, and target code.	it is difficult to reproduce a specific target code.
error	It is difficult to understand error message.	Not sure the meaning of error message related to circle or target code means.
error	it is hard to report unsolvable error.	- Not sure whom to contact to. - Posting an open issue in Github might be psychologically difficult to some developers. - Writing an error report that can be easily reproduced takes long time.

[lemmaa]

https://github.com/Samsung/ONE-vscode/issues/282#issuecomment-1028849954

• note: Even ourselves cannot guarantee if compiler produces correct target code. End-user also feels that. Therefore value test is a reasonable activity.

To be precise, the compiler will generate (and should generate) correct code. But we cannot guarantee that it will produce correct values. Much of it will be due to quantization error.

[hyunsik-yoon]

3. Ideation

These are the list of items after 3. Ideation.

Note:

1. This is a tentative list.
2. Detailed implementation can be made for ONE-vscode, onecc toolchain, or backend compiler. However, here, let's just use the term, ONE-vscode, for simplicity.

• Installation
  • Developers can run ONE-vscode on Windows or Ubuntu. Remote workspace should also be supported.
  • Developers can consult installation guideline.
  • Installation should be tested in CI.
  • During installation, let's check if other prerequisite S/W (e.g., docker?) is installed, if any.
  • let's check if the version of each file in toolchain are all correct.
• Update
  • Do not update toolchain in automatic way.
  • Developer can install multiple versions of toolchains (e.g., some daily versions and official versions)
  • Developer can select version of daily or official version of toolchain.
• Uninstall
  • Developer can uninstall some specific version of toolchains.
• Runtime environment
  • Although Ubuntu 16.04 is old, people are still using it. Let's support 16.04.
  • ONE-vscode should work with SSH extension.
• HW support
  • For value-test, developers can select HW or simulator supported by each backend.
• Input model
  • Input model format should be guided to developers.
  • Developers do not need to enter input names or shapes if these could be found automatically.
  • Any error during conversion of tflite or onnx should be reported to developers with messages they can understand.
• Input and output format (for value-test or quantization)
  • Developers can use various format (image, *.npy, f32 or quantized data, etc)
  • Developers can use random data with range that use can specify.
• Value test (golden test)
  • Developers want to check if compiled model works correctly by running value test.
  • ONE-vscode supports value test to see diff between expected and actual result.
  • Developers can see some metrics for diff and they should be presented with description.
  • In case of image data, expected and actual result in image format need to be shown.
  • Temp files generated during value test should be removed later.
• Latency test
  • Developers want to make sure if compiled model fits for real product by checking latency.
  • Developers can use visualized info of 'latency per op'.
  • Too much info can make developers confused. Let's provide exactly as much as they need.
  • Use terms from the view point of developers.
  • Format of result from H/W or simulator of backend should be similar.
• Memory test
  • Developers want to make sure if compiled model fits for real product by memory test.
  • Developer can see statically measured DRAM memory usage.
• Compilation step
  • Steps of compilation is automatically decided (e..g, if input model is already quantized, we don't need to perform quantization)
  • Developers can check option to generate Intermediate files (e.g., circle).
• File Management
  • Developers may need easy way to manage input model, input data, output model, etc.
• CLI
  • Developesr can see CLI commands and their output in a terminal screen.
  • Assume that a new CLI param in toolchain was newly introduced in a daily release. If the param is not yet supported by ONE-vscode, developers should know that some daily release of ONE-vscode does not support daily release of toolchain.
• Error handling
  • Message should be developer friendly.
  • Developers should know what s/he should do after reading the error message.

[hyunsik-yoon]

Let's prioritize the above items, considering 2022 goal and preference of involved people. Let me close this issue.