Requirements for the config interface

zachkinstner commented 9 years ago

The "config" interface is described in this document. If I'm understanding correctly, we can send events (to the app) and receive setup/configuration requests (from the app). The document is currently in a draft state.

At this point, the most important event seems to be the Leap::Controller "is connected" on/off event. However, as the discussion in the document mentions, we could also provide this information via the "analog" interface (numerically as 0 or 1). We could provide a variety of information this way, such as: left hand available, right hand available, left/right hand confidence, etc.

The document describes services like allowing apps to cycle/reset the device, configure it, calibrate it, etc. It's not clear, however, whether device-specific things like this are encouraged, vs. having a device-agnostic API for performing common services.

One of the open questions is:

How do we handle device-specific configuration?

A note about sending device-specific data (by @russell-taylor):

One job of the OSVR device layer is to hide the need to know this from the client code by factoring and then mapping whatever decisions need to be made based on this into the API. I view having the need to send this information to the client directly as a failure to appropriately factor and map the devices.

zachkinstner commented 9 years ago

@jdonald, @rpavlik, I think this is probably the next priority for the plugin. Let's discuss how best to communicate this event/service data.

Technically speaking, it's not clear to me right now how to send this information via OSVR. There isn't a specific ConfigInterface available in the PluginKit. Is it meant to be sent as messages via the device interface?

rpavlik commented 9 years ago

That's correct, there is not a specified config interface yet. Considering JSON-RPC as the technique, and/or a few stock messages. We'd like to have shared interfaces, and failing that, self-describing interfaces.

Addressing the comment by @russell-taylor - I don't expect that clients will typically interact with the config interface. Some clients might show things like low battery alerts for devices in use, but I imagine it will mostly be for centralized config by some enhanced server process/gui. We have considered potentially two interfaces: one for "status" type data that client apps might want to convey, and one for the more config-related items. While I don't think it's true for the Leap Motion device, some device classes effectively require some configuration (eye tracker, for instance, typically requires calibration), while others might have some configuration as optional, which is one of the reasons why we factored configuration out into its own interface.

zachkinstner commented 9 years ago

That's correct, there is not a specified config interface yet. [...] one for "status" type data that client apps might want to convey

Would this interface be similar to the analog interface? That way, the plugin continually sends its status values (like "is connected" or "battery percentage"), and client app can listen for these values on demand.

one for the more config-related items. While I don't think it's true for the Leap Motion device, some device classes effectively require some configuration

I assume that these values would be more "static" -- they would start out with their default value, and then only change if the client app requests it. Sound correct?

Leap Motion has a calibration process that exists within the "control panel" app, but I don't think it's accessible/adjustable via API. There are several configuration values related to gestures, set via Leap::Config, but I'm not sure how important it is to expose these via OSVR.

rpavlik commented 9 years ago

I think the status stuff would be as you mention, but that's still under discussion. In any case, there is no harm in exposing analog status values as analogs, and binary status values as buttons, with appropriate semantic mapping in the device descriptor.

The config for the Leap Motion itself would probably be relatively static, yes.

zachkinstner commented 9 years ago

The commit referenced above adds several values to the analog interface. These particular values are less about "configuration" and more about "status". New alias paths:

/connection/device
/connection/service

/arms/left/available
/arms/left/hand/confidence
/arms/left/hand/grabStrength
/arms/left/hand/pinchStrength
/arms/left/hand/palmWidth

/arms/right/available
/arms/right/hand/confidence
/arms/right/hand/grabStrength
/arms/right/hand/pinchStrength
/arms/right/hand/palmWidth

zachkinstner commented 9 years ago

@jdonald, which (if any) configuration values should we expose via OSVR?

We could get/set the values supported by Leap::Config. For now, these are all gesture-related parameters.
The Leap::Config::save() event could occur whenever a config value changes, rather than requiring the client app to do this.
The Leap::Controller get/set PolicyFlag config could likely be ignored. The plugin sets these flags internally -- they all need to be enabled.
The Leap::Controller get/enable/disable Gesture::Type config. Could we simply enable them all? Perhaps there would be performance concerns for doing that.

jdonald commented 9 years ago

Cool that we've resolved the 'status' bits. I think things like isConnected were most important.

Config: You can set a whole bunch of values via Leap::Config. Running LeapSvc.exe --help reveals some of these. Examples include disabling robust mode or turning on tool tracking.

I don't think we should set all PolicyFlags by default. Background app mode should be off by default. For the images policy, it's typically enabled if someone's asset uses image hands, but when an app doesn't need images this policy flag should be disabled.

It's true that gestures don't use up a lot of CPU and you could just enable them all. However, developers don't know this and are often aggressively looking to turn things off. If you enable the other config mechanisms, I don't think this would be a lot of work to support too.

zachkinstner commented 9 years ago

For reference, here's the output of LeapSvc.exe --help:

Service options:
  -h [ --help ]         Prints this message
  -i [ --ignore ]       Do not attempt to load the default config.json file
  --install             Installs the service at its current path
  --remove              Uninstalls the service, if installed
  --test                Runs a simple internal service test suite
  --run                 Run the service in interactive mode
  --usermode            Run the usermode process
  --config arg          The path to a configuration file to be used

Public Configurations:
  --camera_type arg (=UVCI)
  --camera_framerate_limit arg (=1000)
  --camera_requires_calib [=arg(=1)]
  --camera_y_sample_rate arg (=0)
  --camera_x_sample_rate arg (=0)
  --camera_mode arg (=HVGA)
  --image_width arg (=640)
  --image_height arg (=480)
  --image_calib_width arg (=640)
  --image_calib_height arg (=480)
  --tracking_mode arg (=balanced)
  --tracking_tool_enabled [=arg(=1)]
  --tracking_hand_enabled [=arg(=1)]
  --tracking_version arg (=v2)
  --sampling_sparse_gap arg (=0.100000001)
  --sampling_max_gap arg (=1)
  --sampling_min_gap arg (=1)
  --robust_mode_enabled [=arg(=1)]
  --force_robust_mode [=arg(=1)]        0 = standard, 1 = mandatory robust mode
  --low_resource_mode_enabled [=arg(=1)]
  --avoid_poor_performance [=arg(=1)]
  --background_app_mode arg (=1)        0 = disallow, 1 = not set, 2 = always
  --images_mode arg (=1)                0 = disallow, 1 = not set, 2 = always
  --parallelizing_filter_graph [=arg(=1)]
  --support_xs_transport [=arg(=1)]
  --websockets_enabled [=arg(=1)]
  --websockets_allow_remote [=arg(=1)]
  --ws_port arg (=6437)
  --wss_port arg (=6436)
  --use_autonet [=arg(=1)]              Enable Autonet monitoring server
  --server_connect_ip arg (=127.0.0.1)
  --print_timestamps_in_log [=arg(=1)]
  --interval_record_path arg            Write timestamps to the specified file,
                                        empty disables this behavior
  --training_image_width arg (=64)
  --training_image_height arg (=48)
  --training_image_hand_size arg (=16)
  --training_image_fix_scale [=arg(=1)]
  --training_image_interval arg (=0)
  --skeleton_initialization_error arg (=6)
  --skeleton_palm_filtering arg (=0.5)
  --skeleton_finger_filtering arg (=0.59999999999999998)
  --skeleton_velocity_filtering arg (=0.69999999999999996)
  --skeleton_half_filter_vel arg (=900)
  --skeleton_half_tip_filter_vel arg (=1500)
  --skeleton_half_palm_filter_vel arg (=600)
  --skeleton_iterations arg (=10)
  --skeleton_error_bound arg (=0.01)
  --skeleton_finger_samples arg (=85)
  --skeleton_separation arg (=10)
  --skeleton_use_sampling [=arg(=1)]
  --skeleton_extreme [=arg(=1)]
  --skeleton_finger_recovery [=arg(=1)]
  --skeleton_try_ik [=arg(=1)]
  --skeleton_reinitialize [=arg(=1)]
  --skeleton_v [=arg(=1)]
  --skeleton_use_swarm [=arg(=1)]
  --skeleton_swarm_iterations arg (=15)
  --skeleton_swarm_particles arg (=15)
  --skeleton_recovery [=arg(=1)]
  --skeleton_match_conf [=arg(=1)]
  --skeleton_sampling_interlace [=arg(=1)]
  --skeleton_finger_pivot [=arg(=1)]
  --skeleton_inertia [=arg(=1)]
  --skeleton_use_palm_weight [=arg(=1)]
  --skeleton_update_stubbiness [=arg(=1)]
  --skeleton_use_loop_init [=arg(=1)]
  --skeleton_block_flow [=arg(=1)]
  --skeleton_default_scale arg (=1)
  --skeleton_use_origin_basis [=arg(=1)]
  --skeleton_max_scale arg (=1.0800000000000001)
  --skeleton_min_scale arg (=0.71999999999999997)
  --use_new_matching [=arg(=1)]
  --use_new_matching_hmd [=arg(=1)]
  --use_new_matching_hp [=arg(=1)]
  --use_contour_tree [=arg(=1)]
  --use_optical_flow [=arg(=1)]
  --use_hardware_crop [=arg(=1)]
  --key_tap_max_comp_down_ratio arg (=0.5)
  --no_cp_startup [=arg(=1)]
  --tool_refinement_use_Newton [=arg(=1)]
  --tool_max_init_RMS_error arg (=2)
  --tool_max_init_error arg (=0.001)
  --tool_max_growth_error arg (=0.00050000000000000001)
  --tool_refinement_iterations arg (=2)
  --tool_refinement_step_size arg (=1)
  --tool_refinement_update_reg arg (=0.01)
  --tool_removal_max_error arg (=2.2999999999999998)
  --tool_removal_straightness arg (=0.002)
  --tool_axial_adjustment [=arg(=1)]
  --tool_course_adjustment [=arg(=1)]
  --tool_growth_dist arg (=10)
  --tool_filtering arg (=0.5)
  --tool_axial_filtering arg (=0.90000000000000002)
  --tool_radius_filtering arg (=0.999)
  --hand_to_tool_reinitialization [=arg(=1)]
  --use_halide [=arg(=1)]
  --run_candidate_detection [=arg(=1)]
  --image_processing_auto_flip [=arg(=1)]
  --image_processing_standby [=arg(=1)]
  --image_processing_r_weight arg (=0.25)
  --image_processing_max_dist arg (=700)
  --image_processing_min_dist arg (=45)
  --image_processing_sample_step arg (=4)
  --image_processing_use_peaks [=arg(=1)]
  --image_processing_use_2d_peaks [=arg(=1)]
  --image_processing_region_ids [=arg(=1)]
  --image_processing_dynamic_thresh [=arg(=1)]
  --image_processing_hessian_edge [=arg(=1)]
  --image_processing_adaptive_edge_density [=arg(=1)]
  --use_noise_correction [=arg(=1)]
  --head_mounted_display_mode [=arg(=1)]
  --power_saving_adapter [=arg(=1)]
  --power_saving_battery [=arg(=1)]
  --os_interaction_multi_monitor [=arg(=1)]
  --interaction_box_auto [=arg(=1)]
  --interaction_box_height arg (=200)
  --interaction_box_scale arg (=0.80000000000000004)
  --interaction_center_x arg (=0)
  --interaction_center_y arg (=200)
  --interaction_center_z arg (=0)
  --autoplane_closeness_epsilon arg (=4)
  --autoplane_covariance_scale_factor arg (=0.5)
  --autoplane_time_window arg (=0.40000000000000002)
  --autoplane_warmup_time arg (=0.33300000000000002)
  --autoplane_hover_zone_cutoff arg (=70)
  --autoplane_dead_zone_cutoff arg (=220)
  --autoplane_function_steepness arg (=1)
  --autoplane_function_width arg (=4)
  --autoplane_function_speed arg (=0.5)
  --adjust_radiance [=arg(=1)]
  --radiance_use_bg [=arg(=1)]
  --screen_detected [=arg(=1)]
  --screen_point0_x arg (=0)
  --screen_point0_y arg (=0)
  --screen_point0_z arg (=0)
  --screen_point1_x arg (=0)
  --screen_point1_y arg (=0)
  --screen_point1_z arg (=0)
  --screen_point2_x arg (=0)
  --screen_point2_y arg (=0)
  --screen_point2_z arg (=0)
  --screen_point3_x arg (=0)
  --screen_point3_y arg (=0)
  --screen_point3_z arg (=0)
  --screen_point4_x arg (=0)
  --screen_point4_y arg (=0)
  --screen_point4_z arg (=0)
  --screen_point5_x arg (=0)
  --screen_point5_y arg (=0)
  --screen_point5_z arg (=0)
  --screen_point6_x arg (=0)
  --screen_point6_y arg (=0)
  --screen_point6_z arg (=0)
  --screen_point7_x arg (=0)
  --screen_point7_y arg (=0)
  --screen_point7_z arg (=0)
  --reflection_removal_enabled [=arg(=1)]
  --tracking_quad_enabled [=arg(=1)]
  --quad_color arg (=BluePink)
  --quad_num_cols arg (=4)
  --quad_num_rows arg (=7)
  --quad_step_x arg (=16.5)
  --quad_step_y arg (=16.5)
  --quad_res_x arg (=1080)
  --quad_res_y arg (=1920)
  --testing_output_json arg
  --testing_interface_type arg
  --testing_verbose [=arg(=1)]
  --testing_indented_output [=arg(=1)]
  --auto_check_updates [=arg(=1)]
  --auto_install_updates [=arg(=1)]
  --force_accept_updates [=arg(=1)]
  --update_channel arg (=stable)
  --allow_firmware_write [=arg(=1)]
  --metrics_enabled [=arg(=1)]
  --v4l2_pid arg (=19)
  --ref_impl arg

zachkinstner commented 9 years ago

@jdonald, some questions:

Are all these config values able to be changed at runtime?
The config keys in the Leap::Config have a format like Gesture.Circle.MinRadius. Would I still pass in the config keys exactly how they are shown above, like low_resource_mode_enabled?
Several of these config values seem like things the user would set based on their own preferences (i.e. auto-install updates) or needs (i.e. they are in a bright environment). Does it make sense for an app to control these vs. the user choosing their own settings?

zachkinstner commented 9 years ago

Background app mode should be off by default.

The Controller::POLICY_BACKGROUND_FRAMES policy needs to be enabled, otherwise the controller only sends frames when the OSVR Server window has focus.

For the images policy, it's typically enabled if someone's asset uses image hands, but when an app doesn't need images this policy flag should be disabled.

Sounds good. Once the config interface is available, we can allow the Controller::POLICY_IMAGES policy to be toggled. If it is disabled, the plugin won't send anything through the imaging interface.

zachkinstner commented 9 years ago

The commit referenced above includes a new Configure class. We're not sure exactly how the OSVR config interface will look, so as a starting point, I just used string keys to identify each configuration value.

Currently, the class supports toggling the policy and gesture types, and changing the gesture-related Leap::Config values. We can expand the class to support more config keys as necessary. Once the OSVR config interface is ready, this class will connect to it, which will allow apps to get/set these values at runtime.

zachkinstner commented 9 years ago

Via email:

Looking at the correspondence, we don't think a new type of interface is required. The Leap Motion plugin could just implement an interface that has bottons (e.g. booleans) and analogs, and this should be enough to convey any real-time status information that you wish.

The plugin currently sends status values via the analog interface (see this comment). I could update the plugin to also send the config values. However, is there currently a way for apps to set the plugin's config values? As I understand it, the Leap plugin is supposed to allow apps to set/toggle certain features, like device policies, gesture support, and gesture parameters.

russell-taylor commented 9 years ago

I would imagine that the configuration would be done in the config file, not live by the app.

zachkinstner commented 9 years ago

@jdonald, perhaps you could review this latest discussion?

With the Leap Motion device, there are several configuration scenarios which are app-specific. If the app used a config file (rather than sending live changes), the app would need to enable everything that it might need for execution. For example, an app may only need image pass-through in some specific scenarios, but the image support would be enabled the entire time.

In the case of app-specific config files, is the app able to automatically provide/install this file to OSVR?

Please let me know if there are any config file examples that I can review.

russell-taylor commented 9 years ago

The config files will specify which devices are installed, what rendering modes you want to use, and so forth. They include things like the distance between the user's eyes. My understanding of the current plan is to have a separate configuration app that generates these files, auto-detecting where possible, to keep the user from having to much about with editing them.

If the behavior needs to change during the execution of the application (not for configuration, but for actual use), then I don't see a way around adjusting these parameters from the client side. This will require a new backwards-flowing API. If it is for configuration/calibration, then a separate calibration app could enable those interfaces.

The underlying VRPN protocol used for many devices has an AnalogOutput that could be used to send configuration values back to the app, but I don't think that has been exposed as an interface in OSVR yet because it has not yet been needed. And presumably because it will require a lot of careful thought to make it right the first time.

jdonald commented 9 years ago

I have not been following lately, and would like to defer to @GabrielHare

GabrielHare commented 9 years ago

@zachkinstner Most of the configurations pertain to specific modes of operation, and many others handled by the control panel. So, the main issue, as pointed out by James is to ensure that the configuration for use with a VR headset is correct.

The following are the only options that pertain, all of which can be found in the control panel... and my preference for the default values:

HMD Mode (head_mounted_display_mode) = true (1)

This modifies the tracking algorithms based on the typical view of the hands. Must be enabled.

Hand Tracking (tracking_tool_enabled) = true (1)

Could be turned off, if only images are desired.

Tool Tracking (tracking_tool_enabled) = true (1)

Could be turned off, but consumes very little CPU time.

Allow Images (images_mode) = true (2)

Could be turned off, but actually consumes very little CPU time or memory if not used by an application. Images are required for the canonical "ImageHands" representation of the user's hands.

Robust Mode (robust_mode_enabled) = false (0)

If robust mode is activated while viewing external images the image quality and rate is degraded, which is a bad experience for users. Should be disabled when using images in HMD.

Auto-Orient Tracking (image_processing_auto_flip) = true (1)

This allows the device to adapt to its orientation on the HMD. Essential for peripherals, should be disabled for built-in devices.

Low Resource Mode (low_resource_mode_enabled) = false (0)

Reduces the frame rate, in the case that images are not used. In head mounted mode this degrades the experience of hand tracking (albeit not the instantaneous accuracy).

Avoid Poor Performance (???) = false

If this mode is activated while viewing external images the images will freeze, usually inducing vertigo and nausea in the user. Must be disabled in HMD mode.

Cheat Death (klaatu_barada_nikto) = true (1)

I don't always die... but when I do, I prefer resurrection.

I'm not sure that it makes sense to enable users to toggle any of these. The only ones that might actually free CPU time are Tool and Hand tracking... but those are the point of have a Leap.

zachkinstner commented 9 years ago

Avoid Poor Performance (???)

Found the new avoid_poor_performance setting. Updated the LeapSvc output above.

zachkinstner commented 9 years ago

I have added the default config values that @GabrielHare provided. My understanding is that these default values will be sufficient for now; we do not need to allow apps to get/set these values via OSVR.

The existing Configure class was built in preparation for getting/setting values, so it is currently more complex than it needs to be. This seems fine for now, but I wanted to mention it here.

One potential (minor) downside here might be that apps cannot adjust the gesture-related config settings. For example, the app may need to set the minimum radius of a "circle" gesture, or the minimum speed of a "swipe" gesture.

I'm going to close this issue for now, and we can re-open if necessary.

OSVR / OSVR-Leap-Motion

Requirements for the config interface #4