dboltovskyi
commented
4 years ago -
We're Ok with this approach if a recursion wouldn't lead to an issues. One small update we would like to have
maxvalue="100"
Closed theresalech closed 4 years ago
dboltovskyi
commented
4 years ago We're Ok with this approach if a recursion wouldn't lead to an issues.
One small update we would like to have maxvalue="100"
dboltovskyi
commented
4 years ago 6. Taking item 8. into account please suggest our understanding is correct:
<function name="OnAppCapabilityUpdated" functionID="OnAppCapabilityUpdatedID" messagetype="notification" since="x.x">
<param name="appCapability" type="AppCapability" mandatory="true" />
</function>
<struct name="AppCapability" since="x.x">
<param name="capabilityType" type="SystemCapabilityType" mandatory="true" />
<param name="videoStreamingCapability" type="VideoStreamingCapability" mandatory="false" />
</struct>
kshala-ford
commented
4 years ago Sorry for jumping into this discussion so late but as I support two of the major mobile navigation app partners I thought my feedback would be valuable.
First of all I highly recommend to design this feature with as little effort as possible especially for the app partner side. Generally I see some good use cases of informing Core about app capabilities. In this case I have some concerns. This issue has so many comments I couldn't figure out if the current suggestion is to have the app send an array of supported resolutions based on a list from the system or out of nowhere. In both cases I don't think the app developer can code some logic to properly answer to this list. Both of my app partners are not hesitant about different resolutions. Sure a high resolution is preferred but their major concern is the aspect ratio. One app partner has major issues with very wide screen resolutions because UI elements which are "docked" to the lower part of the app's view overlap with UI elements from the upper part.
Furthermore if the app developer is using the library to automate video streaming then we should do our best to also automate this feature. For iOS the CarWindow should straight change the view's bounds and continue capturing and encoding the window.
Instead of sending AppCapability my suggestion would be to provide the additional resolutions to the app in the VideoStreamingCapability. The new resolutions should be named so that it is known for what the resolution is used for. A good example of names was already mentioned here:
+<enum name="VideoStreamingResolutionTypes">
+<element name="DEFAULT"/>
+<element name="FULL_SCREEN"/>
+<element name="PICTURE_IN_PICTURE"/>
+<element name="PREVIEW"/>
+<element name="SPLIT_SCREEN"/>
+<element name="COLLAPSED_VIEW"/>
+</enum>
+ <struct name="AdditionalResolutions"> <!-- There's for sure a better name :) -->
+ <param name="type" type="VideoStreamingResolutionTypes" mandatory="true"/>
+ <param name="resolution" type="ImageResolution" mandatory="true"/>
+</struct>
<struct name="VideoStreamingCapability" since="4.5">
<description>Contains information about this system's video streaming capabilities.</description>
<param name="preferredResolution" type="ImageResolution" mandatory="false">
<description>The preferred resolution of a video stream for decoding and rendering on HMI.</description>
</param>
+ <param name="additionalResolutions" type="AdditionalResolutions" mandatory="false" array="true" />This way the system can provide information what other types it supports. Each type has aspects like picture in picture should basically show the app next to another UI view but everything is shown very small as the content is scaled. Preview however is similar with regards to size on the GUI but the scaling should allow interaction with view controls. Splitscreen may cause a ridiculous aspect ratio etc.
Apps can choose to support different types and exclude some they don't support. They do this by configuring the SDL manager. The manager however doesn't send an RPC like AppCapability. Instead my suggestion is to use the protocol spec.
According to Start Video Service there should be an additional tag in the payload called supportedResolutionTypes which populates with the developer's configuration.
| Tag Name | Type | Introduced | Description |
|---|---|---|---|
| supportedResolutionTypes | String Array | X.Y | Indicates which resolution types the app supports allowing the HMI whether or not it should provide the possibility to switch the app UI to fullscreen, splitscreen etc. |
Based on the supported types the HMI can offer user control to move the app to full screen, splitscreen etc. If the app doesn't want to be shown in small view ("picture in picture or "preview") the HMI shouldn't show the app this way.
If the HMI wants to change the type to e.g. Full screen upon user control, the system should send a control frame of the video service to the app The control frame payload includes one tag called resolutionType and also two tags for width and height.
With this control frame the SDL library can suspend the remote display/car window, stop the video encoder, resize views and restart the video encoder again. Once everything is complete the SDL library should also send the same control frame to the system before it sends the first video data. This is very very important for the UI/UX so that the HMI knows exactly which byte from the protocol session are related to the new resolution. Otherwise the HMI and the video stream are not aligned and present the app in a weird aspect/scale. At all time the video service session must remain open. it must not be stopped and restarted otherwise a huge delay is added for TLS handshake and encryption init.
I know this suggestion is vastly different from what was suggested before and my suggestion is not fleshed out but I am worried that the conversation is not including the protocol level. I don't think we can resolve everything in the RPC level.
okandul
commented
4 years ago 9.
I just want to clarify two topics: 9a. Is it correct the HMI guidelines is the recommendation and describes the reference implementation presented in the Open Source version of SDL? If no, who and how work with the guidelines?
9b. Regarding the touch events for the scaled controls. As I understood, the idea of the proposal is to have similar behavior as described here https://developer.android.com/guide/topics/ui/picture-in-picture. If so, the HMI should prepare the "infrastructure" for the app. Everything else needs to be managed by the mobile app. The "infrastructure" (minimum window size, etc.) might be described in the guidelines but shouldn't limit the app itself.
theresalech
commented
4 years ago The Steering Committee voted to keep this proposal in review until our next meeting, to allow the author time to respond to the latest comments on the review issue, and to allow the PM and other SDLC members to review and respond as well.
theresalech
commented
4 years ago Please find collective feedback from the PM below.
3.
The proposed change does not affect the public API though it adds a few new methods to the API.
We're struggling to comprehend what is being said here. How does something add a few methods but not change the public API? How can we ask devs to declare support / lack of support for various resolutions without any public API changes?
The majority of what has been posted here is unnecessary and not especially helpful for the purposes of the proposal review. We understand that this can be implemented into the library. The discussion is over the public APIs and how resolution switching will occur. The rest, in my mind, are implementation details that can be sorted out during implementation.
6. We do not want to create a subset of SystemCapabilitiy enum values that only apply to app capabilities or a subset of system capabilities. Otherwise we would simply continue to use all of SystemCapability RPCs and structs. We'd recommend:
<function name="OnAppCapabilityUpdated" functionID="OnAppCapabilityUpdatedID" messagetype="notification" since="x.x">
<param name="appCapability" type="AppCapability" mandatory="true" />
</function>
<struct name="AppCapability" since="x.x">
<param name="appCapabilityType" type="AppCapabilityType" mandatory="true" />
<param name="videoStreamingCapability" type="VideoStreamingCapability" mandatory="false" />
</struct>
<enum name="AppCapabilityType" since="x.x">
<description>Enumerations of all available app capability types</description>
<element name="VIDEO_STREAMING"/>
</enum>8. 👍
9a. Yes, there are recommendations present, but there are also requirements for HMI integrators.
9b. Every platform provides some restrictions on applications, whether Android, iOS, CarPlay, Android Auto., including SDL. We provide a number of restrictions on OEM UX with SDL (see templates, etc.). If we provide no boundaries for apps, the appeal to integrate SDL is lessened because the amount of work the app developer needs to do grows exponentially. Therefore, stating that everything is up to the app developer just isn’t true in an SDL context, nor in any other context. If Livio’s proposed restrictions are not followed, we enter a place where app developers don’t know what to expect. Some head units may implement a 1"x1" PIP and pass through touches, which requires the app dev to show buttons, which is unusable to the user. Another head unit might absorb those touches. Then the dev displays buttons even though touches can’t occur. We believe that our proposed restrictions are reasonable to provide consistent app developer expectations and a generally good HMI user experience.
@kshala-ford
This issue has so many comments I couldn't figure out if the current suggestion is to have the app send an array of supported resolutions based on a list from the system or out of nowhere.
It was the former.
Sure a high resolution is preferred but their major concern is the aspect ratio.
This has been Livio's concern as well.
Instead of sending AppCapability my suggestion would be to provide the additional resolutions to the app in the VideoStreamingCapability.
This is the current idea, but the app needs a way to declare whether or not they support those different resolutions, so that the head unit can disable things like the PIP button, etc. Otherwise, the HU will display UI that the app does not support. This was deemed to be poor UX.
VideoStreamingResolutionTypes
I don't think that this enum is helpful. What's important is the resolution / scale, etc. A split screen display could be top/bottom or left/right, for example. A dev declaring that they support certain types like you've defined could have wildly different sizes and aspect ratios.
Your API here is close to the current idea, but the current idea is to embed additional VideoStreamingCapability in the top-level VideoStreamingCapability so that information like scale and actual size (in cm / in) can be sent to the app developer.
However, if these types were standardized (aspect ratios, sizes within ranges, etc.), then this would be more helpful.
If the HMI wants to change the type to e.g. Full screen upon user control, the system should send a control frame of the video service to the app The control frame payload includes one tag called resolutionType and also two tags for width and height.
Is this an ask to add a new protocol control service packet or to modify StartService? The current idea is to use the StartService packet to start the new resolution sizes, etc. The reasoning behind this is that there is no current way to update the resolution without ending the service and starting the service again. If we want to add that capability, then that would be an improvement.
I know this suggestion is vastly different from what was suggested before and my suggestion is not fleshed out but I am worried that the conversation is not including the protocol level.
The protocol spec has been discussed at some point before, and your ideas aren't totally different from the current agreement.
kboskin
commented
4 years ago Hi @dboltovskyi, @kostyaBoss, @yoooriii please see our responses to 3, 6, 8, and 9.
- Please note that this is not a code review. This must be considered example code and will be required to go through a full code review when the PR is put to Github.
Android: We can't change public methods without a breaking change (
stopVideoService). It is also unnecessary. These changes could occur in the VideoStreamingManager itself. Simply using the service listener once the service is stopped, if the flag is set in the manager to restart the service it does so then.* I think adding a new state to the `StreamingStateMachine` PAUSED makes sense * In the `SdlRemoteDisplay` adding a new method as a callback to developers like `on�ResizeView` that the developers can override would be helpful. I would imagine it returned a boolean to state if they wish for the resizing to continue or not.
@joeygrover POC for Android updated with mentioned items:
Stopped statestopVideoService method and encapsulated most of logic in VideoStreamManager using ServiceListener. I didn't change branch of the PoC, all changes can be checked using this link
kshala-ford
commented
4 years ago @ PM thanks for the response. I went through the proposal again and talked to the author and I agree my suggestions are close to the proposed solution. Therefore please disregard my comment. Instead I want to suggest something with regards to issue 3 of how to add new public APIs for the app developers.
3.
We're struggling to comprehend what is being said here. How does something add a few methods but not change the public API? How can we ask devs to declare support / lack of support for various resolutions without any public API changes?
I don't think an app developer will be able to manually distinguish supported resolutions on their own. Also the SDL managers meant to take over as much responsibility as possible to simplify SDL integration. Instead of having the app developer to read the available resolutions, magically calculate which ones it supports and reply with supported resolutions, the SDL libraries should be responsible for this. The libs will require to get additional params from the app dev through the streaming configuration or the builder. I must admit I am not very familiar with Android video streaming but the point is that identifying supported resolutions can be done if the app provides the following information:
Following iOS example should present how the app could configure the SDL manager with the supported ranges.
@interface SDLSupportedStreamingRange
// The minimum supported aspect ratio, Min value is 1
@property (nonatomic) NSUInteger minimumAspectRatio;
// The maximum supported aspect ratio.
@property (nonatomic) NSUInteger maximumAspectRatio;
// The minimum resolution to support
@property (nonatomic, nullable) SDLImageResolution *minimumResolution;
// The maximum resolution to support
@property (nonatomic, nullable) SDLImageResolution *maximumResolution,
@end
@interface SDLStreamingMediaConfiguration
@property (nonatomic, nullable) SDLSupportedStreamingRange supportedLandscapeStreamingRange;
@property (nonatomic, nullable) SDLSupportedStreamingRange supportedPortraitStreamingRange;
+ (instancetype)autostreamingSecureConfigurationWithInitialViewController:(UIViewController *)initialViewController
supportedStreamingRange:(SDLSupportedStreamingRange *)supportedStreamingRange;
+ (instancetype)autostreamingSecureConfigurationWithInitialViewController:(UIViewController *)initialViewController
supportedLandscapeStreamingRange:(SDLSupportedStreamingRange *)supportedLandscapeStreamingRange
supportedPortraitStreamingRange:(SDLSupportedStreamingRange *)supportedPortraitStreamingRange;
@endFrom the two big navigation app partners I support, one prefers portrait views while the other prefers landscape (ideally square views). Therefore this information should be provided two times for portrait and for landscape views. Sure when constructing the SDL manager a convenience function or constructor should allow specifying information once which the library then uses for landscape and portrait.
Once the library receives the list of available resolutions it can distinguish the supported resolutions quite easily.
After all this suggestion is only a small addition to the suggested negotiation flow where "App" should be separated into "App" and "Library". Also the information we request from the app developer should already exist or at least partly. The app's design teams should already know the aspect ratios where the app appears okay without UI elements to overlap. Same appears for minimum resolution.
dboltovskyi
commented
4 years ago Thank you for clarification. We agree with this PM's suggestion.
dboltovskyi
commented
4 years ago 9. Taking all the discussion into account we agree to add suggested by PM section to HMI Integration Guidelines .
yoooriii
commented
4 years ago I would like to clarify the following statement:
The proposed change does not affect the public API though it adds a few new methods to the API.
Here I mean backward compatibility, old apps can use API as they did, "no change in API". New apps can take advantage of "new added methods in API".
Thank you.
joeljfischer
commented
4 years ago 3.
Also the SDL managers meant to take over as much responsibility as possible to simplify SDL integration. Instead of having the app developer to read the available resolutions, magically calculate which ones it supports and reply with supported resolutions, the SDL libraries should be responsible for this.
They would do it in a similar way that we would, by calculating aspect ratios, but I do agree that we may be able to assist them.
I think that if we have to actual sizes that will be used (in inches / centimeters), that may be necessary information for the developer as well. For example, a developer may want to display a 200x200 point screen if the screen is 4" diagonal, but not if it's 1" diagonal. I think we could pass:
This would allow them to provide all the important information without confusion. They could say "we don't support anything at scale 1.0", "or anything less than 2 inches."
However, even this may not be enough. We still need to somehow tell the developer when the streaming size changes and not just change the bounds on them (though we can do that for them too). For example, a 1 inch diagonal screen should probably not have buttons on it because they'd be too small to touch, so the dev needs to re-format their interface for that size (or for portrait vs. landscape, etc.).
So I think this is a good start, but there still needs to be callbacks to the dev informing them of sizing changes. Alternatively, if we just change the bounds on them, are there iOS / Android system calls we can tell developers to use to update their interface? That should be documented in the proposal as well.
kshala-ford
commented
4 years ago @joeljfischer
3.1. What information do we need from app dev?
Another parameter for diagonal screen size could make sense. PointsPerInch could be an alternative allowing easier calculation but it's basically the same information.
We discussed if scale is needed and we believe the resolution in points with screen size is sufficient for the developer to know the size of the app being presented. The intention of scale is to make the app appear the same way regardless of the display's pixel density. It's an SDL internal information and abstracted from the developer.
From our point of view scale is not required and would rather cause confusion to the developer.
3.2. How do we notify the app dev on changes?
I know both my app partners have designed the view dynamically so that the UI elements like buttons rearrange or (dis)appear automatically when changing the view size. However, not every app developer does that. Also the screen size could potentially change but the resolution remains the same, though I couldn't know if there's a use case for this. Notifying the app developer through a callback makes sense and wouldn't hurt anyways.
joeljfischer
commented
4 years ago 3.1. I think that between the diagonal screen size and the resolution in points it should be okay.
3.2. 👍 I want to make sure also for the app partners who change the view dynamically that the view is resized in such a way that their dynamic re-sizing activates, if that makes sense. Or, if we can't trigger those system calls, they will need to hook it up to our own callbacks.
theresalech
commented
4 years ago It was noted that items 6 and 9 have been agreed upon, but details of item 3 are still in discussion. It was requested that instead of a POC for item 3, code samples and explanations for the Public APIs (3.1) and a way to describe how the resolution switching will work (3.2) are all that’s necessary. The Steering Committee voted to keep this proposal in review to allow the author and Project Maintainer to work through the details for item 3.
joeljfischer
commented
4 years ago Hi, I'd like to try to document the agreed upon revisions up to this point. Please let me know if I missed anything or got something wrong:
APP_SERVICES, add DISPLAYS regarding HMI OnSystemCapabilityUpdated. Note that this may no longer be necessary given the updates to 2 and 6.StartService / Core send StartServiceACK while already connected in order to update?Author to Revise. Revise to use new flow to pass potential video capabilities.
<function name="OnAppCapabilityUpdated" functionID="OnAppCapabilityUpdatedID" messagetype="notification" since="x.x">
<description>A notification to inform SDL Core that a specific app capability has changed.</description>
<param name="appCapability" type="AppCapability" mandatory="true">
<description>The app capability that has been updated</description>
</param>
</function>
<struct name="AppCapability" since="x.x">
<param name="capabilityType" type="AppCapabilityType" mandatory="true">
<description>Used as a descriptor of what data to expect in this struct. The corresponding param to this enum should be included and the only other param included.</description>
</param>
<param name="videoStreamingCapability" type="VideoStreamingCapability" mandatory="false">
<description>Describes supported capabilities for video streaming </description>
</param>
</struct>
<struct name="AppVideoStreamingCapability" since="x.x">
<description>Contains information about this app's video streaming capabilities.</description>
<param name="supportedResolutions" type="ImageResolution" mandatory="false" array="true">
<description>Supported streaming resolutions</description>
</param>
</struct>
<enum name="AppCapabilityType" since="x.x">
<description>Enumerations of all available app capability types</description>
<element name="VIDEO_STREAMING"/>
</enum>
<struct name="VideoStreamingCapability" since="4.5">
<!- Existing params ->
<param name="additionalVideoStreamingCapabilities" type="VideoStreamingCapability" array="true" minvalue="1" maxvalue="99" mandatory="false" since="X.X">
</param>
</struct>OnAppCapabilityUpdated information to HMI.Author to Revise. Add HMI Guidelines:
If the change is a PIP-type change, scale the current video stream to the size you desire without going through this flow. Touches should never be passed to the app. Either a system menu/buttons should be displayed when selected, or the selection should bring the user immediately back to the full-screen app.
If the change is a split-screen type change (and cannot be handled by scaling the original video), then go through this flow. If the app does not support your custom split-resolution, either don't allow the user to put the app in split screen, or scale the video stream to a size that fits the window and use "black bars" to fill the rest of the window. Touches may be passed to the app in certain limited cases. (1) The window must have a width of at least 3" and a height of at least 3", (2) The touches must be offset for the developer so that 0,0 continues to be in the top-left corner of the app window. If these requirements are not met, either a system menu / buttons should be displayed when selected, or the selection should bring the user immediately back to the full-screen app.
yoooriii
commented
4 years ago 3.2. How do we notify the app dev on changes?
iOS part
I propose to add add a new delegate property to SDLStreamingVideoLifecycleManager and pass it in the configuration
SDLStreamingMediaConfiguration as a new property:
@property (weak, nonatomic, nullable) id<SDLStreamingVideoDelegate> streamingVideoDelegate;Where SDLStreamingVideoDelegate implements following methods:
- (void)videoStreamingSizeDidUpdate:(CGSize)displaySize;When SDL lib receives the struct SDLVideoStreamingCapability it must compare it with the client app provided supportedLandscapeStreamingRange and supportedPortraitStreamingRange and finds a match.
When a match found then the video streaming delegate will be informed with the method:
-videoStreamingSizeDidUpdate:(CGSize)size and video streaming service starts as usual.
dboltovskyi
commented
4 years ago 3.2. How do we notify the app dev on changes?
Android part
Requied call back is implemented in PoC. Please find details: https://github.com/kostyaBoss/sdl_java_suite/blob/poc/fork/video_streaming_with_capabilities_update/android/hello_sdl_android/src/main/java/com/sdl/hellosdlandroid/SdlService.java#L535
dboltovskyi
commented
4 years ago 5. We propose an approach of restarting the video service with new streaming parameters. In this way we don't expect changes on a protocol level and retain existing negotiation sequence.
This is already implemented in both PoCs for Android and iOS.
joeljfischer
commented
4 years ago 5. Your suggestion conflicts with @kshala-ford's comments here:
At all time the video service session must remain open. it must not be stopped and restarted otherwise a huge delay is added for TLS handshake and encryption init.
I agree with Kujtim here. We ought not stop and restart the service or this will lead to a poor user experience.
dboltovskyi
commented
4 years ago 5. We did an internal analysis and found out Stop/Start approach works fine. As for TLS handshake it will run only once at the start of any service in protected mode (RPC, VIDEO or AUDIO). Once secure channel established it then used when starting a new service in protected mode. This is also applicable for service restart.
@kshala-ford Could you please confirm you're Ok with Stop/Start approach.
kshala-ford
commented
4 years ago Yes this is the case. The TLS handshake is performed only once. The handshake is not performed again when sending another StartService. The handshake authenticates the whole app transport connection, not every individual SDL protocol session.
@joeljfischer This was my major concern but the author already investigated here and explained that the handshake is performed only once. The effort to add new protocol frame types gives us probably a few milliseconds only. This is why I wrote
I went through the proposal again and talked to the author and I agree my suggestions are close to the proposed solution. Therefore please disregard my comment.
kboskin
commented
4 years ago 3.1 What information do we need from app dev?
Providing code snippets below to describe idea on high-level
To support different screen resolutions the client App must configure SDLStreamingVideoLifecycleManager properly.
For this there were added 2 new properties to SDLStreamingMediaConfiguration class:
@property (strong, nonatomic, nullable) SDLSupportedStreamingRange *supportedLandscapeStreamingRange;
@property (strong, nonatomic, nullable) SDLSupportedStreamingRange *supportedPortraitStreamingRange;
`SDLSupportedStreamingRange` implemented as it was proposed earlier with a little change:
@interface SDLSupportedStreamingRange : NSObject
// The minimum supported normalized aspect ratio, Min value is 1
@property (nonatomic, assign) float minimumAspectRatio;
// The maximum supported normalized aspect ratio, Min value is 1
@property (nonatomic, assign) float maximumAspectRatio;
// The minimum supported diagonal screen size in inches, defaults to 0 (0 matches any size)
@property (nonatomic, assign) float minimumDiagonal;
// The minimum resolution to support, it overrides .minimumAspectRatio
@property (nonatomic, assign) float minimumResolution;
// The maximum resolution to support, it overrides .maximumAspectRatio
@property (nonatomic, assign) float maximumResolution;
// Check if the argument is within the [.minimumResolution, .maximumResolution] range
- (BOOL)isImageResolutionInRange:(SDLImageResolution*)imageResolution;
// Check if the argument is within the [.minimumAspectRatio, .maximumAspectRatio] range
- (BOOL)isAspectRatioInRange:(float)aspectRatio;
<……>
@endIf the app does not provide either supportedLandscapeStreamingRange or supportedPortraitStreamingRange then SDLStreamingVideoLifecycleManager will take preferred resolution supported by the head unit.
The most important here is SDLStreamingVideoLifecycleManager which implements the match logic and chooses which resolution it should take from all resolutions supported by the head unit.
When it receives system capabilities in the callback:
- (void)sdl_displayCapabilityDidUpdate:(SDLSystemCapability *)systemCapability
it has to check all video streaming capabilities against the app provided min and max image resolutions:
- (SDLVideoStreamingCapability * __nullable)matchVideoCapability:(SDLVideoStreamingCapability*)videoStreamingCapability {
if (!videoStreamingCapability) {
return nil;
}
NSArray <SDLVideoStreamingCapability*>* allCapabilities = [videoStreamingCapability allVideoStreamingCapabilitiesPlain];
if (!self.supportedLandscapeStreamingRange || !self.supportedPortraitStreamingRange) {
// there is nothing to match, return any (1st) object
return [allCapabilities firstObject];
}
for (SDLVideoStreamingCapability* nextCapability in allCapabilities) {
SDLImageResolution *imageResolution = [nextCapability makeImageResolution];
if ([self.supportedPortraitStreamingRange isImageResolutionInRange:imageResolution] ||
[self.supportedLandscapeStreamingRange isImageResolutionInRange:imageResolution]) {
return nextCapability;
}
const float diagonal = nextCapability.diagonalScreenSize.floatValue;
if (0 < diagonal) {
if (diagonal >= self.supportedPortraitStreamingRange.minimumDiagonal ||
diagonal >= self.supportedLandscapeStreamingRange.minimumDiagonal) {
return nextCapability;
}
}
const float ratio = nextCapability.preferredResolution.normalizedAspectRatio;
if (1 <= ratio) {
if ([self.supportedPortraitStreamingRange isAspectRatioInRange:ratio] ||
[self.supportedLandscapeStreamingRange isAspectRatioInRange:ratio]) {
return nextCapability;
}
}
}
return nil;
}The full iOS implementation is available in the following repo: GitHub
In SdlRemoteDisplay's implementations by developers
@Override
public Integer getMinSupportedWidth() {
return 800;
}
@Override
public Integer getMinSupportedHeight() {
return 600;
}
@Override
public Integer getMaxSupportedWidth() {
return 1600;
}
@Override
public Integer getMaxSupportedHeight() {
return 1200;
}
@Override
public double getMinScreenDiagonal() {
return 6;
}
@Override
public Double getMinAspectRatio() {
return 1.;
}
@Override
public Double getMaxAspectRatio() {
return 4.;
}In SdlRemoteDisplay
public abstract Integer getMinSupportedWidth();
public abstract Integer getMinSupportedHeight();
public abstract Integer getMaxSupportedWidth();
public abstract Integer getMaxSupportedHeight();
public abstract double getMinScreenDiagonal();
public abstract Double getMinAspectRatio();
public abstract Double getMaxAspectRatio();In VideoStreamManager
List<VideoStreamingCapability> getSupportedCapabilities(
Integer constraintHeightMax,
Integer constraintHeightMin,
Integer constraintWidthMax,
Integer constraintWidthMin,
Double constraintDiagonalMin,
Double aspectRationMin,
Double aspectRationMax
){
List<VideoStreamingCapability> validCapabilities = new ArrayList<>();
List<VideoStreamingCapability> allCapabilities = getMockedAllCapabilities();
// get the first one - the Desired resolution to guarantee streaming will start
validCapabilities.add(allCapabilities.get(0));
for (VideoStreamingCapability capability : allCapabilities) {
int resolutionHeight = capability.getPreferredResolution().getResolutionHeight();
int resolutionWidth = capability.getPreferredResolution().getResolutionWidth();
double diagonal = capability.getDiagonalScreenSize();
if (diagonal < constraintDiagonalMin) {
continue;
}
if (!(aspectRationMax > aspectRationMin && aspectRationMin > 0)) {
if (constraintHeightMax == null && constraintHeightMin == null) {
continue;
}
}
if (resolutionHeight > 0 && resolutionWidth > 0 && constraintHeightMax != null && constraintHeightMin != null)
{
if (!(resolutionHeight > constraintHeightMin && resolutionHeight < constraintHeightMax)) {
continue;
}
if (!(resolutionWidth > constraintWidthMin && resolutionHeight < constraintWidthMax)) {
continue;
}
}
validCapabilities.add(capability);
}
return validCapabilities;
}Finally params are passed to getSupportedCapabilities method in VideoStreamManager, which has access to remoteDisplay
The full implementation could be found here
dboltovskyi
commented
4 years ago For convenience here is the summary of responses for Open items:
3.1. Proposed public API changes for iOS and Android: comment-643313592
3.2. Examples of notification on VSC changes for iOS: comment-642797352 and Android: comment-642852789
5. Proposed sequence for video streaming resolution switching comment-643288059 and comment-643292703
theresalech
commented
4 years ago @dboltovskyi, thank you! We are okay with your proposed sequence for item 5, and don't believe any further action is required for this item. For 3.1 and 3.2, we'd like to return this proposal for the revisions summarized in this comment, and have these code snippets added to the proposal as part of that revision. We may have additional feedback on the code snippets, but will provide those in the next round of review.
theresalech
commented
4 years ago The Steering Committee voted to return this proposal for revisions. The author will revise to include the agreed upon items (1, 6, 7, and 9) included in this comment, and to include the code snippets for iOS/Android manager-level public API changes (item 3). It was noted that there may be additional feedback on the public API changes in the next review, but reviewing these within the proposal will be clearer than across comments on the review issue. There was discussion about accepting this proposal with revisions, but it was determined that, given the number of changes requested and public API changes review noted above, returning this proposal for revisions is the best way to allow members to review the proposal as a whole before voting to accept.
theresalech
commented
4 years ago The author has updated this proposal to reflect the requested revisions, and the revised proposal is now under review until June 30, 2020. Exact changes made since the last version of this proposal was reviewed can be found in this merged PR: https://github.com/smartdevicelink/sdl_evolution/pull/1003.
dboltovskyi
commented
4 years ago 10.
We propose to remove AppVideoStreamingCapability struct as it's not required (and not used) since whole VideoStreamingCapability struct is utilized for new additionalVideoStreamingCapabilities parameter.
theresalech
commented
4 years ago @dboltovskyi , after reviewing the revised proposal, we have minor comments about the iOS and Java Suite implementations (item 3):
3.
For iOS, we think that the property name streamingVideoDelegate can be simplified to just delegate.
For both, we'd rather see a naming like VideoStreamingRange as it is more descriptive that this is specific to video streaming, but also that it is simply a range and the context in which it is used will define if it is "supported" or not.
Additionally, we've responded to your new comment and provided another new comment (11) below:
10.
We disagree, and think it would be better to change the AppCapability struct to take a AppVideoStreamingCapability type in the videoStreamingCapability param. This would allow us to expand that type in the future if needed. It would mean that the supportedResolutions would need to match the resolutions sent by the head unit.
11. We would like to propose changes for HMI Integration Guidelines: The HMI should stop displaying video to the user after resizing the window instead of waiting for the StopStream request from Core. Video would resume after the video service is restarted.
atiwari9
commented
4 years ago @theresalech
It is better to provide entirety of VSC from app side as well rather than just ImageResolution. If you do want to use a new struct AppVideoStreamingCapability for scalability reasons, then i'd recommend to change supportedResolutions -> videoStreamingCapability (type=VideoStreamingCapability) with-in that struct.
joeljfischer
commented
4 years ago 10.
@atiwari9 That was my initial thought as well, but I think it's unnecessary. All that the mobile side has to do is state which of the head unit's video capabilities it supports. Because there should not be multiple capabilities from the head unit with the same ImageResolution, the app declaring that it supports video capabilities with those resolutions should be enough, right?
atiwari9
commented
4 years ago @joeljfischer
Scale is also an important factor in determining the true resolution here. Now we can set some rules around ImageResolution being the calculated resolution after scale, but that will create additional computation on HMI side and add some possible error entry points. Since HMI is sending the supported resolutions in one format(all of VSC), then for simplicity it should receive the response back from app in same format. This would also save SDL managers some computation to trim the VSCs or recalculate resolution after applying scale etc..dboltovskyi
commented
4 years ago @theresalech
3. We agree with provided suggestion.
11. We agree with proposed change.
Sohei-Suzuki-Nexty
commented
4 years ago 11.
The HMI should stop displaying video to the user after resizing the window instead of waiting for the StopStream request from Core.
Why aren't you waiting for a StopStream request? Since the state management is not clear, there are some concerns that problems will occur.
theresalech
commented
4 years ago @dboltovskyi @atiwari9 @Sohei-Suzuki-Nexty , please find responses and one new comment from the PM below. Thank you!
10.
After internal discussions, we believe we can use VideoStreamingCapability struct for the AppCapability feature. While each capability will have to be reviewed to see if they can be reused for both system and app capabilities, we do feel confident enough to reuse VideoStreamingCapability for both.
11. The reason for that is to ensure that the video stream is not displayed to the user in a distorted format. The state flow would be "user attempts to resize" -> "video stops streaming / head unit sends OnAppCapabilityUpdated" -> "HMI resizes / waits for stop and start of video stream" -> "display new video stream"
12. HMI needs to be added as an impacted platform.
Shohei-Kawano
commented
4 years ago @theresalech -san Thank you for reply.
joeljfischer
commented
4 years ago @Sohei-Suzuki-Nexty
theresalech
commented
4 years ago We did not have a quorum present during the 2020-07-07 Steering Committee meeting, so voting on this proposal could not take place. This proposal will remain in review until 2020-07-14.
theresalech
commented
4 years ago @dboltovskyi, given the additional questions from SDLC members regarding Item 11, we'd like to add a bit more detail to the HMI Integration Guidelines, and suggest the following:
Add the following to the HMI Integration Guidelines: The HMI should stop displaying video to the user after resizing the window instead of waiting for the StopStream request from Core. Video would resume after the video service is restarted. See the following example state flow: "user attempts to resize" -> "hmi stops displaying video -> head unit sends OnAppCapabilityUpdated" -> "HMI resizes / waits for stop and start of video stream" -> "display new video stream".
HMI Integration Guidelines updates should include a flow chart documenting this behavior.
dboltovskyi
commented
4 years ago @theresalech We agree to add suggested notice to HMI Integration Guidelines. Instead of creating a new diagram we would propose to extend existing 'Resolution Switching' diagram with additional actions.
theresalech
commented
4 years ago The Steering Committee voted to accept this proposal with the following revisions:
streamingVideoDelegate to delegate. For Both iOS and Java Suite, use VideoStreamingRange.VideoStreamingCapability struct for the AppCapability feature. While each capability will have to be reviewed to see if they can be reused for both system and app capabilities, we do feel confident enough to reuse VideoStreamingCapability for both.Resolution Switching diagram to document this behavior.
The HMI should stop displaying video to the user after resizing the window instead of waiting for the StopStream request from Core. Video would resume after the video service is restarted. See the following example state flow: "user attempts to resize" -> "HMI stops displaying video" -> "head unit sends OnAppCapabilityUpdated" -> "HMI resizes/waits for stop and start of video stream" -> "display new video stream".
HMI as an impacted platform.theresalech
commented
4 years ago @dboltovskyi please advise when a new PR has been entered to update the proposal to reflect the agreed upon revisions. I'll then merge the PR so the proposal is up to date, and enter issues in respective repositories for implementation. Thanks!
theresalech
commented
4 years ago The author has updated the proposal to include the requested revisions, and implementation issues have been entered:
theresalech
commented
4 years ago Added impacted platforms, based on accepted proposal revisions: https://github.com/smartdevicelink/sdl_evolution/issues/1059.
Hello SDL community,
The review of the revised proposal "SDL 0296 - Possibility to update video streaming capabilities during ignition cycle" begins now and runs through June 30, 2020. The original review of this proposal took place March 24 - June 16, 2020. The proposal is available here:
https://github.com/smartdevicelink/sdl_evolution/blob/master/proposals/0296-Update-video-streaming-capabilities-during-ignition-cycle.md
Reviews are an important part of the SDL evolution process. All reviews should be sent to the associated Github issue at:
https://github.com/smartdevicelink/sdl_evolution/issues/983
What goes into a review?
The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of SDL. When writing your review, here are some questions you might want to answer in your review:
More information about the SDL evolution process is available at
https://github.com/smartdevicelink/sdl_evolution/blob/master/process.md
Thank you, Theresa Lech
Program Manager - Livio theresa@livio.io