Chrome user doesn't get any notification when iOS user is sharing the screen and tries to share the file also

[ImranBinShoukat]

When iOS user is sharing the screen and it tries to transfer a file also then Chrome user doesn't get any notification and File Transfer doesn't work.

Repro Steps:

1. Join Conference
2. Share screen (iOS user)
3. Share a file (iOS user)

[sumairasaeed]

Looked into this. unable to find any error in iOS logic. this might be error on web logic that file notification is not shown on web. Will have to debug this more with sojharo or zarmeen tomorrow.

On Wed, Apr 13, 2016 at 10:47 PM, Imran Shoukat notifications@github.com wrote:

When iOS user is sharing the screen and it tries to transfer a file also then Chrome user doesn't get any notification and File Transfer doesn't work.

Repro Steps:

1. Join Conference
2. Share screen (iOS user)
3. Share a file (iOS user)

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/Cloudkibo/iOS/issues/170

[jekram]

@sumairasaeed Do we know if the notification reached Chrome but not display?

[sumairasaeed]

Yes chrome receives notification and it is shown on web chat. However, when web tries to process it, it gets error. I have debugged it further and here are the findings:

It is an issue on web side. When we share screen from iOS, it goes through data channel and when iOS do file sharing, it also goes through data channel. So, web is unable to identify if the image chunk received is a chunk of iOS screen or File. Thus, it shows error on console as shown:

One solution of the problem can be to stop iOS from sharing file while it is sharing screen. This diagram below shows the logic of how we do screen sharing:

[jekram]

Thanks

@zarmeen92 Please work with @sumairasaeed to resolve this.

[zarmeen92]

sir I discussed with sumaira,on iOS when we share screen it uses datachannel that is also used by fileTransfer (since we have only one datachannel ) therefore the user is not able to share file while screen sharing is going on

[jekram]

@sumairasaeed

Why are we going in circles? Is this a core design issue. Why are we finding this out now?

I was told that it would work with /conference and we do not need /webmeetings

@sumairasaeed please meet with @sojharo and sort this out.

Copying @sojharo

Do we have this issue between two browsers?

[sojharo]

This issue is not between two browsers. Screen sharing and file transfer on two browsers can't do conflict as file transfer uses data channel and screen sharing happens on streams.

From the above discussion, we can conclude that iPhone uses DataChannel for Screen Sharing (iOS to web). When screen is being shared from iPhone, the continuous images go from iOS to web using datachannel in form of binary chunks.

The file transfer also uses the datachannel to send file binary chunks. When they both happen at same time (iOS sending screen images and file transfer chunks), the part on the web becomes confused and don't understand whether the binary chunk is for screen sharing or file transfer.

This happens when mobile wants to share screen and file together.

DataChannel either sends the text or binary data at one time. Zarmeen suggested creating a separate datachannel when iOS wants to share screen so that it won't conflict with datachannel sending file. This might increase the complexity like webmeeting.

I was of opinion that we should send text before each binary to indicate that upcoming binary belongs to file or screen image. Zarmeen pointed out it might do uncertainity due to latency in data transmission.

We are thinking more on this and would let you know soon on this.

[jekram]

There needs to be consistency in our design.

All clients should behave the same way.

We cannot have special logic for each end device.

If the browser is receiving the data we cannot have three ways of getting the data (from browser, Android, iOS). In fact the browser should NOT know who is sending the data.

[sojharo]

Yes, I think we should once again revisit the webrtc community to know if they have given screen sharing support in WebRTC on Android.

This bug was also opened that webrtc on android doesn't support screen sharing.

https://bugs.chromium.org/p/webrtc/issues/detail?id=3620

This is why we had to go for the way of sending images of screen using datachannel instead of sending screen as stream.

Skylink plugin also does the same way and sends images of shared screen from android to web using data channel as webrtc android or iOS doesn't support screen sharing:

https://github.com/Temasys/skylink-android-screen-sharing/blob/master/SkylinkShare/app/src/main/java/skylink/temasys/com/sg/skylinkshare/MainActivity.java

Here is the main web site of SkyLink:

https://github.com/Temasys/skylink-android-screen-sharing

http://skylink.io/android/

We can't allow the standard way until webrtc android / iOS library supports it.

On Mon, Apr 18, 2016 at 7:01 PM, Cloudkibo notifications@github.com wrote:

There needs to be consistency in our design.

All clients should behave the same way.

We cannot have special logic for each end device.

If the browser is receiving the data we cannot have three ways of getting the data (from browser, Android, iOS). In fact the browser should NOT know who is sending the data.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Cloudkibo/iOS/issues/170#issuecomment-211393264

Regards,

Sojharo

[jekram]

@sojharo @sumairasaeed

Please describe who these are working for us?

Is the working from Android to Browser? Is this working from Browser to Andriod? Is this working from Browser to iOS?

[sumairasaeed]

yes sir. it is working from browser to iOS,

On Mon, Apr 18, 2016 at 8:23 PM, Cloudkibo notifications@github.com wrote:

@sojharo https://github.com/sojharo @sumairasaeed https://github.com/sumairasaeed

Please describe who these are working for us?

Is the working from Android to Browser? Is this working from Browser to Andriod? Is this working from Browser to iOS?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Cloudkibo/iOS/issues/170#issuecomment-211429369

[sojharo]

Yes, all three works the error of this issue will occur in android to web screen sharing if file transfer starts.

On Mon, Apr 18, 2016 at 8:32 PM, sumairasaeed notifications@github.com wrote:

yes sir. it is working from browser to iOS,

On Mon, Apr 18, 2016 at 8:23 PM, Cloudkibo notifications@github.com wrote:

@sojharo https://github.com/sojharo @sumairasaeed https://github.com/sumairasaeed

Please describe who these are working for us?

Is the working from Android to Browser? Is this working from Browser to Andriod? Is this working from Browser to iOS?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Cloudkibo/iOS/issues/170#issuecomment-211429369

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Cloudkibo/iOS/issues/170#issuecomment-211433448

Regards,

Sojharo

[jekram]

Downgrading this to medium as we have other higher priority.

The next step is a design on how we would resolve this issue. Please work with @sojharo and @zarmeen92 for design.

[sumairasaeed]

Had discussion with @sojharo and here are the possible solutions and their challenges we can think of:

Solution#1: We can attach a header along with image chunk before sending through data channel to indicate if the chunk is of “Screen” or “File”. Data will be sent as a string from iOS so web can read the value of header and see if data is of screen or file. Web then will convert that chunk into binary.

Solution#2: Before sending image chunk(binary data) , each time we first a flag to indicate that next coming chunk is of “screen” or “file” . The difference between this solution and solution#1 is that in this, we are sending flag and image chunk in separate cycles. However, in solution # 1 we are sending both information in single cycle.

Solution#3: We can show a friendly message to user that you will have to pause screen sharing if you wish to share a file. After file sharing is complete, we can resume screen sharing.

Challenges in Solution # 1: We would have to check first if it is possible to send both text (header) and binary (data chunk). I have seen the function of data channel requires the check parameter called isBinaryData. so it should be set to true if data is binary. So, I think it would be difficult to send both text and binary together.

Challenges in Solution # 2: There will be uncertainty involved in this. It is possible that in one cycle we send header that coming chunk is from “screen”, however, due to network speed, it is possible that “file” chunk is sent before “screen” chunk. As, data is sent continuously, It will be difficult to maintain order of sending data and header through data channel.

I tried to code for solution#1 a bit. But needs some change in web code also to test it. We decided not to push anything on production so will test after your demo is done.

[jekram]

@sumairasaeed

Thanks for the update

3 is not a solution so it is not an option.

The valid options are #1, #2, or Mebmeeting.

Q: How others have solved this problem?

Q: What is the work effort for # 1 and #2

[sumairasaeed]

Ans1: Sir, I did not find any application which is doing both 2-way screen sharing(i.e. browser to mobile and mobile to browser) along with file transfer. Most Application offer only one of the two functionalities. However, an app called "Zoom" has implemented file sharing and only one-side screen sharing (i.e. browser to mobile). Currently on cloudkibo, file sharing works fine when screen is shared from web to iOS. But file transfer doesnot work when screen is shared from iOS to web.

Ans#2: Currently we cannot say if Solution # 1 is applicable unless we code and test it. However, it will just take a day to code and test. Effort is no more than one day. Solution # 2 will also take just one day effort but it has the risk involved which is mentioned in last comment.

[ImranBinShoukat]

During testing this issue I found out that when iOS user clicks on share button and select the drive, then iOS screen on chrome browser turns into black and it remains black until iOS shares the file. When iOS shares the file then chrome browser gets a message but doesn't get any notification of receiving file with download file option. Here is the screen shots:

[jekram]

@ImranBinShoukat @sumairasaeed looks like from Imran description that the basic file transfer issue is not working

[sumairasaeed]

First screenshot in which screen turns black is because at that point of time, we went out from our cloudkibo application into iCloud Drive application. We had already discussed that iOS screensharing works within our application only. It pauses(turns blank) if we move out of our app and resumes/starts again when we return back into our application.

Second screenshot shared by @ImranBinShoukat is the same issue of datachannel for which we have decided to implement solution#1(discussed in earlier comments).

[jekram]

@sumairasaeed There can be two File transfer

1. iOS doing file transfer during screen sharing (I understand this that this is the one we are fixing it).
2. iOS doing file transfer not during screen sharing. Is this working?

[sumairasaeed]

yes sir. Without screen sharing, file transfer works fine.

On Wed, Apr 20, 2016 at 8:52 PM, Cloudkibo notifications@github.com wrote:

@sumairasaeed https://github.com/sumairasaeed There can be two File transfer

1.

iOS doing file transfer during screen sharing (I understand this that this is the one we are fixing it). 2.

iOS doing file transfer not during screen sharing. Is this working?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Cloudkibo/iOS/issues/170#issuecomment-212486942

[sumairasaeed]

Worked with @zarmeen on this. Made changes on iOS to send image chunk along with a header. However, it requires changes on web-side to test if it is correctly received on web.

On Wed, Apr 20, 2016 at 8:54 PM, Sumaira Saeed sumaira.syedsaeed@gmail.com wrote:

yes sir. Without screen sharing, file transfer works fine.

On Wed, Apr 20, 2016 at 8:52 PM, Cloudkibo notifications@github.com wrote:

@sumairasaeed https://github.com/sumairasaeed There can be two File transfer

1.

iOS doing file transfer during screen sharing (I understand this that this is the one we are fixing it). 2.

iOS doing file transfer not during screen sharing. Is this working?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Cloudkibo/iOS/issues/170#issuecomment-212486942

[jekram]

@sumairasaeed

How did we validate this?

Did we not know that this will require changes on Web?

[sumairasaeed]

yes sir. Zarmeen and I spent most of time on our other high-priority task. We discussed what needs to be changed on iOS and web and started making those changes. I found that it is not possible to send a binary and string data together in one cycle through data channel. So decided that iOS will send both chunk and header in string form. Then, web will convert image chunk part into binary. I started to make change on iOS and zarmeen started to make changes on web-side. It is not yet completed. We will test it tomorrow IA when we both are done.

On Thu, Apr 21, 2016 at 5:01 AM, Cloudkibo notifications@github.com wrote:

@sumairasaeed https://github.com/sumairasaeed

How did we validate this?

Did we not know that this will require changes on Web?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Cloudkibo/iOS/issues/170#issuecomment-212658442

[zarmeen92]

Sir today I and sumaira sit together and worked on this issue.But on web I am having problem in converting file data from binary to string.On the sending side,file data is not converting as an empty string. I didn't spend much time on it.I have found some solutions for converting ArrayBuffer into string and vice versa.But I need more time to read these solutions in detail

[jekram]

That will be fine. This is a lower priority than KiboEngage. ONLY work on this when you have time available. I want 110% focus on KiboEngage.

[jekram]

@sumairasaeed

Is this issue with /Conference or /Webmeeting?

[sumairasaeed]

This is issue with /conference.

[jekram]

@sumairasaeed Please provide a summary:

What is working?

What is not working?

What is effort to close this task?

[jekram]

what is the update here

provide a summary what needs to be done here?

[sumairasaeed]

This issue is related to kiboConference. Moving it to KiboConference repo now and copying this decision there. https://github.com/Cloudkibo/Kibo_Conference_iOS/issues/5

We can close this from here now