Integrate latest camera selection API from getUserMedia

[jywarren]

Potentially using a new getUserMedia shim library?

https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#Parameters

Maybe using https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/enumerateDevices

Code for this is here:

https://github.com/publiclab/infragram/blob/f6f7919493b5ab9e7cc8fc873f8bddf3ff5f4599/src/io/camera.js#L30-L41

We can apparently also specify different resolutions!

{
  audio: true,
  video: {
    width: { min: 1024, ideal: 1280, max: 1920 },
    height: { min: 776, ideal: 720, max: 1080 }
  }
}

Let's do both!

[jywarren]

Noting this was done in spectral-workbench.js here -- using Temasys - https://github.com/publiclab/spectral-workbench.js/pull/173

Also noting: https://github.com/webrtcHacks/WebRTC-Camera-Resolution

[jywarren]

This could be integrated with the initial work of this PR where we try to dynamically set the pixel resolution/size: https://github.com/publiclab/infragram/pull/21

[Somya-Singhal]

Hello @jywarren, I am an outreachy applicant. I would like to work on this issue.

[jywarren]

Hello @Somya-Singhal this will be part of the work of the Outreachy project itself, so we aren't quite ready to begin coding. However, if you would like to help research where changes will have to be made, or add other contextual information here, to prepare, that would be appreciated! Thank you!!

[Somya-Singhal]

I have gone through the link you provided above and found that we could some more features like: 1) implement it in a way that it could also be used in old browsers.

2) In some cases, lower frame rates may be desirable, so we could add bandwidth restrictions. var constraints = { video: { frameRate: { ideal: 10, max: 15 } } };

3) For mobile devices, rear camera would be preferred over the front camera, so we could add: { audio: true, video: { facingMode: { exact: "environment" } } }

If we want to add both, front as well as rear camera for mobile devices, we can also use this: var front = false; document.getElementById('flip-button').onclick = function() { front = !front; }; var constraints = { video: { facingMode: (front? "user" : "environment") } };

[Forchapeatl]

The integration of Temasys on infragram has been a success . Now iOs users can perform vegetation analysis over Webcam stream.

Problem: only The first input frame is captured for both live stream and Prerecorded video on Safari (iOS)

Hello @jywarren , @TildaDares , @cesswairimu , @stephaniequintana and @RaviAnand111 please help me test the webcam and local video behavior at your end https://forchapeatl.github.io/infragram/indexT.html#

https://user-images.githubusercontent.com/24577149/182050116-24807581-59ab-42bd-82c2-25883c8e66f2.mp4

[RaviAnand111]

Hey @Forchapeatl , I tried the link and its working great on chrome in Manjaro OS, both the image and live video. Thanks

[stephaniequintana]

@Forchapeatl - I'm on a MacBook Air (macOS Catalina) and both uploading a video and connecting to the webcam are working great on Chrome and Firefox, but neither work on Safari.

For the webcam on Safari, I get a black screen after clicking the webcam button and when I try to upload a video (mp4) the selection is disabled in the file list and thus I am unable to select it. Perhaps this is a simple setting on my end(?) - Please let me know if you would like me to try another browser or video format.

[jywarren]

Hi @stephaniequintana if you don't first try to connect to a webcam, does video upload work normally in Safari? That is, is the attempt to connect to a webcam interfering with video upload, or does it not work at all in Safari? Thank you!

[stephaniequintana]

@jywarren, I believe it's due to a bug in Safari that hasn't been addressed. I found a short discussion on stack overflow (from 2014!!) about it - the suggested "solution" does not work for me - and am hoping it's isolated to macOS Catalina. I didn't dive deeper than this, though.

The webcam in not causing interference. This happens whenever I try to upload a video to the site in Safari.

[Forchapeatl]

@jywarren , @stephaniequintana and @RaviAnand111 , I appreciate the feedbacks. Thank you!

[Forchapeatl]

Hello @jywarren . This is the implementation of P5.js https://forchapeatl.github.io/infragram/indexP.html

Existing Limitations

• CreateCaputure( ) gets called only within the setup( ) function.
• The setup() gets called once.
• Hence we can stream once.
• We won't be able to toggle between camera and video streams.

References https://p5js.org/reference/#/p5/createCapture https://p5js.org/reference/#/p5/setup https://forchapeatl.github.io/infragram/indexP.html

Request

Please may we use getUserMedia API to move forward on camera stream. Here is a working example on safari https://codepen.io/ocinpp/pen/EpbXKz . The trick on safari is to use async functions

[jywarren]

I think this sounds great and if it works in safari iOS I'm super happy. I don't think it has to toggle back and forth. People can reload. Let's do this! Is it ready to merge?

I think it's interesting to know if it works on Firefox in Arch Linux but if it doesn't, let's try thinking on that in a separate issue/pr. Thanks all!

[Forchapeatl]

@jywarren Please I am abit confused. P5.js or getUserMedia ? They both work on Safari

[jywarren]

@jywarren, I believe it's due to a bug in Safari that hasn't been addressed. I found a short discussion on stack overflow (from 2014!!) about it - the suggested "solution" does not work for me - and am hoping it's isolated to macOS Catalina. I didn't dive deeper than this, though.

The webcam in not causing interference. This happens whenever I try to upload a video to the site in Safari.

Regarding this shall we open it in a new issue? Thank you?

[jywarren]

Oh I'm sorry, I thought p5.js uses getusermedia doesn't it? If you can do it using getusermedia just like p5js implements it I think it will work well, p5js is more an example of how to use it!

On Thu, Aug 4, 2022, 6:32 PM FORCHA @.***> wrote:

@jywarren https://github.com/jywarren Please I am abit confused. P5.js or getUserMedia ? They both work on Safari

— Reply to this email directly, view it on GitHub https://github.com/publiclab/infragram/issues/39#issuecomment-1205833987, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAF6J4HZN5IX7V6BOYETMLVXRAIRANCNFSM4GNCIDSQ . You are receiving this because you were mentioned.Message ID: @.***>

[Forchapeatl]

@jywarren It seems all libraries (Temasys, P5.js and getUserMedia) work fine on safari problem is Safari doesn't recognize setInterval() https://stackoverflow.com/questions/12683510/ios-6-safari-setinterval-doesnt-get-fired https://github.com/publiclab/infragram/blob/f40efdae8f6aa14107fe57eb83398767464f4e78/src/Infragram.js#L41-L46

@jywarren, I am sorry for the whole confusion. I just discovered this bug/malfunction now. Thank you I think getUserMedia will help me maintain the codebase. Thank you!

[Forchapeatl]

Safari doesn't recognize setInterval

[jywarren]

Oh whoa, really! Wild. It's listed ok here, it's a pretty fundamental JavaScript feature...

https://caniuse.com/?search=setinterval

[jywarren]

But is it only safari iOS 6 then? Does that happen to be the version you're testing with?

[jywarren]

Maybe if it's just the one ios version that's older, we can live with it?

[Forchapeatl]

Yes. I works on chrome IOS smoothly. But on safari nothing happens after capturing the first frame. Please have a look here https://forchapeatl.github.io/infragram/indexs.html

[stephaniequintana]

@jywarren

Regarding this shall we open it in a new issue?

I don’t know how we could address it, but opening an issue is a good place to document that Safari on MacOS Catalina does not support video upload. We can also use it as a place to collect info on other Mac operating systems…(although my limited research leads me to believe it is isolated to Catalina)

[Forchapeatl]

@stephaniequintana Please could you test this https://forchapeatl.github.io/infragram/indexs.html . Just the webcam feature on your MacOS . please

[stephaniequintana]

edit below...

[Forchapeatl]

@stephaniequintana , Please,please test on all browsers currently existing on your MacOS. Please

[Forchapeatl]

@stephaniequintana , Please,please test on all browsers currently existing on your MacOS. Please

It doesn't work on my Phone Safari version 15.5. PLease could you try it on your phone. Sorry for the trouble.

[stephaniequintana]

@Forchapeatl - No trouble at all, I'm glad I can help... On my Mac, it works on Chrome and Firefox - but i'm still getting a black screen on Safari. On my iPhone, it works with the Google Chrome app but it is just picking up the first frame on Safari.

[Forchapeatl]

Thank you so much @stephaniequintana

[stephaniequintana]

Is Safari the new Internet Explorer? 😆

[Forchapeatl]

Steaming works fine on Safari but setInterval() is problematic

https://user-images.githubusercontent.com/24577149/182980699-7fe82a6d-c329-49d8-aa05-ea502a805e42.mp4

We have an issue with the setInterval() function. may be it doesn't update large buffers

@jywarren should I prepare a PR for the good steaming functionality on safari but open an issue on setInterval( ) malfunction on safari?

[jywarren]

Yes, let's just move ahead and open a separate issue for the Safari iOS thing. @Forchapeatl would you mind opening it? We may also be able to put a notice that it works in iOS Chrome, so people can still use it.

Safari on MacOS Catalina does not support video upload

I think we'd better open a separate issue for this since it's not iOS, just so the two don't get confused with one another. @stephaniequintana would you mind?

Thanks, both of you!

[jywarren]

But is it ios 6 only? Thanks for looking into it so carefully!!

On Thu, Aug 4, 2022, 8:33 PM FORCHA @.***> wrote:

@stephaniequintana https://github.com/stephaniequintana Please could you test this https://forchapeatl.github.io/infragram/indexs.html . Just the webcam feature on your MacOS . please

— Reply to this email directly, view it on GitHub https://github.com/publiclab/infragram/issues/39#issuecomment-1205911395, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAF6J2Q4BU2PNTRGUMLQKDVXROMFANCNFSM4GNCIDSQ . You are receiving this because you were mentioned.Message ID: @.***>

[proy03]

@jywarren hey Jeffery, I would love to contribute to this issue.