search

jywarren / spectral-workbench

Web-based tools for collecting, analyzing, and sharing data from a DIY spectrometer
http://spectralworkbench.org
GNU General Public License v3.0
95 stars 12 forks source link

native iOS port of mobile client #116

Closed jywarren closed 9 years ago

jywarren commented 12 years ago

i was thinking I'd try wrapping the spectral workbench platform in PhoneGap and patch camera access in to replace the mediastream API used on Opera/Android. That may end up being harder than I expect...

Which is to say, the current mobile version is web-based, and only runs on Opera for Android right now. But it is pretty nice: http://spectralworkbench.org/capture

My ideal is to wrap this already-working system in a native app so that any future interface changes can simply be pushed out on all platforms at once.

gitizenme commented 12 years ago

Have you considered using Appcelerator Titanium? Check out: http://appcelerator.com

I have experience in coding Apps on both Titanium and PhoneGap and IMHO Titanium is the better of the two!

jywarren commented 12 years ago

I haven't -- would be interested to see the video frame access API. Is AT open source or compatible with? That's what's nice about PhoneGap/Cordova

gitizenme commented 12 years ago

The SDK and the IDE are open source - https://github.com/appcelerator

I'm not sure what your requirements are for video frame access are but writing a native iOS module to get at the video frames from the camera is one possibility: http://docs.appcelerator.com/titanium/2.1/#!/guide/iOS_Module_Development_Guide

jywarren commented 12 years ago

Darn, iOS 6 will support camera capture (so we might be able to do a single-shot version) but not getUserMedia video access: http://blog.merge.nl/shorts/web-app-html5-camera-capture-getusermedia-file-api-browser-support

gitizenme commented 12 years ago

I have not had time to poke around in the code. Does the analysis require a realtime video stream or can it be feed a series of captures?

gitizenme commented 12 years ago

Okay, I spent a few minutes looking at the code. This lead me to the underlying video capture library - WebRTC. I think I see a path from there to PhoneGap and/or Appcelerator.

jywarren commented 12 years ago

Cool! Sorry I've been scrambling and haven't been able to check in. Basically we need access to raw image data, where we can get the R, G, or B value of any pixel from the latest frame of video, in JavaScript. The rest will be easy to integrate. On Aug 30, 2012 6:07 PM, "gitizenme" notifications@github.com wrote:

Okay, I spent a few minutes looking at the code. This lead me to the underlying video capture library - WebRTC. I think I see a path from there to PhoneGap and/or Appcelerator.

— Reply to this email directly or view it on GitHubhttps://github.com/jywarren/spectral-workbench/issues/116#issuecomment-8176332.

jywarren commented 12 years ago

We could rejigger the interface for stills but live video is much much nicer. On Aug 30, 2012 6:07 PM, "gitizenme" notifications@github.com wrote:

Okay, I spent a few minutes looking at the code. This lead me to the underlying video capture library - WebRTC. I think I see a path from there to PhoneGap and/or Appcelerator.

— Reply to this email directly or view it on GitHubhttps://github.com/jywarren/spectral-workbench/issues/116#issuecomment-8176332.

jywarren commented 11 years ago

We are asking this question basically, and probably need to develop a PhoneGap plugin: http://stackoverflow.com/questions/10909748/getusermedia-shim-for-phonegap-cordova?rq=1

I'm looking into the barcode plugin since I imagine it does something similar -- intercepts video capture without showing a video interface widget, and runs some analysis on the video stream by pixel...

jywarren commented 11 years ago

Super -- more investigation shows that the Barcode plugin (which the more I think about it, the closer it seems architecturally to what we want to do -- record intensity variation in a horizontal row of pixels) uses the AVFoundation API in iOS:

https://developer.apple.com/library/mac/#documentation/AudioVideo/Conceptual/AVFoundationPG/Articles/04_MediaCapture.html#//apple_ref/doc/uid/TP40010188-CH5-SW2

which allows for "silent" video capture and processing. We gotta look at that barcode plugin closer.

seandoyle commented 11 years ago

I've just installed the latest phonegap on my Mac & I'll download the barcode plugin. I don't have any access to an Android device at the moment - but I was wondering if it would make sense to use OpenCV in the plugin if it provides a uniform video API for iOS and Android: http://docs.opencv.org/trunk/doc/tutorials/ios/video_processing/video_processing.html I have no particular reason for pushing OpenCV except that I've been looking for an excuse to learn more about it :-).

Is there any value in the additional image processing libraries that this would offer or is it just additional weight?

jywarren commented 11 years ago

I dunno much about OpenCV, but we were planning on writing a lot of the image rec and matching code in JavaScript so it would be portable to other devices. That said, if you can bake OpenCV into the codebase and it's a more standard way to fetch video frames than writing our own interface code, i don't see a problem. It wasn't clear to me if we could simply copy the barcode codebase and just tweak it to get spectral intensity, but it sounds worth a try.

On Sat, Oct 6, 2012 at 2:08 PM, Sean Doyle notifications@github.com wrote:

I've just installed the latest phonegap on my Mac & I'll download the barcode plugin. I don't have any access to an Android device at the moment

  • but I was wondering if it would make sense to use OpenCV in the plugin if it provides a uniform video API for iOS and Android:

http://docs.opencv.org/trunk/doc/tutorials/ios/video_processing/video_processing.html I have no particular reason for pushing OpenCV except that I've been looking for an excuse to learn more about it :-).

Is there any value in the additional image processing libraries that this would offer or is it just additional weight?

— Reply to this email directly or view it on GitHubhttps://github.com/jywarren/spectral-workbench/issues/116#issuecomment-9200670.

seandoyle commented 11 years ago

I'll have to look at this again tomorrow - I'm having trouble getting the app running. The Cordova distro needed some massaging under Xcode 4.5 (remove the armv6 and armv7s architectures) and a bunch of other small things - and now it's launching but on both the simulator and iPhone 4 it's only partly starting. The last console message is Multi-tasking -> Device: YES, App: YES which is from the CDVViewController - but the javascript initializations don't finish.

My assumption is that I should uninstall cordova-lib and start again.

I've only taken a brief peek at the code underneath - but my instinct is that you're right that this would be a good template and that OpenCV might be overkill and add some complexity.

This should be simpler than it's been which probably means I'm doing something wrong :-).

jywarren commented 11 years ago

Moving this issue mostly to #174

jywarren commented 9 years ago

After much discussion we'll be focusing on our HTML5 web app; thanks!