Unable to "read" OPCode in iOS devices

niccolopaganini commented 11 months ago

Tested in the following	Device	iOS version
iPhone 14 Pro	17.1.1	1.5.8
iPhone 8	15.5	1.5.8
iPhone 8	16.7.2	1.5.8

If you're a new user installing it from the App Store, you're greeted with the opening page i.e. where you get to scan/ paste the Opcode.

If you click the scan code button, you get the following:

When you click paste code, you get: Enter the following token you have followed by Invalid Opcode format...

niccolopaganini commented 11 months ago

Note: I was unable to reproduce this on Android devices (atleast, at this juncture) but I had a problem with the devapp (with this issue) in both the emulators.

niccolopaganini commented 11 months ago

Update: On my iPhone 14 Pro, I uninstalled the OpenPATH and the TestFlight app and then installed them both again. I was able to get the scan button working. If you hit the cancel button and try to scan an OPCode again, you will be able to reproduce the error. This time however, you can force close the app and then open it again to scan the QR code

niccolopaganini commented 11 months ago

Update no.2: In the aforementioned device, when you logout and try logging back in again, you get the message:

NREL OpenPATH
-----------------
Scanning Failed:
----------------
"Scanning is already in progress"

in an alert pop-up

niccolopaganini commented 11 months ago

Update no.3: I was trying to capture the OPcode from the Open Access Study page and once I scan the QR code successfully and go to the consent page and click reject and go back to the welcome page again, I am able to scan the code again. However, if I hit scan code and the hit the cancel button, that's when the issue is triggered.

niccolopaganini commented 11 months ago

I spoke with Katie (@the-bay-kay) yesterday and she was unable to reproduce it consistently (tagging her in this issue for her to throw more clarity)

I tried the same on my personal device and I am able to reproduce it. iOS version: 17/1/1 App version: 1.5.6

the-bay-kay commented 11 months ago

... she was unable to reproduce it consistently (tagging her in this issue for her to throw more clarity)

We were able to reproduce the scan code bug on my phone twice, but couldn't find a pattern in its cause. If I cancel mid scan, the issue doesn't seem to consistently trigger (this was on iOS 16.1.1). It should also be noted that the scanner on android does not have a cancel button, so we weren't able to reproduce it there.

While I'm not entirely sure it's the culprit, I do know that our current cordova barcode scanner is a legacy project, having been archived in 2018. It may be worth considering a more contemporary scanner (A quick search lead me to this react-native plugin). I'm not entirely sure this will fix the issue, but it should at least help with consistency across versions, and stability going forward!

the-bay-kay commented 10 months ago

Just came back around to this issue -- spent some time on it this morning, and finally found the steps to reproduce on my phone, an SE using iOS 16.1 (videos below for demonstration): Using the cancel button will not cause the issue (See Video 1). Rather, if you "swipe away" the camera pop up (Video 2), the error occurs. My assumption is, this is because the camera process isn't properly terminated when it's dismissed with a swipe, hence the "scanning in progress!" error.

I have yet to experiment on Android, but ill report back once I've done some more testing. I'm going to spend a bit more time messing with this -- I know Nitish mentioned having issues when specifically selecting the cancel button, so I worry there's more too this. My assumption is, if Android has a similar "dismiss" feature, this bug will also occur there (I do know that my android phones don't have a "cancel" button). Now that we've got an idea of what's going on under the hood, I'll start hunting for a fix!

https://github.com/e-mission/e-mission-docs/assets/98350084/17afa475-ebeb-416d-943a-3c7abf59fa7a

https://github.com/e-mission/e-mission-docs/assets/98350084/d957834e-4908-499d-8bbe-52cc9d617f12

the-bay-kay commented 10 months ago

Just leaving some more data in this: This issue occurs sometimes, but not all the time, when switching applications with the scanner open. I caught a recording of it, but haven't been able to reproduce this consistently since. Will update with further info once I can make this reproducible!

https://github.com/e-mission/e-mission-docs/assets/98350084/845a5519-cb98-4165-92e8-3f18300ded11

the-bay-kay commented 10 months ago

... our current cordova barcode scanner is a legacy project, having been archived in 2018. It may be worth considering a more contemporary scanner (A quick search lead me to this react-native plugin).

I've spent some time diagnosing the issue, but there doesn't seem to be an easy fix -- I don't see an issue with the way we handle the scanner, anyway. Like I mentioned earlier in the thread (above), I think the best choice is to swap out the plugin entirely. Not only should that fix the issue, but using a contemporary plugin should help with stability down the road. I'll spend a little more time researching our options, but I'm assuming I'll go with this react scanner.

the-bay-kay commented 10 months ago

After further research, it seems best practice (according to the lastest qr-plugin project here) is to use the react-native Vision Camera plugin, which itself supports barcode scanning. I'll keep this thread updated as I explore this further!

JGreenlee commented 10 months ago

I don't think we can use Vision Camera because it is a React Native library, and we aren't using React Native. (We are using React Native Web over top of Cordova, which means we can only access native APIs through Cordova.)

I know it's also possible for web APIs to use cameras, so there might be packages that work with React Native Web and circumvent the need to use native camera APIs.

JGreenlee commented 10 months ago

After reading through all this, though, I'm not sure that this issue is a high priority anyway.

My understanding is that the issue is only reproduced on iOS by 'swiping away', or sometimes by clicking "Cancel". It seems to me that simply silencing the error while allowing the scanner to close would be acceptable.

At some point in the future, the project will move away from Cordova anyway, and at that point we can use a React Native library.

the-bay-kay commented 10 months ago

That's my understanding as well! 'Swiping' is the only way I've been able to consistently reproduce the issue. I have to really mess with the app (leave it open for several minutes on the camera, switch apps a few times) to cause it to happen via the cancel button. In my experience, it's a pretty rare occurrence -- it almost never happens from just cancelling like normal, so I don't think it's the most pressing issue.

I'll see if we can silence the error, my hunch is that it may be tricky. If I understand the cordova plugin correctly, the issue is caused when the camera opens, but doesn't exit properly, and instead remains open as a background process. There may be a way to "kill" and re-open the camera, or re-connect to the current session -- I'll look into it!

the-bay-kay commented 10 months ago

After doing some digging, it seems that the only public methods within the cordova-barcode-scanner library are the .scan() and .encode() methods (link). Since the error occurs internally in the Obj-C portion of the plugin when .scan() is called, I'm not sure there is a way to forcefully restart the service (It would be a service on Android, I'm not sure the iOS term...). I'm looking to see if there's workarounds now!

the-bay-kay commented 10 months ago

Just coming back around to this, here's a quick recap of the situation:

The Issue

When scanning on iOS, users may "swipe down" the camera, breaking future scans.
- This results in an error that reads "Scanning Failed: Scanning Already in progress!"
Occasionally, when switching applications during QR scanning, this error will occur.
Once this fail state has been reached, users must force-quit the application in order to re-open the QR code scanner.

The Culprit

The Cordova Barcode Scanner plugin (link) we rely on has been marked 'read-only' since 2018, and is no longer supported.
After doing a fair bit of reading on iOS camera APIs, I believe the following is the issue:
- Cameras can run in the background, without users directly viewing the screen (See this section of the React-Native Vision docs)
- When the camera is dismissed rather than closed, the camera process isn't terminated.
- Because the camera is still on, the scan() function returns with the "in progress" error
- The cordova scanner API only has two public methods: scan() and encode(). As such, there is no way to terminate the process without the user killing the app entirely.

Potential Solutions

As I understand this cordova plugin, there is nothing we can do manually terminate the camera process. I've spent a fair bit of time with this, and haven't found a workaround
Considering there may not be a workaround, and this project was dropped 6 years ago, the responsible choice is to find a new plugin.
- Like Jack mentioned, there may be a camera API that works off of React Native Web. I'm just now circling back to this task, and will spend a bit more time researching.

What's Next

After reading through all this, though, I'm not sure that this issue is a high priority anyway. My understanding is that the issue is only reproduced on iOS by 'swiping away', or sometimes by clicking "Cancel". It seems to me that simply silencing the error while allowing the scanner to close would be acceptable.

Because we only have the scan() and encode() methods, I don't believe we have a way of silencing this error. Since the scan() method is what fails, there doesn't seem to be a way to bypass the failure and scan despite the already-open camera.

This is where I'm left with a couple questions:

Is this a "Showstopper" bug? It's frustrating when it occurs, and requiring a restart is a pretty critical failure. That being said, it occurs relatively infrequently, and is 100% recoverable upon a restart.
- If it isn't a showstopper: should we put this on the backburner, and replace the plugin during a future rewrite? At a glance, fixing this bug may require a pretty substantial rewrite. From what I can tell, there are no contemporary React Native Web plugins that meet our QR-Code scanning needs.

@shankari , I'd appreciate your thoughts on this issue! With our current project on the horizon, I don't know how high-priority this bug is. I'll keep looking for quick fixes when I have down time, but wanted to put this on the table before things get too busy!

shankari commented 10 months ago

@the-bay-kay I would say that the (simple) fix is to fork the plugin and modify it to do what we need. That's the reason that all our code is open source (with the notable exception of push notifications, which require firebase on android). You should be able to add a new method to the plugin to terminate the camera and then call it from the javascript. In fact, I would suggest that you and @louisg1337 try to do that so that you understand, in greater detail, how plugins work.

Assuming the native camera API has a way to terminate the scan, this should not take very long and should give you a sense of how the native code and the javascript interact, in case we need it for "the big project"

the-bay-kay commented 9 months ago

Update

I've been working on my fork of the barcode scanner plugin, and have run into a few issues with building to a physical iPhone. Here's a few of the things I've tried, and some notes on how I may move forward

Building via XCode:

When working with the .xcworkspace file, there is an option to build to hardware instead of the local simulator.

Build feature for iOS

I'm logged into the internal testing account, so that the accounts are the same across devices (NREL laptop, test phone). When hitting "build", I run into the following certificate error:

Build Error

When attempting to view our account's certificates via Apple's profile (link), I get an error message that reads:

"Access Unavailable: You currently don't have access to this membership resource. Contact your team's Account Holder, or an Admin."

Because our test account is marked as a "personal development team", and Apple has restrictions on this type of development team (see: push notifications), we can't build directly to a device. I've tried a few things, including building with my personal AppleID, but haven't made any progress with this method.

Transfering the `.app`

When following the build instructions in e-mission-phone (link), I can successfully build a .app file. If I drag and drop this into a simulator, this runs fine! The issue, however, is transferring it to hardware.

I've found the documentation on transferring .app files to be scarce at best. Android has the Android Debug Bridge (link, but Apple does not have an equivalent "pipeline" between built app and hardware. This StackOverflow thread suggests we should be able to "drag-and-drop" the app via iTunes, provided we have the correct certificates. That being said, this thread is over a decade old, so I doubt the transfer is so simple. In fact, I spent some time exploring Apple Music's (iTunes successor) iPhone interface, and have found neither an app section nor a "distribution" section. Similar threads have suggested using iTunes as a method to transfer .app files, but I haven't found any contemporary guides.

What's Next

This guide form 2019 suggests I may be able to get a "short-term" development certificate on my personal AppleID. I'm concerned that the mismatch in accounts (test account on iPhone, personal on Xcode) will cause issues, but it's one of the few paths forward. Furthermore, I'm not entirely sure this fixes the issue; the article suggests this circumvents the paid developer certificate restriction, but I haven't seen this claim elsewhere.

This thread also suggests that an individual can get provisional certificates, but organization accounts need a full license. My current plan is to see if I can get the certificate on my personal account, and go from there.

If anyone has any experience building to iOS hardware, please reach out! I'm going to keep trying to set up the certificates on my personal account, but would be much happier working with the team's account if possible -- that way other members of the team wouldn't need to go through the same setup process when working on native builds :)

Edit: As could be expected, the certification process has changed from 2019 -- it seems that apple only has one "Apple Development" certificate now, and then distinguishes between personal and professional teams. The "does not support push notifications" error seems to persist when working with the personal account, I'll keep this thread updated.

shankari commented 9 months ago

Just to clarify, our test account does not have an Apple developer account. The NREL developer account is not managed by me, but by NREL legal. So I would try to run with your personal account. If that doesn't work, we can try to get you and louis on the NREL developer account, but it took me ~ 1 month last time, and I wasn't an intern.

the-bay-kay commented 9 months ago

Building for iOS Update

I've spent a good two and a half days wrestling with the iOS build process, and ran into several issues. I'm still in the process of getting the app to run on hardware (I finally got a new set of error messages, progress!), but I wanted to take a few minutes to outline my build process thus far. This is by no means comprehensive (thought I may update the docs once the dust has settled...), but is meant to give some troubleshooting advice!

Building to the Simulator:

While the build guide in the e-mission-phone repo (link) is generally accurate, I found there were a few additional steps I had to take. Below are some of my notes on this subject

Run the setup and activate scripts for native-ios building, as outlined in the link above
- This is where I ran into my first error; when running the build script via the terminal, I ran into several "Exit Code 65" error.s The fix for this is explained below: once fixed, you should be able to build the emission.app as usual, and then "drag-and-drop" this into an iPhone simulator.
From the e-mission-phone/ directory, navigate to platforms/ios/. Open the emission.xcworkspace file.
The iOS target deployment version needs to be updated in several places. First, go to the emission section on the left hand side of the screen, and select the target on the second from left window. Navigate to the "minimum deployments", and select an iOS version above iOS 12. Navigate to the pods section of the app, and repeat this for every utility listed under the target drop down. Screenshot (1) shows this process!
- This should fix the "Exit Code 65" issue. I can pull up more exact logs for reference later, but I've found 9/10 times, the deployment version is the cause of the issue.
- I suspect these settings are configurable somehow; I've been manually doing this when I re-clone the repository, but I'm certain there's a config file that could be adjusted somewhere...
Next, select which simulator you want to build to from the dropdown menue on the top of XCode. Hit the "play" button, and you will hopefully be able to build to the simulator without errors! (Again, a screenshot has been provided below).
- If, like me, you're building on an M1 machine: beware! I had (and have had) several unusual issues with iOS simulators running versions below iOS 16.0. I'll try to pull up the old threads where I discussed this issue, but TL;DR: M1 chips struggle to emulate OpenPATH on older operating systems, and the simulators run into springboard errors. Build to at least iOS 16!

Profile Problems

After stepping away from this project for a few days, I came back to a completely new problem. When attempting to sign for an app, I ran into two issues: one with the app Bundle Identifier, and one with my Apple Developer Account's Provisioning Profile. With these errors, I was unable to build to hardware or the simulator.

Unfortunately, I forgot to take screenshots of the broken certificates. Below is the rough format of these errors:

Xcode couldn't find a provisioning profile matching [apple id] "Failed to register bundle identifier. The app identifier "edu.berkeley.eecs.emission" cannot be registered to your development team. Change your bundle identifier to a unique string to try again."

To find out more about the Apple Development profiles, I decided to check their website and my developer id. When navigating to the Certificates & Profiles section of the Apple Developer website to investigate further (link), I received a popup error, and was redirected to the customer support page (screenshot (4)). From everything I've read (link_one , link_two), this is a pretty common error... I've got a hunch that this is because I'm not using a paid developer account, so I haven't pursued help from Support.

After manually deleting and resetting several portions of XCode, wrestling with the certificate website, and completely re-cloning and setting up the repository, and even creating a brand new apple id, I finally found a fix to the problem here. Using Keychain Access, I cleared XCode's password / account cache. With this flush, I went back to settings, re-signed into my personal account, and the error completely disappeared. I'll be completely honest: I have no idea why this happened, but am glad to have found a fix.

A note on Certificates

When building to hardware, personal developer accounts are not allowed to use push notifications. As such, you must delete this certificate before building. This certificate can be found under the "Signatures and Certificates" tab (Screenshot 3). I don't have a screenshot of the certificate itself, but I'll add one if I re-clone the repository.

What's Next:

With my current build, I've run into a few more issues. At the time of writing, I've just started investigating these; since I don't know if they're of any significance, I won't detail them yet. I'll keep this thread updated as I continue!

XCode Screenshots

(1) XCode Deployment Change

XCode Screenshot

(2) XCode run-on-simulator

XCode Screenshot

(3) App Signing Section

Popup Error

the-bay-kay commented 9 months ago

Woohoo, successfully build to hardware!! I'll update this thread further once I've build & tested the OPCode changes.

One final note on the build process: When you first successfully build to hardware, you will most likely receive an error along the lines of:

Untrusted Developer: Your device management settings does not allow using an app from [DeveloperID]

To fix this error: unlock your iPhone, open the settings app, and navigate to "Settings > General > VPN & Device Management". You should be able to trust your developer account here, and launch the application as normal!

shankari commented 9 months ago

@the-bay-kay great job! Please document how to get this done so @louisg1337 can follow the same steps!

the-bay-kay commented 9 months ago

Got the OpCode fork to build too! Now that I know we can edit native code and test on hardware, I'll start compiling my notes into a writeup. I did have a few questions about how I should organize that info:

Should I write up the process for building to Android as well? I have some notes from when Louis and I went through that process, it shouldn't be too hard to add to our current instructions.
Where should this build guide be written? My thought was, I could edit/add a page to the docs, and then link to the guide somewhere in the e-mission-phone repository's readme!

louisg1337 commented 4 months ago

Scanner Issue

Thanks to all of @the-bay-kay's work above, I was able to fix the scanner. The issue, which Katie pointed out above, was that when you swiped away the screen the camera process was still active, meaning that when you would try to reopen the camera you would get the "camera in progress" error. I was able to fix that by calling the camera's close method whenever the view disappeared. The changes to the plugin can be found here.

Invalid Study Reference Issue

Another scanner related issue Katie and I discussed was the "Invalid Study Reference" error below that we would see from time to time on iOS.

Error Screenshot

![unnamed](https://github.com/e-mission/e-mission-docs/assets/79764532/e0e54900-3216-4987-8bfc-baafda76381b)

At first, me and Katie suspected that this error was happening because of the extra nrelopenpath://login_token?token= at the start of the op code. I tried to manually remove that from the op code, but I ended up getting more errors than before. I then realized I overlooked a logDebug statement that actually told us what checks we failed that causes the "Invalid Study Reference" error. Here is the logDebug statement...

DEBUG:QR code nrelopenpath://login_token?token=nrelop_open-access_default_Efo4Wni739V4b5o9UP2Udhyt2NXe74aN checks: 
      cancel, format, prefix, params, code: 
      true, true, false, true, nrelop_open-access_default_Efo4Wni739V4b5o9UP2Udhyt2NXe74aN

As can be seen, we are failing the prefix case, which is let hasPrefix = url.protocol == 'emission:';. The url does not have 'emission' in it, so it makes sense as to why we are failing our checks. I am just a bit confused now as to what the issue is. I'm not sure if its an issue with the op code url not having emission in the protocol, or if the prefix check isn't meant to be there anymore, or if there is another bug deeper down somewhere. If anyone has any idea on what the issue may be I would appreciate the help!

shankari commented 4 months ago

As can be seen, we are failing the prefix case, which is let hasPrefix = url.protocol == 'emission:';. The url does not have 'emission' in it, so it makes sense as to why we are failing our checks. I am just a bit confused now as to what the issue is. I'm not sure if its an issue with the op code url not having emission in the protocol, or if the prefix check isn't meant to be there anymore, or if there is another bug deeper down somewhere. If anyone has any idea on what the issue may be I would appreciate the help!

There are two versions of the app; the one in the public repo has the package name edu.berkeley.eecs.emission, which is because OpenPATH came out of my PhD thesis, which was called e-mission. This is the core app that multiple organizations can and do use.

We then have an internal repo for OpenPATH (which both you and Katie should have access to) where we have changed the package name to openpath, which is the version that we publish into the stores under our organization.

The public repo listens to emission:// the internal repo listens to nrelopenpath:// (you can verify this by checking the internal repo). This ensures that the UNSW app (for example) will not join openpath studies by mistake.

You need to change the QR code to start with emission:// or hack (but not commit) a change to verify that the URL starts with nrelopenpath://

louisg1337 commented 4 months ago

The public repo listens to emission:// the internal repo listens to nrelopenpath:// (you can verify this by checking the internal repo).

Oh, that makes a lot of sense, thank you for the clarification, I didn't know about that. Now that I am filled in on the above, I used an online QR code generator and plugged in the URL with emission instead of nrelopenpath...

emission://login_token?token=nrelop_open-access_default_83Fy2WNIZveHQCdkwNoAVDIWfNK1pAwH

Once I did that I was able to successfully scan and login to the app. Now that both issues are resolved, I think these changes are finished. The PR which updates the package.cordovabuild.json to point to my scanner plugin can be found here.

shankari commented 4 months ago

@louisg1337 I am happy to merge this for now, but we should avoid having fixes in private organizations. I have created a repo in the e-mission org - https://github.com/e-mission/phonegap-plugin-barcodescanner

please submit your fix as a PR to it and change the reference in package.cordovabuild.json.

e-mission / e-mission-docs