Barcode scanning using device camera

[MaxDiz]

Is your feature request related to a problem? Please describe. As a nurse/CHW, I do home visits to follow-up with patients who are in treatment for disease management, but did not show for clinic appointments. Patients are often taking multiple medications (or using other health products), which are hard to keep track of. Having an accurate list of these products is critical to creating and managing up-to-date health record for patients. The process for collecting this information is currently manual, making it difficult to capture it in a timely and accurate way.

Describe the solution you'd like As a nurse, I would like the ability to scan barcodes for patient IDs or barcodes on medications (or other health products) to quickly and accurately acquire patient health information.

Describe alternatives you've considered

• manually inputting information
• taking photographs of products

Additional context bar code scanner is supported by XLSForm and ODK Collect

Worth noting that ZXing is in maintenance mode, and ODK Collect had to build in the scanner as opposed to an intent since 1.7. There is more investigation needed to know if this is a relatively simple task (handle the XLSForm type and make/handle a new intent) or a big lift.

[MaxDiz]

@isaacholeman please add any additional context specific to the project (or correct anything I misunderstood)

[isaacholeman]

Looks great @MaxDiz. I added that it's also a requirement to be able to scan barcodes to get patient ID. Thanks!

[garethbowen]

Also investigate supporting QR codes.

[MaxDiz]

Relevant CHT Forum conversation

[latin-panda]

Update:

i-tech-malawi project is also interested in scanning a barcode to save the patient's ID, this barcode is generated by the EMR system. They also see very valuable being able to scan a barcode to search for a patient. From this we can highlight two requirements:

1 - Registration of a new patient and getting the patient's ID: Currently achievable by using the Android App Launcher where a form can be configured to open an android app that can scan barcodes, the information in the barcode will be saved in this form.

2- Searching for a patient in contact tab by scanning the barcode: This one needs implementation in CHT-Core to open an android app to scan the barcode and search the patient.

CC: @garethbowen @craig-landry for prioritisation. @mmureithi @katanu @antonykhaemba please add any relevant information. Useful links: feature request doc, forum post

[latin-panda]

@michaelkohn @lorerod @n-orlowski FYI

Nb. This will be delivered in 4 different PRs:

• Search by barcode [this PR]
• Enketo widget to read barcode [To be developed]
• Autoselect if search by barcode returns 1 result [To be developed]
• Automation test coverage [To be developed]

This work uses the native Barcode Detector API from the browser by using a hidden input type file to natively open the camera in Android devices. This avoids custom code in the CHT Android and allows more accessibility since the feature can be used in CHT Android, PWA, and Android browsers.

It works the following way:

• The user clicks on the barcode scanner icon
• Prompt is displayed by the OS to take a picture or pic from the gallery
• User takes a picture
• CHT loads the image and then uses the Barcode Detector API to read the barcode.
• CHT enters that code into the search input field and triggers the search.

More considerations:

• It works in Android Chrome, PWA, and CHT Android app.
• This feature is enabled by using can_use_barcode_scanner permission
• This feature isn't supported on desktops and in browsers older than Chrome/Wedview v90.
• When clicking on the barcode icon to scan, the OS will open an option to take a picture or pick from the gallery. This is fully managed by the OS, and we can't limit these options (like preventing from selecting from the gallery)
• The user needs to grant permission to use the camera, sometimes it's already granted, but they can remove it. We can't force it, and we can't know if they have removed the camera access since we are using the native browser APIs.
• Some browsers might not support all the barcode types. Supported barcodes are printed in the console from the browser's developer tools.
• The DB Admin user doesn't have access to this feature. Otherwise, they will always have access to this.

Telemetry:

• search_by_barcode:open -> When a user clicks on the barcode scanner icon
• search_by_barcode:not_supported -> When the feature isn't supported because the device is a desktop or the Barcode Detector API isn't available in the browser.
• search_by_barcode:scan -> When the user has taken the picture.
• search_by_barcode:trigger_search -> When Barcode Detector gets the code from the image and triggers the search.
• search_by_barcode:barcode_no_detected -> When Barcode Detector couldn't detect the code from the image.
• search_by_barcode:failure -> Any system failure when the Barcode Detector tries to get the code from the image.

Snackbar messages:

• On error when the feature is enabled and the picture is loaded but cannot read the barcode: Failed to read the barcode. Retry

Videos

• Happy path: https://github.com/medic/cht-core/assets/66472237/17bfa6a9-f1ae-4632-b213-83d80a329899
• Errors: When a picture is bad quality (distorted) and it can’t detect the barcode: see snackbar message: https://github.com/medic/cht-core/assets/66472237/8354f530-b82f-4bff-9f53-4ac276dc8d27
• Error: Barcode Detector API no supported / Desktop machine (see snack bar and absence of icon to scan): https://github.com/medic/cht-core/assets/66472237/e9c3f442-7070-4c4f-ae6a-2e94afd25bfc

[latin-panda]

Read the previous comments for more details.

Knowns

• It can take a bit to open the camera when clicking on the barcode icon. This is dependent on the Android OS and phone specs.
  • We are using an input element type file, Android handles the click event and opens the options to take a picture or pick from the gallery, which Android might take some milliseconds in old phones with low specs.
• After taking the picture, extracting the value from the barcode can take milliseconds to 1 or 2 seconds. This also depends on the phone's specs.
  • After taking the picture, it needs to upload the picture into the app (which takes most of the time); then, it will process the image and extract the value from the barcode.
  • We might add a loading indicator somewhere, but that's yet to be designed and developed in the next phase - cc: @n-orlowski
• This feature is only supported in Chrome and Webview for Android.

Troubleshooting

• It doesn't show the option to take a picture using the camera
  • Ensure the user has granted permission to use the camera in the app (Chrome, PWA, or CHT Android).
• It might not scan the barcode if the picture is bad quality, meaning the image is blurry, obstructed, incomplete, has reflection, or too dark.
  • Try retaking the picture; ensure the image is clear.
• Other situations where it might not scan the barcode are when the barcode is damaged, there are more barcodes (or QR codes) in the same image, or the picture is taken from a barcode on a monitor/digital screen.
  • Try printing or renewing the barcode on paper.
  • Ensure there is only one barcode in the picture.
  • Avoid scanning a barcode that is on a monitor/digital screen.
• The button to scan the barcode isn't available
  • Ensure you have set the can_use_barcode_scanner permission to the users.
  • Ensure the user is using Android OS.
  • Update Chrome Browser to the latest.
  • Update Google Play services - this app is usually available on all Android phones; here is the Play Store link.
  • The browser uses the Android OS to detect different barcode types; these types are packed in Google Play services
  • Still not scanning after retry
    • Try retaking the picture; ensure the image is clear. It should not be blurry, obstructed, incomplete, have reflection, or be too dark.
    • Update Google Play services - this app is usually available on all Android phones; here is the Play Store link.
    • The browser uses the Android OS to detect different barcode types; these types are packed in Google Play services
  • The device might not support the barcode type.

How it works technically

As mentioned in the previous comment, this approach uses a hidden input type, file to make Android open the camera and take the picture. This picture is uploaded into the app, and then we use the new native API called Barcode Detector API [experimental feature - it has full support in Chrome and Webview for Android, but it's not supported by Firefox for Android].

When the browser uses this API, it utilises the OS tooling to be able to interpret barcodes from image elements. There are many types of barcode, and the support for these type of barcodes are built into the OS (MacOS, ChromeOS, and Android OS).

The browsers are using Android's ML Kit (previously known as GMS Vision) to read and interpret barcodes, this is usually available in the phone with the Google Play services app (see release notes). This is one reason why it's recommended that users keep their Android OS (and services) and Chrome Browser updated. Worth mentioning for completeness that when it comes to MacOS, the browsers are using CIQRCodeFeature.

[garethbowen]

@latin-panda Thanks for the update! Did you have a look at this barcode-detector library? It looks well supported (12k weekly downloads, recent updates, etc) and works as a polyfill for the experimental API you're using so it may be easy to drop in. I haven't tested it or checked bundle size, etc but it might be worth a look.

This is one reason why it's recommended that users keep their Android OS (and services) and Chrome Browser updated.

We know that this doesn't usually happen (despite our recommendations). In fact, last time I was in Kenya, safaricom txted me and recommended disabling app auto-updates to save data. Is there an easy way to figure out what is available in the MLKit that's available in Google Play services that shipped with Android 5 (minimum supported android version)?

[latin-panda]

Thanks @garethbowen the barcode-detector polyfill looks nice, and I like that it follows the same specification as the native API. To make it work for offline users, my guess is that we need to make their C++ library available on the client side using the service worker. This is based on Zxing work, which is quite famous, and many have forked his work to make it available on different platforms. I'm going to give it a try as a polyfill when the native API isn't supported. I think this isn't blocking the Feature Release and can go in parallel.

These are other libraries I looked at before:

• Maintenance Mode Only:

  • Zxing - Uses Media Devices API that is experimental and not fully supported. It needs polyfills
  • html5-qrcode - No PRs will be merged.

• Abandoned:

  • ngx-barcode-scanner - last released two years ago - Angular 12.
  • Javascript-Barcode-Reader - last released two years ago - it doesn't rotate the image
  • QuaggaJS - last released 6 years ago - interesting article on how the image is processed
  • Barcode reader - last released 7 years ago

[latin-panda]

Is there an easy way to figure out what is available in the MLKit that's available in Google Play services that shipped with Android 5 (minimum supported android version)?

@garethbowen The release notes are published online since these are toolings that Android developers need:

• MLKit
• Google Play services

I haven't analyzed the release notes in detail, but a quick search on those pages shows mentions of the Barcode feature in Mobile Vision package of Google Play services v9.2 (June 2016)

[latin-panda]

Removing from milestone as it wasn't completed on that iteration