Need remote troubleshooting of sms gateway phones

medic / cht-core

The CHT Core Framework makes it faster to build responsive, offline-first digital health apps that equip health workers to provide better care in their communities. It is a central resource of the Community Health Toolkit.

https://communityhealthtoolkit.org

GNU Affero General Public License v3.0

438 stars 208 forks source link

Need remote troubleshooting of sms gateway phones #3284

Open estellecomment opened 7 years ago

estellecomment commented 7 years ago

Currently you need a person to physically look at gateway phones to diagnose them. We're at a training on site away from the office, and can't troubleshoot or restart the gateway.

Remote access would be nice. Or just remote rebooting.

We can probably leave gateways plugged into a laptop, and then ssh + adb when needed. (another option is grounding @jaymedic at the office and remote-controlling him by phone, which is the current workaround)

estellecomment commented 7 years ago

Exporting logs remotely would also be awesome

estellecomment commented 7 years ago

adb logcat | grep --line-buffered Medic gives you log.

Note that in the adb log, you don't have the status of messages, which is the most useful part. So that would have to be added to logs.

(For the UI, the message statuses are fetched from DB, and getting that from logcat is complicated/impossible.)

alxndrsn commented 7 years ago

adb logcat | grep --line-buffered Medic gives you log.

I'd recommend this as being more correct:

adb logcat MedicGateway:V AndroidRuntime:E '*:S'

alxndrsn commented 7 years ago

We can probably leave gateways plugged into a laptop, and then ssh + adb when needed.

@bishwas-medic interested in your thoughts on this, especially when trying to make the gateways more independent rather than less.

alxndrsn commented 7 years ago

Exporting logs remotely would also be awesome

Any thoughts on how this would work? Where would you export them to? Perhaps possible for an android phone to read the logs and send them as an email?

alxndrsn commented 7 years ago

We're at a training on site away from the office, and can't troubleshoot or restart the gateway.

It may be more practical to take the gateway phone with you to the training.

bishwasBhatta commented 7 years ago

Just clarifying/adding few things here.

This happened on a Sunday (which can possibly happen again as Sunday is a work day and not a holiday in Nepal's official calendar), so I was not at the office to directly check and get logs from the gateway phone and was supporting from my home. On normal weekdays, this isn't a problem.
Taking the gateway phone to the training is not advised as the training area may not have proper internet connection.
We are working on to directly push/pull SMS with NTC's SMPP server with our own gateway server. This should eliminate the need to maintain gateway phones and make it lot easier to access logs and remote troubleshooting.

estellecomment commented 7 years ago

Exporting logs remotely would also be awesome Any thoughts on how this would work?

If you have physical access to the phone, then either a button that emails logs yes, or use adb to get logs through USB. If you have remote log access through ssh, you can save them to wherever, so no need for extra features.

This would have come in useful because when you reboot the gateway phone you apparently lose the logs. So if you reboot because of an issue, you lose the log that could have helped fix it.

estellecomment commented 7 years ago

It may be more practical to take the gateway phone with you to the training.

If a project runs into a problem and the person troubleshooting the problem isn't physically with the phone, you're stuck. Like if Bishwas is at another training/meeting, or working from home, or it's after hours/holiday...

nice-snek commented 7 years ago

hi frand @browndav, please triage before the end of this sprint.

garethbowen commented 7 years ago

There are multiple issues here...

Remote control sounds really exploitable and hard to pull off.
Logs should be persistent, and rotated if necessary. Can we fix this?
An easy way to get access to the logs sounds like a good idea, but a button would still require you to be on site. Could we set the app up to push logs to a server somewhere that can then be accessed remotely (eg: the medic-os server)? Does the gateway device usually have stable abundant cheap internet access? This would also solve (2).

@alxndrsn @estellecomment Any thoughts? I'm leaning towards 3.

estellecomment commented 7 years ago

Details from conversation with @bishwas-medic

~~phones are often~~ some phones are at the partner's office, so we currently train someone to run adb on a laptop and pull logs for us. Partners are not complaining about the situation, but for tech leads it can get really painful. So it's a low priority externally, but pretty important internally.
logs currently contain PHI.
remote-rebooting of the phone is also an interesting feature, but lower priority, since asking someone at the partner office to reboot the phone is fairly easy (low-tech task).

3 sounds sensible. Android 6 (which we're already supporting) has a data backup feature, which encrypts and pushes to Google Drive.

garethbowen commented 7 years ago

Ok sounds good. Store the logs somewhere in the cloud, secure and backed up, and accessible to tech leads.

medic-bot commented 6 years ago

Hi @sglangevin,

This ticket has not been touched in 90 days. Is it still relevant?

Please also ensure this ticket has a Priority, Status and Type label.(See triaging old issues for more detail)

sglangevin commented 6 years ago

We've had a couple of requests to do a deeper dive into gateway reliability which has improved but isn't where we need it to be yet. I'm thinking we can include this one as part of that work.