ARM-software / workload-automation

A framework for automating workload execution and measurement collection on ARM devices.
Apache License 2.0
151 stars 120 forks source link

Reliable power measurement with ADB connected #1178

Open jneanne opened 3 years ago

jneanne commented 3 years ago

I'm using workload automation to perform power measurements. At this point my device run an android then ADB is required.

I used USB cable but connecting the USB is powering the device from a different source then my power measurement from the main supply is then not reliable.

On top of this USB cable connected is avoiding system to reach suspend which can be a problem for me.

Ethernet might be a better option but in that case I won't be able to reach flight mode for some use cases...

Ideally I would need the framework to rely only on the uart serial while the workload is running so that it don't put any constraint on the system (then I can cut USB 5V or disable the ethernet), then recover ADB at the end of workload...

Do you have some recommendation on how to proceed to have non intrusive power measurements? Could you please tell me what is possible not possible in this area?

setrofim commented 3 years ago

Ideally I would need the framework to rely only on the uart serial while the workload is running so that it don't put any constraint on the system (then I can cut USB 5V or disable the ethernet), then recover ADB at the end of workload...

I don't believe adb supports UART, so that's not an option. Using UART directly would be very difficult. We currently have nothing to support this, and implementing something like that would be complicated (especially if you want to swap between UART and adb during a run). Additionally, some WA functionality relies on being able to have multiple parallel communication channels to the device (e.g. workload collecting output on one, while using the other to signalling workload to terminate), which would not be possible with raw UART (you'd need to implement some sort of multiplexing on top of it, and would likely require support on the device side as well).

Ethernet might be a better option but in that case I won't be able to reach flight mode for some use cases...

I assume you mean WiFi rather than Ethernet? If so, then that is the typical solution for measuring power on Android devices (ether adb-over-wifi, or by deploying an ssh server on the device); however, as you say, that will preclude you from running workloads in flight mode. What is the reason for that requirement? If the concern is about some of the workloads "phoning home", or the device otherwise communicating with the outside world, you could set up a dedicated network not connected to the internet for those use cases.

If you do actually mean Ethernet (via an adapter), then I believe that ought to work even in flight mode (thought I haven't personally tried this)?

jneanne commented 3 years ago

Thanks for the reactive support.

The reason for this requirement is the first sanity check when starting power optimization on a system is in our methodology is checking where we stand in idle and suspend with everything else stopped so that we have the floor power consumption and can add activity on top of it then check that it matches models/refs...

I already have my home-brewed test framework that relies on uart only and can achieve this. But I should admit that for all the other capabilities it's much less capable that WA...

In a perfect world I would have used WA only and stop maintaining my "framework".

I understand that WA has not been designed for that and would be difficult to adapt.

I'll then keep both my own framework for low power use cases and WA for everything else. Let me know if this option is revisited one day (I understand it's not a minor change).

Here is a pointer to my "solution" for reference (please be kind, I'm not competing in the same category): https://gitlab.com/baylibre-public/thermoregulpm/tpmp_ctrl/-/tree/tpmpv2

khilman commented 3 years ago

Does WA support the notion of "background" test cases? Would it be possible to do something like

This would allow the USB (and/or networking) to be disabled during the test so low-power test cases could be measured.

setrofim commented 3 years ago

WA's execution assumes a constant connection to the device (apart from during specific operations such as rebooting the device). However, once the job execution starts, what is actually needed depends largely on the workload and the enabled instrumentation.

If the above is all you want, and one of your instrumentation or workloads rely on a connection during workload execution, you could just pull the usb cable from the device, and then reconnect it at the end. As long as the instruments/workloads don't try to talk to the device during that time, WA should be none the wiser (in case of ssh, it will detect the broken connection next time it tries to talk to the device, but it will just attempt to reconnect; for adb-over-usb, it shouldn't notice at all).

You can create a instrument similar the the delay instrument to block on user input in order to give you time to reconnect. (And if you have some mechanism for automatically pulling and reconnecting the usb cable, you can write an instrument to trigger that). You can set priories on the callback for that instrument in such a way that it does its thing at the correct time relative to other instrumentation (very_fast priority will ensure that it will run as close to the workload run as possible --i.e. last before the workload runs, and first afterwards).

khilman commented 3 years ago

Excellent, thank you for your suggestions!

marcbonnici commented 3 years ago

As long as the instruments/workloads don't try to talk to the device during that time, WA should be none the wiser (in case of ssh, it will detect the broken connection next time it tries to talk to the device, but it will just attempt to reconnect; for adb-over-usb, it shouldn't notice at all).

Apologies if I've misunderstood the usecase, however following on from this, if @jneanne is looking for a similar setup and has a seperate physical serial connection to the board and wants to use it just to perform the kick off / know when the connection has been reestablished (which is easier to automated if you can connect over a network), you might be able to create and utlise a seperate serial connection [1] and use that to to send simple commands just for that purpose, similar how we use this to obtain an ip address of a specific device before the main connection has been established [2].

[1] https://github.com/ARM-software/devlib/blob/master/devlib/utils/serial_port.py#L96 [2] https://github.com/ARM-software/devlib/blob/301d43d1404b3378ed96bab0bb071a6e01142ab9/devlib/platform/arm.py#L102

jneanne commented 3 years ago

Thanks a lot! We'll give it a try and migrate everything on WA if possible