How to decipher touch data

[Axemasta]

Hi, thanks for the comprehensive readme, it was very helpful.

I'm currently researching for a project where our app will read all the device data whilst a user performs an action (typing a sentence 3 times). We need to know everything the user does, including where they tap, this looks very promising because I've been unable to get complete touch data for the whole screen... Without going down the rooting path!

Do you know how I can transform the data from the driver into something more useable? Currently the data looks like:

Timestamp	Type	Code	Value
[ 36355.147068]	0000	0005	08c36ca7

Whereas I would like to have the data look something more like:

Timestamp (Unix)	Action	X Coord	Y Coord
1549970121	TOUCH_DOWN	153.2	452.5

Would that be possible with the current data available (I know action is a reach).

Feel free to close the issue once you've responded, I know it's not an issue with your code perse, I just hoped you'd know more on this topic!

Thanks

[Cartucho]

This is a great GitHub issue (here issues include questions as well)!

So one way towards that goal is to use the -l option displays textual labels.

So, for example, you can replace step 3 by: adb shell getevent -lt /dev/input/event7 > recorded_touch_events.txt

Could you give it a try?

[Axemasta]

Wow thanks for the fast response!

I've given it a go and we're definitely getting closer! I got the following results

Timestamp	Type	Code	Value
[,46112.124651]	EV_ABS	ABS_MT_TRACKING_ID	000005b0
[,46112.124651]	EV_ABS	ABS_MT_POSITION_X	0000010f
[,46112.124651]	EV_ABS	ABS_MT_POSITION_Y	00000479
[,46112.124651]	EV_ABS	ABS_MT_PRESSURE	0000005f
[,46112.124651]	EV_ABS	ABS_MT_TOUCH_MAJOR	0000005f
[,46112.124651]	EV_SYN	SYN_REPORT	00000000
[,46112.173948]	EV_SYN	0004	0000b420
[,46112.173948]	EV_SYN	0005	0a5db877

I think all I need now is to be able to convert the value field from hexadecimal into something more useful to analyse. Any ideas? is there any documentation you'd recommend?

Thanks

[Cartucho]

Now relatively to the timestamp in their documentation they say:

getevent timestamps use the format $SECONDS.$MICROSECONDS in the CLOCK_MONOTONIC timebase.

CLOCK_REALTIME vs. CLOCK_MONOTONIC

[Cartucho]

I think all I need now is to be able to convert the value field from hexadecimal into something more useful to analyse.

Exactly!

I would now use the following command to get the min and max values of each type of event: adb shell getevent -lp /dev/input/event7

  name:     "Melfas MMSxxx Touchscreen"
  events:
    ABS (0003): ABS_MT_SLOT           : value 0, min 0, max 9, fuzz 0, flat 0, resolution 0
                ABS_MT_TOUCH_MAJOR    : value 0, min 0, max 30, fuzz 0, flat 0, resolution 0
                ABS_MT_POSITION_X     : value 0, min 0, max 720, fuzz 0, flat 0, resolution 0
                ABS_MT_POSITION_Y     : value 0, min 0, max 1280, fuzz 0, flat 0, resolution 0
                ABS_MT_TRACKING_ID    : value 0, min 0, max 65535, fuzz 0, flat 0, resolution 0
                ABS_MT_PRESSURE       : value 0, min 0, max 255, fuzz 0, flat 0, resolution 0
  input props:
    INPUT_PROP_DIRECT

And I think it should be possible to convert the hexadecimal to int using those values. Could you give it a try?

[Axemasta]

So if I run that command I see:

name:     "synaptics_dsx_i2c"
  events:
    KEY (0001): KEY_POWER            
    ABS (0003): ABS_MT_SLOT           : value 0, min 0, max 9, fuzz 0, flat 0, resolution 0
                ABS_MT_TOUCH_MAJOR    : value 0, min 0, max 255, fuzz 0, flat 0, resolution 0
                ABS_MT_POSITION_X     : value 0, min 0, max 1079, fuzz 0, flat 0, resolution 0
                ABS_MT_POSITION_Y     : value 0, min 0, max 1919, fuzz 0, flat 0, resolution 0
                ABS_MT_TRACKING_ID    : value 0, min 0, max 65535, fuzz 0, flat 0, resolution 0
                ABS_MT_PRESSURE       : value 0, min 0, max 255, fuzz 0, flat 0, resolution 0
  input props:
    INPUT_PROP_DIRECT

If I convert the hex values to integers (duh i should have tried this before) I now get:

Timestamp	Type	Code	Hex Value	Int Value
[,46112.124651]	EV_ABS	ABS_MT_TRACKING_ID	000005b0	1456
[,46112.124651]	EV_ABS	ABS_MT_POSITION_X	0000010f	271
[,46112.124651]	EV_ABS	ABS_MT_POSITION_Y	00000479	1145
[,46112.124651]	EV_ABS	ABS_MT_PRESSURE	0000005f	95
[,46112.124651]	EV_ABS	ABS_MT_TOUCH_MAJOR	0000005f	95
[,46112.124651]	EV_SYN	SYN_REPORT	00000000	0
[,46112.173948]	EV_SYN	0004	0000b420	46112
[,46112.173948]	EV_SYN	0005	0a5db877	173914231

I'm currently working on a script to parse the output of adb with a regex and convert the system time & hex to int and then output everything into a csv. I'll get back to you with how I get on!

[Cartucho]

Oh right! It makes sense.

Yeah, I think we should add that script to this repo so that people in the future can use it if they face the same situation!

[Axemasta]

I agree, once i've built it I'll send you a message and we can get it added!

[Axemasta]

I've forked the repo and I've added a .netcore console app to parse the file and output it to a csv. I just used .net because it's what I'm comfortable with. I can work on porting the app to other languages and just have it run as 1 script instead of an ""app"". What languages would you prefer to see it in? (since we're basically talking linux here, maybe python? .netcore will run on anything so I guess thats also cool).

Also I didn't parse the timestamp. From what I read the time is relative to an arbitrary time in the OS and doesn't correlate to an absolute time (ref). I'll need to do more to get this information into something more absolute. Off the top of my head I would get the time that it is relative to and then I can calculate the absolute time.

At some point I'll also write up a readme explaining how to extract & parse the data so it can be utilised for analysis etc.

How do you want to go about merging my branch back into your repo? it's yours so I'll let you decide what you want to do with it, just let me know if I've named/done anything wrong and I can get it fixed :)

Here the link to my fork!

[Cartucho]

I had never heard about .netcore before, but if it works, than that's ok with me :+1:

So a way to get the "human time" would be to run the command adb shell getevent -lt /dev/input/event7 as a subprocess in python and then parse the data / convert it directly. In the begining of the code you would store the starting time and then you would use that as a referance.

[Axemasta]

Ok cool, I will look into turning the whole thing into a single python script that manages the whole thing end to end.

[Cartucho]

Hello!

I managed to do it via a Python script, you can easily use it to export to .csv format, have a look!

[Axemasta]

Wow thanks for the python script! I've had a look at it today and I've not got it writing to a csv.

I've altered the script so that the touchscreen can be passed in as an argument. For whatever reason find_touchscreen_name.sh wasn't working on my mac. I found changing ref adb shell getevent -lp "$line to adb exec-out getevent -lp "$line worked but then it found the wrong touchscreen, so it was easier just to tell the script which device to look for.

You also tell it where you want your csv to go

It's tested and working on my end, there's no real error handling but i'm not too concerned about that. Below is my script, do you want it adding to this repo? maybe I can submit a PR and add it as a new file called "human_readable_data_output.py".

Let me know your thoughts!

csv output script:

import subprocess
import datetime
import sys
import csv

#Input your touchscreen device name to the file
#cmdline usage: python human_readable_data.py /dev/input/event$

touchscreen = sys.argv[1]

headers = ["Timestamp", "Type", "Code", "Value"]

with open('touch_output.csv', 'w') as writeFile:
    writer = csv.writer(writeFile)
    writer.writerow(headers)
    writeFile.close()

p = subprocess.Popen(['adb', 'exec-out', 'getevent', '-lt', touchscreen], stdout=subprocess.PIPE)

while True:
    output = p.stdout.readline()
    if output == '' and p.poll() is not None:
        break
    if output:
        time, event_data = output.split("]", 1)
        event_type, event_code, event_value = event_data.split()
        try:
            # convert hexadecimal to int
             event_value = int(event_value, 16)
        except ValueError:
            pass

        with open('touch_output.csv', 'a') as writeFile:
            writer = csv.writer(writeFile)
            writer.writerow([datetime.datetime.now(), event_type, event_code, event_value])
            writeFile.close()

        print("[ {}] {} {} {}".format(datetime.datetime.now(), event_type, event_code, event_value))

[Cartucho]

Please submit a PR for the same file human_readable_data.py.

[Axemasta]

Ok will do, I am also updating the readme to document all of this. I will submit both changes in the same PR once i've finished!