Unable to unpack ADB command... error('unpack requires a buffer of 24 bytes',)

javierxio commented 3 years ago

This error keeps occurring for me when I run an automated script. Its occurrence is random; the error sometimes happens after looping through my program 20 times or sometimes after looping 200 times. I have tried looking at adb_message.py to see if there is anything I can do to fix this but nothing I have tried so far has helped.

My last idea is to change encoding/decoding from utf-8 to latin-1, but this would be a change in several adb_shell files. I have searched the other issues and the solutions there did not help; mainly setting adb_device._maxdata = 2048.

log_dev=device.shell('logcat -v epoch -d')   
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 445, in shell
    return self._service(b'shell', command.encode('utf8'), transport_timeout_s, read_timeout_s, timeout_s, decode)
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 333, in _service
    return b''.join(self._streaming_command(service, command, adb_info)).decode('utf8')
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 997, in _streaming_command
    for data in self._read_until_close(adb_info):
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 932, in _read_until_close
    cmd, data = self._read_until([constants.CLSE, constants.WRTE], adb_info)
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 888, in _read_until
    cmd, remote_id2, local_id2, data = self._read(expected_cmds, adb_info)
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 825, in _read
    cmd, arg0, arg1, data_length, data_checksum = unpack(msg)
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_message.py", line 122, in unpack
    raise ValueError('Unable to unpack ADB command. (length={})'.format(len(message)), constants.MESSAGE_FORMAT, message, e)
ValueError: ('Unable to unpack ADB command. (length=18)', b'<6I', 
b'CLSE\xa2\x06\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00', error('unpack requires a buffer of 24 bytes',))

JeffLIrion commented 3 years ago

A log with the logging level set to debug might help.

Looking at what you provided, it's not clear why the packet is 6 bytes too small. For your purpose, a workaround could be to catch the exception, close the connection, and reconnect.

javierxio commented 3 years ago

@JeffLIrion The error is different each time. Here is another error which shows it is 9 bytes too small.

ValueError: ('Unable to unpack ADB command. (length=15)', b'<6I', b'CLSEC\x0c\x00\x00\x01\x00\x00\x00\x00\x00\x00', 
error('unpack requires a buffer of 24 bytes',))

Where exactly should I catch the exception? In my program or in adb_message.py?

JeffLIrion commented 3 years ago

Where exactly should I catch the exception? In my program or in adb_message.py?

You would catch it in your own code.

javierxio commented 3 years ago

@JeffLIrion I have turned on debugging and also I have caught the exception in my code:

Unknown command: 2952790016 = 'b'\x00\x00\x00\xc2\xb0'' (arg0 = 1135001268, arg1 = 3175437132, msg = 'b'\x00\x00\x00\xb0\xb4\xbe\xa6CLSE\xbd=\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00'')

Another: Unknown command: 3154116608 = 'b'\x00\x00\x00\xc2\xbc'' (arg0 = 1337633971, arg1 = 3042525515, msg = 'b'\x00\x00\x00\xbc\xb3\xac\xbaOKAY\xb5=\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00'')

    print(d.shell('uptime'))
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 445, in shell
    return self._service(b'shell', command.encode('utf8'), transport_timeout_s, read_timeout_s, timeout_s, decode)
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 333, in _service
    return b''.join(self._streaming_command(service, command, adb_info)).decode('utf8')
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 995, in _streaming_command
    self._open(b'%s:%s' % (service, command), adb_info)
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 778, in _open
    _, adb_info.remote_id, their_local_id, _ = self._read([constants.OKAY], adb_info)
  File "/usr/local/lib/python3.6/dist-packages/adb_shell/adb_device.py", line 829, in _read
    raise exceptions.InvalidCommandError("Unknown command: %d = '%s' (arg0 = %d, arg1 = %d, msg = '%s')" % (cmd, int_to_cmd(cmd), arg0, arg1, msg))
adb_shell.exceptions.InvalidCommandError: Unknown command: 3154116608 = 'b'\x00\x00\x00\xc2\xbc'' (arg0 = 1337633971, arg1 = 3210297675, msg = 'b'\x00\x00\x00\xbc\xb3\xac\xbaOKAY\xbf=\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00'')

JeffLIrion commented 3 years ago

@javierxio that's not a debugging log, that's just the exception traceback. Instructions for enabling debug level logging are in the issue template:

To enable debug logging in Python:

import logging

logging.basicConfig(level=logging.DEBUG)

In particular, you would see these logging statements that log all the data sent and received:

https://github.com/JeffLIrion/adb_shell/blob/f25ae912423a02c1bedfc8088c1d04c00334b3c0/adb_shell/adb_device.py#L822-L824

https://github.com/JeffLIrion/adb_shell/blob/f25ae912423a02c1bedfc8088c1d04c00334b3c0/adb_shell/adb_device.py#L839-L842

https://github.com/JeffLIrion/adb_shell/blob/f25ae912423a02c1bedfc8088c1d04c00334b3c0/adb_shell/adb_device.py#L960-L965

ggallo commented 3 years ago

Experiencing a similar issue. If I run the README example, I get the following. This is all the output that I get even when I enable DEBUG logging at the top of the file:

Traceback (most recent call last):
  File "/mnt/d/code/threes-ai/test.py", line 16, in <module>
    device1.connect(rsa_keys=[signer], auth_timeout_s=2)
  File "/mnt/d/code/adb_shell/adb_shell/adb_device.py", line 250, in connect
    cmd, arg0, arg1, banner = self._read([constants.AUTH, constants.CNXN], adb_info)
  File "/mnt/d/code/adb_shell/adb_shell/adb_device.py", line 827, in _read
    raise exceptions.InvalidCommandError("Unknown command: %d = '%s' (arg0 = %d, arg1 = %d, msg = '%s')" % (cmd, int_to_cmd(cmd), arg0, arg1, msg))
adb_shell.exceptions.InvalidCommandError: Unknown command: 1397511251 = 'b'STLS'' (arg0 = 16777216, arg1 = 0, msg = 'b'STLS\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xac\xab\xb3\xac''

JeffLIrion commented 3 years ago

@ggallo your exception is different: it upacks the response just fine, but it doesn't recognize the STLS command header. I don't know what to make of that...

javierxio commented 3 years ago

Jeff, I have also turned on debug logging. Where do we catch this, in logcat?

atti92 commented 3 years ago

Hello, we are getting the same issue now, on newly updated physical android devices. Reproduced with 0.2.3, 0.3.1.

The issue seems to be random, it goes well until 2Gigs, but i haven't debugged it yet, I'll look into this.

16%|#########3 | 2.11G/13.0G [07:18<28:26, 6.40MB/s]

[09:04:53] Traceback (most recent call last):
[09:04:53]  
[09:04:53]    File "\lib\site-packages\adb_shell\adb_message.py", line 120, in unpack
[09:04:53]      cmd, arg0, arg1, data_length, data_checksum, _ = struct.unpack(constants.MESSAGE_FORMAT, message)
[09:04:53]                                                       â”‚      â”‚      â”‚         â”‚               â”” b'CLSE\xdbH\x00\x00\x01\x00\x00'
[09:04:53]                                                       â”‚      â”‚      â”‚         â”” b'<6I'
[09:04:53]                                                       â”‚      â”‚      â”” <module 'adb_shell.constants' from '\\lib\\site-packages\\adb_...
[09:04:53]                                                       â”‚      â”” <built-in function unpack>
[09:04:53]                                                       â”” <module 'struct' from '\\lib\\struct.py'>
[09:04:53]  
[09:04:53]  struct.error: unpack requires a buffer of 24 bytes

 File "\lib\site-packages\censored.py", line 166, in __lsdir
[09:04:53]      iterator = self._shell(rf"find {target} -print0 | xargs -0 stat -c '%f %s %Y %n'", hide_stderr=True)
[09:04:53]                 â”‚    â”” <function AdbDevice._shell at 0x00000246AA9E2048>
[09:04:53]  
[09:04:53]    File "\lib\site-packages\censored.py", line 45, in _shell
[09:04:53]      return self.adb_device.shell(str_command).splitlines()
[09:04:53]             â”‚    â”‚          â”‚     â”” "find /sdcard/CENSORED/path -print0 | xargs -0 stat -c...
[09:04:53]             â”‚    â”‚          â”” <function AdbDevice.shell at 0x00000246AA8F0048>
[09:04:53]             â”‚    â”” <adb_shell.adb_device.AdbDeviceTcp object at 0x00000246AA9ECA88>
[09:04:53]  
[09:04:53]    File "\lib\site-packages\adb_shell\adb_device.py", line 445, in shell
[09:04:53]      return self._service(b'shell', command.encode('utf8'), transport_timeout_s, read_timeout_s, timeout_s, decode)
[09:04:53]             â”‚    â”‚                  â”‚       â”‚               â”‚                    â”‚               â”‚          â”” True
[09:04:53]             â”‚    â”‚                  â”‚       â”‚               â”‚                    â”‚               â”” None
[09:04:53]             â”‚    â”‚                  â”‚       â”‚               â”‚                    â”” 10.0
[09:04:53]             â”‚    â”‚                  â”‚       â”‚               â”” None
[09:04:53]             â”‚    â”‚                  â”‚       â”” <method 'encode' of 'str' objects>
[09:04:53]             â”‚    â”‚                  â”” "find /sdcard/CENSORED/path -print0 | xargs -0 stat -c...
[09:04:53]             â”‚    â”” <function AdbDevice._service at 0x00000246AA8E9F78>
[09:04:53]             â”” <adb_shell.adb_device.AdbDeviceTcp object at 0x00000246AA9ECA88>
[09:04:53]  
[09:04:53]    File "\lib\site-packages\adb_shell\adb_device.py", line 333, in _service
[09:04:53]      return b''.join(self._streaming_command(service, command, adb_info)).decode('utf8')
[09:04:53]                      â”‚    â”‚                  â”‚        â”‚        â”” <adb_shell.hidden_helpers._AdbTransactionInfo object at 0x00000246AAC57DC8>
[09:04:53]                      â”‚    â”‚                  â”‚        â”” b"find /sdcard/CENSORED/path -print0 | xargs -0 stat -...
[09:04:53]                      â”‚    â”‚                  â”” b'shell'
[09:04:53]                      â”‚    â”” <function AdbDevice._streaming_command at 0x00000246AA8F08B8>
[09:04:53]                      â”” <adb_shell.adb_device.AdbDeviceTcp object at 0x00000246AA9ECA88>
[09:04:53]  
[09:04:53]    File "\lib\site-packages\adb_shell\adb_device.py", line 997, in _streaming_command
[09:04:53]      for data in self._read_until_close(adb_info):
[09:04:53]          â”‚       â”‚    â”‚                 â”” <adb_shell.hidden_helpers._AdbTransactionInfo object at 0x00000246AAC57DC8>
[09:04:53]          â”‚       â”‚    â”” <function AdbDevice._read_until_close at 0x00000246AA8F0798>
[09:04:53]          â”‚       â”” <adb_shell.adb_device.AdbDeviceTcp object at 0x00000246AA9ECA88>
[09:04:53]          â”” b'81b0 4413440 1576167330 /sdcard/CENSORED/path/A...
[09:04:53]  
[09:04:53]    File "\lib\site-packages\adb_shell\adb_device.py", line 932, in _read_until_close
[09:04:53]      cmd, data = self._read_until([constants.CLSE, constants.WRTE], adb_info)
[09:04:53]      â”‚           â”‚    â”‚            â”‚         â”‚     â”‚         â”‚      â”” <adb_shell.hidden_helpers._AdbTransactionInfo object at 0x00000246AAC57DC8>
[09:04:53]      â”‚           â”‚    â”‚            â”‚         â”‚     â”‚         â”” b'WRTE'
[09:04:53]      â”‚           â”‚    â”‚            â”‚         â”‚     â”” <module 'adb_shell.constants' from '\\lib\\site-packages\\adb_...
[09:04:53]      â”‚           â”‚    â”‚            â”‚         â”” b'CLSE'
[09:04:53]      â”‚           â”‚    â”‚            â”” <module 'adb_shell.constants' from '\\lib\\site-packages\\adb_...
[09:04:53]      â”‚           â”‚    â”” <function AdbDevice._read_until at 0x00000246AA8F0678>
[09:04:53]      â”‚           â”” <adb_shell.adb_device.AdbDeviceTcp object at 0x00000246AA9ECA88>
[09:04:53]      â”” b'WRTE'
[09:04:53]  
[09:04:53]    File "\lib\site-packages\adb_shell\adb_device.py", line 888, in _read_until
[09:04:53]      cmd, remote_id2, local_id2, data = self._read(expected_cmds, adb_info)
[09:04:53]                                         â”‚    â”‚     â”‚              â”” <adb_shell.hidden_helpers._AdbTransactionInfo object at 0x00000246AAC57DC8>
[09:04:53]                                         â”‚    â”‚     â”” [b'CLSE', b'WRTE']
[09:04:53]                                         â”‚    â”” <function AdbDevice._read at 0x00000246AA8F0708>
[09:04:53]                                         â”” <adb_shell.adb_device.AdbDeviceTcp object at 0x00000246AA9ECA88>
[09:04:53]  
[09:04:53]    File "\lib\site-packages\adb_shell\adb_device.py", line 825, in _read
[09:04:53]      cmd, arg0, arg1, data_length, data_checksum = unpack(msg)
[09:04:53]                                                    â”‚      â”” b'CLSE\xdbH\x00\x00\x01\x00\x00'
[09:04:53]                                                    â”” <function unpack at 0x00000246AA8E25E8>
[09:04:53]  
[09:04:53]    File "\lib\site-packages\adb_shell\adb_message.py", line 122, in unpack
[09:04:53]      raise ValueError('Unable to unpack ADB command. (length={})'.format(len(message)), constants.MESSAGE_FORMAT, message, e)
[09:04:53]                                                                              â”‚          â”‚         â”‚               â”” b'CLSE\xdbH\x00\x00\x01\x00\x00'
[09:04:53]                                                                              â”‚          â”‚         â”” b'<6I'
[09:04:53]                                                                              â”‚          â”” <module 'adb_shell.constants' from '\\lib\\site-packages\\adb_...
[09:04:53]                                                                              â”” b'CLSE\xdbH\x00\x00\x01\x00\x00'
[09:04:53]  
[09:04:53]  ValueError: ('Unable to unpack ADB command. (length=11)', b'<6I', b'CLSE\xdbH\x00\x00\x01\x00\x00', error('unpack requires a buffer of 24 bytes'))

atti92 commented 3 years ago

I think I found a problem in the AdbDevice._read method. I reads until receives a expected_cmds, but it discards every other command, and a problem raises when the discarded command has data. I'm currently testing a solution for this.

JeffLIrion commented 3 years ago

@atti92 can this be closed via https://github.com/JeffLIrion/adb_shell/pull/153?

atti92 commented 3 years ago

@atti92 can this be closed via #153?

I'm not sure, it is still possible that we missed something, we will see.

javierxio commented 3 years ago

@JeffLIrion @atti92 Thanks a lot for this commit. I will test it out. How do I get the latest code? Will "import adb_shell" automatically take the latest? @ggallo fyi.

atti92 commented 3 years ago

@javierxio install version 0.3.2

javierxio commented 3 years ago

@atti92 thank you. i will do that shortly: pip install adb-shell

javierxio commented 3 years ago

''''pip install adb-shell''' gives this result: Requirement already satisfied: adb-shell in /usr/local/lib/python3.6/dist-packages (0.3.1)

JeffLIrion commented 3 years ago

@javierxio do pip install -U adb-shell

atti92 commented 3 years ago

I think my problem is fixed by this. We were not getting the same issue since 0.3.2, pushing gigabytes of data while sending shell commands in-between seems to be working fine. I hope this fixes it for @javierxio too.

javierxio commented 3 years ago

@atti92 I have updated to 0.3.2 and I haven't seen the unable to unpack ADB error yet.

However, I still see an error as before in my previous post here: https://github.com/JeffLIrion/adb_shell/issues/152#issuecomment-822641601

This is the exception: Unknown command: 0 = 'b'\x00\x00\x00\x00'' (arg0 = 0, arg1 = 2952790016, msg = 'b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6CLSEo\x0b\x00\x00\x01'')

Might this be a unrelated error? I'll keep testing overnight and look for unable to unpack error. Thanks.

JeffLIrion commented 3 years ago

@javierxio i think my previous instructions for setting the log level to debug didn't work, I think what you want is:

import logging

logging.basicConfig(level=logging.DEBUG)

I edited my previous comment (https://github.com/JeffLIrion/adb_shell/issues/152#issuecomment-822951505).

Also, there was a bug in 0.3.2. Can you please upgrade to 0.3.3 and verify that you're using the latest release:

import adb_shell

assert adb_shell.__version__ == "0.3.3"

And an "Unknown command" error is different than a "ValueError: ('Unable to unpack ADB command...)" error, but it's possible that the root cause could be related.

javierxio commented 3 years ago

@JeffLIrion thanks. I've updated the version to 0.3.3 and i'll fix the logging command. I will test some more later tonight.

JeffLIrion commented 3 years ago

Any updates on this issue?

javierxio commented 3 years ago

@JeffLIrion I believe I am still seeing the issue. It happens mainly when running multiple devices at the same time. I will get a log next time.

javierxio commented 3 years ago

Yes, I still see this issue. Here is a log:

DEBUG:adb_shell.adb_device:bulk_write(24): b'OPEN\x01\x00\x00\x00\x00\x00\x00\x00\x17\x00\x00\x00\x84\x07\x00\x00\xb0\xaf\xba\xb1'
DEBUG:adb_shell.adb_device:bulk_write(23): b'shell:logcat -b all -c\x00'
DEBUG:adb_shell.adb_device:bulk_read(24): b'OKAY\xfc\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
DEBUG:adb_shell.adb_device:bulk_read(24): b'CLSE\xfc\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
DEBUG:adb_shell.adb_device:bulk_write(24): b'CLSE\x01\x00\x00\x00\xfc\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
DEBUG:adb_shell.adb_device:bulk_write(24): b'OPEN\x01\x00\x00\x00\x00\x00\x00\x00\x13\x00\x00\x00\xa7\x05\x00\x00\xb0\xaf\xba\xb1'
DEBUG:adb_shell.adb_device:bulk_write(19): b'shell:date +%s.%3N\x00'
DEBUG:adb_shell.adb_device:bulk_read(24): b'OKAY\xfd\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
DEBUG:adb_shell.adb_device:bulk_read(24): b'WRTE\xfd\x00\x00\x00\x01\x00\x00\x00\x0f\x00\x00\x00\xdd\x02\x00\x00\xa8\xad\xab\xba'
DEBUG:adb_shell.adb_device:bulk_read(15): b'1625162759.117\n'
DEBUG:adb_shell.adb_device:bulk_write(24): b'OKAY\x01\x00\x00\x00\xfd\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
DEBUG:adb_shell.adb_device:bulk_read(24): b'CLSE\xfd\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
DEBUG:adb_shell.adb_device:bulk_write(24): b'CLSE\x01\x00\x00\x00\xfd\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
Error: Connection refused
DEBUG:adb_shell.adb_device:bulk_write(24): b'OPEN\x01\x00\x00\x00\x00\x00\x00\x00\x19\x00\x00\x00o\x08\x00\x00\xb0\xaf\xba\xb1'
DEBUG:adb_shell.adb_device:bulk_write(25): b'shell:logcat -v epoch -d\x00'
DEBUG:adb_shell.adb_device:bulk_read(24): b'CLSE\xfd\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xbc\xb3\xac\xba'
DEBUG:adb_shell.adb_device:bulk_read(24): b'OKAY\xfe\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
DEBUG:adb_shell.adb_device:bulk_read(24): b'WRTE\xfe\x00\x00\x00\x01\x00\x00\x00\xad)\x00\x00e\xb0\x0b\x00\xa8\xad\xab\xba'
DEBUG:adb_shell.adb_device:bulk_read(10669): b'--------- beginning of main\n         1625162758.068  3294  4288 D audio_lHAL_hw_primary: out_standby_new: enter aml_out->out_device 2\n       
DEBUG:adb_shell.adb_device:bulk_read(7797): b" : CREATE function success\n         1625162758.825  7621  7621 I AVUtils : createInstance(32bit) : _ZN7android19createExtendedUtilsEv\n            1625162758.961  7621  7621 E System  : Unable to open boot classpath entry: /system/framework/exoplayer.jar\n         1625162758.961  7621  7621 E System  : java.io.FileNotFoundException: File doesn't exist: /sys
DEBUG:adb_shell.adb_device:bulk_read(2005): b"05, 9435]          1625162762.247  3294  4288 
DEBUG:adb_shell.adb_device:bulk_write(24): b'OKAY\x01\x00\x00\x00\xfe\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
DEBUG:adb_shell.adb_device:bulk_read(24): b'WRTE\xfe\x00\x00\x00\x01\x00\x00\x00)O\x00\x00mj\x16\x00\xa8\xad\xab\xba'
DEBUG:adb_shell.adb_device:bulk_read(20265): b'         1625162762.501  4640  4801 D 
DEBUG:adb_shell.adb_device:bulk_read(14473): b'rned from \n         1625162762.514  4640  7598 E AndroidRuntime: \tat com.example.androidaudioservice.core.HWEngine$HWStatusTracker.handleHighSensitiveUp
DEBUG:adb_shell.adb_device:bulk_read(8681): b'ore: Storing event with priority=HIGHEST, name=FIREBASE AASApplication: >>>>>>>>>>>>>>>>>>>>>>>>> UnhandledException,AudioService Exit <<<<<<<<<<<<<<<<<<<<<<<<<\n         1625162762.638  4640  7598 W System.err: java.lang.RuntimeException: 
DEBUG:adb_shell.adb_device:bulk_read(2889): b'se:{"success":"true"}\n         1625162763.625  4640  4680
DEBUG:adb_shell.adb_device:bulk_write(24): b'OKAY\x01\x00\x00\x00\xfe\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xb0\xb4\xbe\xa6'
DEBUG:adb_shell.adb_device:bulk_read(24): b'WRTE\xfe\x00\x00'
Traceback (most recent call last):
  File "/home/jx/.local/lib/python3.6/site-packages/adb_shell/adb_message.py", line 120, in unpack
    cmd, arg0, arg1, data_length, data_checksum, _ = struct.unpack(constants.MESSAGE_FORMAT, message)
struct.error: unpack requires a buffer of 24 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "_test_v1.py", line 217, in <module>
    run_recording_test_harness(args)
  File "_test_v1.py", line 132, in run_recording_test_harness
    log_dev=device.shell('logcat -v epoch -d')  #TODO limit to duration
  File "/home/xio/.local/lib/python3.6/site-packages/adb_shell/adb_device.py", line 446, in shell
    return self._service(b'shell', command.encode('utf8'), transport_timeout_s, read_timeout_s, timeout_s, decode)
  File "/home/xio/.local/lib/python3.6/site-packages/adb_shell/adb_device.py", line 334, in _service
    return b''.join(self._streaming_command(service, command, adb_info)).decode('utf8')
  File "/home/xio/.local/lib/python3.6/site-packages/adb_shell/adb_device.py", line 1027, in _streaming_command
    for data in self._read_until_close(adb_info):
  File "/home/xio/.local/lib/python3.6/site-packages/adb_shell/adb_device.py", line 962, in _read_until_close
    cmd, data = self._read_until([constants.CLSE, constants.WRTE], adb_info)
  File "/home/xio/.local/lib/python3.6/site-packages/adb_shell/adb_device.py", line 918, in _read_until
    cmd, remote_id2, local_id2, data = self._read(expected_cmds, adb_info)
  File "/home/xio/.local/lib/python3.6/site-packages/adb_shell/adb_device.py", line 868, in _read
    cmd, arg0, arg1, data_length, data_checksum = unpack(msg)
  File "/home/xio/.local/lib/python3.6/site-packages/adb_shell/adb_message.py", line 122, in unpack
    raise ValueError('Unable to unpack ADB command. (length={})'.format(len(message)), constants.MESSAGE_FORMAT, message, e)
ValueError: ('Unable to unpack ADB command. (length=7)', b'<6I', b'WRTE\xfe\x00\x00', error('unpack requires a buffer of 24 bytes',))

removed some private logs

JeffLIrion commented 3 years ago

@javierxio thanks for the log. The changes in this commit might fix the issue: https://github.com/JeffLIrion/adb_shell/commit/7d0d8327385f1942ae07e4dda83ec0351ea0015e

(You can ignore the changes outside of adb_device.py)

javierxio commented 3 years ago

@javierxio thanks for the log. The changes in this commit might fix the issue: 7d0d832

(You can ignore the changes outside of adb_device.py)

Jeff, are you saying that version 0.3.2 does already have this commit or I need to modify the adb_device file?

JeffLIrion commented 3 years ago

You would need to modify the file.

JeffLIrion commented 3 years ago

I think this pull request should fix the issue: https://github.com/JeffLIrion/adb_shell/pull/172

You can install that branch via

pip install https://github.com/JeffLIrion/adb_shell/archive/support-multiple-streams.zip

fatinghenji commented 3 years ago

I think this pull request should fix the issue: #172

You can install that branch via
pip install https://github.com/JeffLIrion/adb_shell/archive/support-multiple-streams.zip

First of all, sorry for my poor English, I am using Google Translate to communicate. I ran into the issue mentioned above https://github.com/JeffLIrion/adb_shell/issues/152#issuecomment-826415067

The operating environment I am using is as follows:

windows11
Android11
python 3.8
adbshell
- pip install https://github.com/JeffLIrion/adb_shell/archive/support-multiple-streams.zip
  Through different link methods, you will get different error results:

Via wired connection

File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 667, in connect
self._available, self._maxdata = self._io_manager.connect(self._banner, rsa_keys, auth_timeout_s, auth_callback, adb_info)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 199, in connect
cmd, arg0, maxdata, banner2 = self._read_expected_packet_from_device([constants.AUTH, constants.CNXN], adb_info)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 383, in _read_expected_packet_from_device
cmd, arg0, arg1, data = self._read_packet_from_device(adb_info)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 461, in _read_packet_from_device
msg = self._read_bytes_from_device(constants.MESSAGE_SIZE, adb_info)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 430, in _read_bytes_from_device
raise exceptions.AdbTimeoutError("Timeout: read {} of {} bytes (transport_timeout_s = {}, read_timeout_s = {})".format(len(data), len(data) + length, adb_info.transport_timeout_s, adb_info.read_timeout_s))
adb_shell.exceptions.AdbTimeoutError: Timeout: read 0 of 24 bytes (transport_timeout_s = 10.0, read_timeout_s = 10.0)

Connect via wireless

File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 667, in connect
self._available, self._maxdata = self._io_manager.connect(self._banner, rsa_keys, auth_timeout_s, auth_callback, adb_info)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 199, in connect
cmd, arg0, maxdata, banner2 = self._read_expected_packet_from_device([constants.AUTH, constants.CNXN], adb_info)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 383, in _read_expected_packet_from_device
cmd, arg0, arg1, data = self._read_packet_from_device(adb_info)
File "C:\Users\XXX\AppData\Local\Programs\Python\Python38\lib\site-packages\adb_shell\adb_device.py", line 466, in _read_packet_from_device
raise exceptions.InvalidCommandError("Unknown command: %d = '%s' (arg0 = %d, arg1 = %d, msg = '%s')" % (cmd, int_to_cmd(cmd), arg0, arg1, msg))
adb_shell.exceptions.InvalidCommandError: Unknown command: 1397511251 = 'b'STLS'' (arg0 = 16777216, arg1 = 0, msg = 'b'STLS\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xac\xab\xb3\xac'')

JeffLIrion commented 3 years ago

@fatinghenji thanks for testing it.

Wired connection

I assume you mean Ethernet, not USB; USB support is experimental. I don't know why there was no response from the device.

Wireless connection

It looks like "STLS" is a newer command that isn't supported by this package, so in that regard the error makes sense. https://cs.android.com/android/platform/superproject/+/master:packages/modules/adb/protocol.txt;l=82-88

--- STLS(type, version, "") --------------------------------------------

Command constant: A_STLS

The TLS message informs the recipient that the connection will be encrypted
and will need to perform a TLS handshake. version is the current version of
the protocol.

Question

Are you supplying a key when you connect, as demonstrated in the README? Also, maybe it would work if you copy a key pair that has already been accepted (e.g., connect using the adb binary, disconnect, and then try to connect using this package with the key that adb used)?

NateScarlet commented 3 years ago

@fatinghenji is using my code, it should auto generate a key if not exists

https://github.com/NateScarlet/auto-derby/blob/147c1d816f654de458d1b24c79ada98bfc3b7de2/auto_derby/clients/adb.py#L46-L50

fatinghenji commented 3 years ago

Wired connection

I assume you mean Ethernet, not USB; USB support is experimental.

Sorry again for my poor English... I mean to connect via USB, not via a wired Ethernet.

I don't know why there was no response from the device.

I also find it strange because I can see touch feedback on my phone when I perform swipe or tap commands via adb.

Question

Are you supplying a key when you connect, as demonstrated in the README? Also, maybe it would work if you copy a key pair that has already been accepted (e.g., connect using the adb binary, disconnect, and then try to connect using this package with the key that adb used)?

I replaced the mentioned files manually and the problem still persists. The error code is the same.

JeffLIrion commented 3 years ago

@fatinghenji the USB route won't work the way that you're doing it. @NateScarlet's code uses the AdbDeviceTcp class, which tries to connect via TCP. Your computer is connected via USB, which is why you see output. But I believe that only one ADB connection is allowed at a time, which is why the device didn't respond to your connection attempt using this package.

javierxio commented 3 years ago

do you see the device when you type adb devices ?

JeffLIrion commented 3 years ago

@javierxio the changes that I mentioned in a previous comment are in the latest release (0.4.0). Could you please let me know if the issue is fixed.

pip install -U adb-shell

javierxio commented 3 years ago

@javierxio the changes that I mentioned in a previous comment are in the latest release (0.4.0). Could you please let me know if the issue is fixed.
pip install -U adb-shell

will test and let you know. thanks a lot.

javierxio commented 3 years ago

@JeffLIrion Jeff, I'm testing v0.4.0 overnight (for almost 18 hours now) and it has been quite stable so far. Thanks a lot. I will keep running tests in the near future and with multiple devices so if I come across the issue again, I'll open another ticket at that time.

JeffLIrion commented 3 years ago

@javierxio thanks for reporting back! Would the issue have occurred during your 18 hours of testing with a prior version of this package?

javierxio commented 3 years ago

Jeff, previously, it was so unstable that I would get crashes within a couple of hours. It seems to be stable now. Thanks.

JeffLIrion / adb_shell