Unpack requires a bytes object of length 4

[justiceamoh]

Thanks for an awesome evernote plugin. I have been using your plugin for years and it has been a core piece of my workflow.

Recently, (perhaps after my upgrade to macOS Mojave), the plugin has been unable to connect to Evernote. When I attempt to send a new note, I get a plugin error. In the console, I get the following messages:

Evernote plugin error: unpack requires a bytes object of length 4 Last command: {}

I have tried uninstalling and reinstalling but to no avail. I looked through the Issues and wondered if it might be related to the bug referenced in #127. But no solutions are provided there and I wonder if this is an unrelated bug since I've always had this plugin working. Please advise. Thank you.

[bordaigorl]

Hi, it is most probably related to the mentioned issue. This is one of those bugs I could not fix, because I cannot reproduce them. My best guess is that there is a problem in the protocol layer of the Thrift library which is used by the Evernote SDK to exchange data across the network with the Evernote API. Now, unpack is responsible for converting C structures from raw bits to Python values. Unpack is used to do this conversion on some bytes received from the network. The error is complaining about wrong sixe of date to unpack. In Evernote/lib/thrift/protocol/TBinaryProtocol.py you can see that there is a pattern of usage of unpack: first you read k bytes, then you unpack them using the format which is supposed to consume exactly those bytes, for example:

buff = self.trans.readAll(1)
val, = unpack('!b', buff)

I have the feeling that there may be some platform dependent mismatch between the sizes required by the "!x" formats and the standard sizes hardcoded in the arguments of readAll.

Could you please open the ST console and run:

from struct import *
[calcsize('!'+x) for x in "ibhqd"]

if the result is different from [4, 1, 2, 8, 8] then we identified the problem.

Otherwise, my best guess is that the network is somehow not communicating these bytes smoothly so the buffering logic of Thrift gets confused and not enough bytes get returned by readAll. This hypothesis could be tested by adding some logging after each readAll call in Evernote/lib/thrift/protocol/TBinaryProtocol.py with the pattern:

buff = self.trans.readAll(SIZE)
print(">> EXPECTED: %d | RECEIVED %d <<" % (SIZE, len(buff)))
val, = unpack('!b', buff)

I cannot try this because I get no error...

[justiceamoh]

Thanks for getting back @bordaigorl, really appreciate it.

So I ran the first command and my output matched the expected [4, 1, 2, 8, 8].

Then i included the print statements in TBinaryProtocol.py and got the following output:

>> EXPECTED: 4 | RECEIVED 0 <<
Evernote plugin error: unpack requires a bytes object of length 4
    Last command:  {}

I think that's happening in the readI32() method. Any ideas what might be causing that?

[bordaigorl]

Hey thank you very much for helping out!

I am really not getting this. The definition of readAll is

  def readAll(self, sz):
    buff = ''
    have = 0
    while (have < sz):
      chunk = self.read(sz - have)
      have += len(chunk)
      buff += chunk

      if len(chunk) == 0:
        raise EOFError()

    return buff

So the question is: how is it possible to get b = self.trans.readAll(4) with len(b)==0? Since sz == 4, if we get any data at all buff is not zero-length. Then if we get an empty chunk (which is what should happen if we ever try to return some empty buff) then we should raise an exception, but that does not happen because b is set to some value without exceptions and we get an exception from running unpack on it. I'm out of ideas, but it would seem this code is not receiving data. Are you behind a firewall/proxy?

[justiceamoh]

Thank you for getting back. Yeah, that's very curious. So on my macbook pro, I use a firewall. But the problem persists even after I turn off the firewall.

Furthermore, I use Sublimetext and Sublime-Evernote on 2 other computers: a linux box (running Ubuntu Mate), and an iMac (which I just upgraded to Mojave). I synchronize my Sublime-text settings across all three using Dropbox. And i have the same problem on all three machines, where Sublime-Evernote wouldn't work. I don't have a firewall on those machines.

[bordaigorl]

I would start from readAll. Change it to

  def readAll(self, sz, logtag=None):
    buff = ''
    have = 0
    while (have < sz):
      chunk = self.read(sz - have)
      if logtag:
        print(logtag,sz,chunk,len(chunk),have,buff)
      have += len(chunk)
      buff += chunk

      if len(chunk) == 0:
        raise EOFError()

    if logtag:
      print(logtag,sz,len(buff),have,buff)

    return buff

Then replace the suspect calls to self.trans.readAll(SIZE) with self.trans.readAll(SIZE, "SOME ID") and look at the trace.

[bordaigorl]

Hi @justiceamoh, could you have a look at #127 and see if the last comment applies to you?

[justiceamoh]

You're right! that actually worked -- reconfiguring and resetting the developer token and keys. Thank you!