newstr object instead of str object python2.7

[vvynohradov]

When I am using param to get a string inside proxy object, I can't directly receive a str object, I am receiving an object with <class 'future.types.newstr.newstr'> type. My proxy object:

from iotile.core.hw.proxy.proxy import TileBusProxyObject
from iotile.core.utilities.typedargs.annotate import context, param

@context("DemoProxy1")
class DemoProxyObject(TileBusProxyObject):

    @classmethod
    def ModuleName(cls):
        return 'Demo01'

    @param("test_var", "str")
    def get_test_rpc_v1(self, test_var):
        print(type(test_var))

Steps to reproduce:

1. Run get_test_rpc_v1

[vvynohradov]

As far as I understood, we are receiving a unicode object instead of an ascii string, and we have the same behavior in python3 and higher, we are always receiving a unicode string instead of an ascii string. I think this is not really understandable, because if we are going to pass our param directly to the rpc that takes a byte string, we are going to receive this exception struct.error: argument for 's' must be a bytes object. Maybe we should return an ascii string when we are using python2.7 and return a unicode string when we are using python3 and higher. Or we can add different(unicode and ascii) string types to typedargs. Do you have any suggestions, @timburke ?

[timburke]

@vvynohradov It is true that you get a newstr object from typedargs on both python 2 and 3. It cannot be directly passed to struct.pack because it does not specify an encoding to know how it should be represented in binary form.

So, an explicit param.encode('utf-8') or param.encode('ascii') if you prefer is required before you can pass it to struct.pack or anything that internally calls struct.pack.

While it is True that this would have worked on python 2 with a native str when we were initially porting to python 3 we determined that there were too many subtle cases where our code would need to know whether it needed to call encode or not before passing to struct.pack and rather than changing all of those places to check if they were on python 2 or 3, we just changed to always use the python 3 behavior on both python 2 and 3 so that it is required to call encode on a string you receive from the terminal before you pass it to struct.pack.

This was determined to be the safest way to migrate our historically python 2 only codebase to python 2 and 3. The impact of this choice is not very visible inside the public Coretools packages but there is a large amount of internal code at Arch that builds on CoreTools (and hence typedargs) for which the difference in work would have been considerable.

Note that we're now in the process of moving CoreTools to be python 3 only so we will be dropping the future package and newstr entirely and just returning python 3 strings natively in the near future for new CoreTools releases.

[vvynohradov]

Thank you for you reply and clarification. I think that we should also add info into to the manual that @param decorator returns a unicode string.