search

pothosware / PothosPython

Python bindings for the Pothos framework
https://github.com/pothosware/PothosPython/wiki
Boost Software License 1.0
7 stars 5 forks source link

Currently no way to get high-value uint64 values from Python to C++ #18

Closed ncorgan closed 4 years ago

ncorgan commented 5 years ago

A current limitation of the Proxy infrastructure is that a specific target-language type can only be converted to a single C++ type. This poses an issue with Python's int or long types. Given the previously mentioned limitation, the conversion back to C++ is implemented as returning a long long.

This means that numbers that are in the range of unsigned long long that aren't in the range of long long cannot accurately be returned to C++, even though internal to the blocks, NumPy supports using these types.

guruofquality commented 5 years ago

Thats a good point, I think that because the long long actually can store the 64-bit number without loss, it should work out. If the target type is lets say uint64_t, pothos should convert the pylong into a long long, and then from the long long into the uint64_t. Because there is no direct conversion it will use this intermediate method. I will have to confirm with a unit test once I merge your branch.

ncorgan commented 5 years ago

I made the issue separate because I couldn't fix it at the same time I was doing the C++ -> Python branch. I have a local unit test that does a number -> proxy -> number equality test with the numeric_limits min() and max() for all supported types. As of my latest push, everything passes except uint64_t because Pothos converts the PyLong to long long (int64_t) and then to uint64_t, so anything higher than a long long's range is lost.

guruofquality commented 5 years ago

Your right, I think its the python APIs prevent getting the number out when it overflows with the wrong type, but I should be able to make converters for both signed and unsigned separately (at least for python3) with PyLong_AsUnsignedLongLong. Do you happen to have the change with the unit test checks?

ncorgan commented 5 years ago

I've attached a git diff patch here.

I couldn't get past the issue of a single conversion function being linked to int/long, no matter if the number is signed or unsigned.

guruofquality commented 5 years ago

I remember now. The conversions really only support a 1:1 type mapping, which is probably fine for most use cases. And so I wanted to make unsigned long long the universal integer type because PothosCore knows how to convert that to a number of other native types. This has a downside for the extreme ranges of the uint64ts.

Funny enough if I use PyLong_AsUnsignedLongLongMask and remove the bounds checks in PothosCore, this also starts working (at least in the tests I did). Because the bits arent lost, they are just set to the output type and truncated, which twos compliment is fine with.

The thing I was avoiding doing which would have really solved this is having a Pothos::BigInteger type which can hold any native integer type. The python function would just output a Pothos::BigInteger, and in PothosCore it would just convert between BigInteger and a bunch of native types. This also would have saved a bunch of those intermediate conversions where I supported both unsigned and signed long long for all types.

ncorgan commented 5 years ago

I'm looking into this now.

ncorgan commented 5 years ago

I have an attempt here that works for this case but breaks others:

PothosCore: re-enable conversion from string to slong, sllong, ulong, ullong PothosPython: return number as string to avoid dealing with signs here

The new test (now part of this PothosPython branch) passes now, but it seems to break other string conversions, as they try to call these functions and error out when the string isn't a valid number.