search

fellerts / procal

Simple Qt-based programming calculator
GNU General Public License v3.0
167 stars 8 forks source link

fix problem with lost precision when casting #11

Closed amundas closed 1 year ago

amundas commented 1 year ago

Some inputs cause unexpected behaviour. One example is the input "0xffffffffffffff" in 64-bit mode which is shown as "0x100000000000000". This problem comes from lost precision when inputs are cast to float after evaluation. Since python's built in float type is 64-bit, it cannot represent all 64-bit integers. This commit allows the evaluated input to take whichever type suits it, removing the problem due to casting.

fellerts commented 1 year ago

Looks good, works well here! Thanks :)