Floating point exception converting arxml -> dbc

[kfurge-egi]

I have a proprietary arxml file (that, unfortunately, I cannot share) which is causing a floating point exception while converting to a .dbc format.

Signals causing the exception have the following XML definition:

   <SYSTEM-SIGNAL UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx">
     <SHORT-NAME>ABC_XYZ</SHORT-NAME>
     <DYNAMIC-LENGTH>true</DYNAMIC-LENGTH>
   </SYSTEM-SIGNAL>

The DYNAMIC-LENGTH field being the key. Whenever this is present, the LENGTH field in the complimentary signal description contains an unreasonably long length like:

   <I-SIGNAL UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx">
     <SHORT-NAME>ABC_XYZ</SHORT-NAME>
     <LONG-NAME>
       <L-4 L="EN">ABC_XYZ_LONG</L-4>
     </LONG-NAME>
     <DESC>
       <L-2 L="EN">ABC_XYZ_DESC</L-2>
     </DESC>
     <DATA-TYPE-POLICY>LEGACY</DATA-TYPE-POLICY>
     <LENGTH>32760</LENGTH>
     <SYSTEM-SIGNAL-REF DEST="SYSTEM-SIGNAL">/x/y/ABC_XYZ</SYSTEM-SIGNAL-REF>
   </I-SIGNAL>

This causes Signal.calculateRawRange() to blow up with a floating point exception. The hack below keeps the exception from happening:

    def calculateRawRange(self):
        size = 0
        if self.size < 1024:
            size = self.size
        else:
            print("huge size found Signal: %s size: %d" %
                  (self.name if self.name else "unknown", self.size))
        rawRange = 2 ** size
        if self.is_signed:
            rawRange /= 2
        return (self.float_factory(-rawRange if self.is_signed else 0),
                self.float_factory(rawRange - 1))

I'm not proposing this code as a solution. Rather, it is an illustration that allows the conversion to complete with this particular file.

[altendky]

Are you using latest from the develop branch? I thought we switched over to decimal, but perhaps we missed some spots.

[kfurge-egi]

I'm working with the development branch from late last week some time. The last commit to canmatrix.py in the code I have is:

commit cdb8bd36b21321d7febbc45a705d7afb56ab198f
Author: ebroecker <eduard@gmx.de>
Date:   Sat Aug 11 23:49:26 2018 +0200

    do runtime check for attr.s lib version

I believe that's the same as the current development branch. I'll sync up again later today with the latest to confirm.

[altendky]

It looks like there was at least a float factory related change after that (e2c187595b4ff04e0a91ec5ae4d6848fba91dae0). If that doesn't fix it could you maybe provide a minimal arxml file that causes the trouble? You seem to have isolated the relevant part already so maybe that won't be hard. I'm vaguely familiar with the autosar concept, but I can't say I actually know the formats yet.

[kfurge-egi]

I confirmed that the latest development branch code has the same result. Please see below for a more specific output dump of the crash.

Regarding a minimal arxml file, unfortunately I'm a newbie with the format myself and my odds of being able to put together a proper file are about zero at this point.

I was hoping someone who knows the format/specs better than I would be able to wrap some rules around what appears to be an unreasonable value (2^32760) with the xml extracts I provided.

INFO - convert - Importing XYZ.ARXML ... 
DEBUG - arxml - Read arxml ...
DEBUG - arxml -  Done

DEBUG - arxml - Build arTree ...
DEBUG - arxml -  Done

DEBUG - arxml - DEBUG 547 frames in arxml...
DEBUG - arxml - DEBUG 547 can-frame-triggering in arxml...
DEBUG - arxml - DEBUG 0 SIGNAL-TO-PDU-MAPPINGS in arxml...
DEBUG - arxml - DEBUG 4276 I-SIGNAL-TO-I-PDU-MAPPING in arxml...
DEBUG - arxml - Busname: CAN_5_Cluster
DEBUG - arxml -  Speed: 500
DEBUG - arxml - 547 frames found in arxml

DEBUG - arxml - processing Frame: XXX
DEBUG - arxml - XXX_PDU
DEBUG - arxml - No Compmethod found!! - try alternate scheme 1.
DEBUG - arxml - No Compmethod found!! - fuzzy search in syssignal.
Traceback (most recent call last):
  File "/home/kfurge/.local/bin/canconvert", line 11, in <module>
    load_entry_point('canmatrix', 'console_scripts', 'canconvert')()
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/convert.py", line 393, in main
    convert(infile, outfileName, **cmdlineOptions.__dict__)
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/convert.py", line 41, in convert
    dbs = canmatrix.formats.loadp(infile, **options)
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/formats.py", line 64, in loadp
    return load(fileObject, importType, key, flatImport, **options)
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/formats.py", line 74, in load
    dbs = moduleInstance.load(fileObject, **options)
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/arxml.py", line 1556, in load
    db.addFrame(getFrame(frameTrig,topLevelPackages,multiplexTranslation,ns, float_factory))
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/arxml.py", line 1338, in getFrame
    getSignals(signaltopdumaps, newFrame, xmlRoot, ns, None, float_factory)
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/arxml.py", line 1155, in getSignals
    is_float=is_float)
  File "<attrs generated init 487f1c0fc5040942b627787b10dedace8cd2fabc>", line 12, in __init__
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/canmatrix.py", line 123, in setDefaultMin
    return  self.calcMin()
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/canmatrix.py", line 265, in calcMin
    rawMin = self.calculateRawRange()[0]
  File "/home/kfurge/Customers/EEPod/arxml/canmatrix2/canmatrix/canmatrix/canmatrix.py", line 253, in calculateRawRange
    rawRange /= 2
  File "/usr/local/lib/python2.7/dist-packages/future/types/newint.py", line 175, in __itruediv__
    mylong /= other
OverflowError: integer division result too large for a float

[altendky]

I recreated something similarish anyways. I'll try to address that and we'll see where we are.

[altendky]

@kfurge-egi, can you give that branch in my fork a try?

@ebroecker, perhaps float_factory should just be number_factory (or drop the factory thing but...). Anyways, I didn't dig hard at all but I'm assuming we've got an <int> / <int> scenario somewhere infecting us with a float (see below). Stick a decimal on either side and you get a decimal out instead.

But... perhaps it's more a py2/3 thing and we need to be forcing integer division. I'll try to track down the actual trigger I guess. reluctantly goes back to the code

[altendky]

Yeah, that's probably better here. 330391c87d016d2e5f5d1eed695f10746b591952

We do import division from the future so no 2/3 variability. Rather yeah, don't raise 2 to an exponent then divide by 2. Just raise it to one lower power. But, please really review it (wouldn't unit tests be nice :] #185).

[kfurge-egi]

Apologies for the delay, I tried the suggested code and it no longer blew up, but got caught in an endless loop somewhere that I'll need to track down. I'll be able to look at it further in the next day or two.

[kfurge-egi]

I did a little more investigation into the lockup. It's due to the recent changes introduced in 'development' by following commit:

commit 693df7d0eaa4c0667dc162f64f4807e63d146f07
Author: ebroecker <eduard@gmx.de>
Date:   Fri Aug 17 21:51:39 2018 +0200

    arxml update, use a more xpath like solution

The following function in arxml.py is not returning:

def getArPath(tree, arPath, namespaces):
    global ArCache
    namespaceMap = {'A': namespaces[1:-1]}
    baseARPath = arPath[:arPath.rfind('/')]
    if baseARPath in ArCache:
        baseElement = ArCache[baseARPath]
    else:
        xbasePath= arPath2xPath(baseARPath)
        baseElement = tree.xpath(xbasePath, namespaces=namespaceMap)[0]
        ArCache[baseARPath] = baseElement
    found = baseElement.xpath(".//A:SHORT-NAME[text()='" + arPath[arPath.rfind('/')+1:] + "']/..", namespaces=namespaceMap)[0]
    return found

An invalid path is causing tree.xpath() to never return. Now that I'm past this, I'll test the PR without my hack applied.

[kfurge-egi]

I applied 330391 and it works well. Thanks for chasing this down.

[altendky]

@kfurge-egi, so the PR is good for you as is? @ebroecker, review? (after vacation, of course :] )

[kfurge-egi]

Yes, PR #186 is good for me. Thanks.

[ebroecker]

I merged PR #186. @kfurge-egi : could I close this issue?

[kfurge-egi]

Sure will, thank you both for all of your support.