search

ebroecker / canmatrix

Converting Can (Controller Area Network) Database Formats .arxml .dbc .dbf .kcd ...
BSD 2-Clause "Simplified" License
924 stars 399 forks source link

Issue converting arxml -> dbc #23

Closed nostar closed 8 years ago

nostar commented 8 years ago

Build arTree ... Traceback (most recent call last): File "/usr/bin/canconvert", line 9, in load_entry_point('canmatrix==0.2', 'console_scripts', 'canconvert')() File "build/bdist.cygwin-2.3.1-i686/egg/canmatrix/convert.py", line 145, in main File "build/bdist.cygwin-2.3.1-i686/egg/canmatrix/convert.py", line 50, in convert File "build/bdist.cygwin-2.3.1-i686/egg/canmatrix/importarxml.py", line 356, in importArxml File "build/bdist.cygwin-2.3.1-i686/egg/canmatrix/autosarhelper.py", line 47, in arParseTree TypeError: 'NoneType' object is not iterable

Im pretty sure this is a problem with the arxml file. Is there somedebug code I can add or uncomment to help identify where in the arxml file the issue is?

ebroecker commented 8 years ago

please try the branch "issue23". I made arParseTree a bit more stable and added debuginfo to autosarhelper.py;

I guess you can't send me your arxml?

Eduard

nostar commented 8 years ago

unfortunately I cant. Here is the output now:

Traceback (most recent call last): File "/usr/bin/canconvert", line 9, in load_entry_point('canmatrix==0.2', 'console_scripts', 'canconvert')() File "build/bdist.cygwin-2.3.1-i686/egg/canmatrix/convert.py", line 145, in main File "build/bdist.cygwin-2.3.1-i686/egg/canmatrix/convert.py", line 50, in convert File "build/bdist.cygwin-2.3.1-i686/egg/canmatrix/importarxml.py", line 354, in importArxml AttributeError: 'NoneType' object has no attribute 'len'

ebroecker commented 8 years ago

seems, your .arxml doesn't have 'TOP-LEVEL-PACKAGES'. I'm not sure why the code starts building the artree under these tags only. Please try once more the branch (I have to iterate the code to a unknown target...).

nostar commented 8 years ago

This time it succeeded but with a dbc file with no frames:

no yaml-export-support, some dependencys missing ... try pip install pyyaml no yaml-import-support, some dependencys missing ... , try pip install yaml Importing GB_ASR_TCM.arxml ... Read arxml ... Done

Build arTree ... Done

Busname: CAN_2_Cluster done

Exporting gb.dbc ... CAN_2_Cluster 0 Frames found done

nostar commented 8 years ago

FYI this is a 17MB arxml with hundreds of frames, but even vector autosar explorer complains about the file. I can continuie and browse all of the frames, signals, etc:

arxmlerror

ebroecker commented 8 years ago

OK - we are getting closer to the issue. The can-cluster is found now.

I think your system description has frames in (otherwise the vector tool cannot show them) but the frames are not references in the defined can-cluster.

I added even more debug-prints. If it helps, I could add an option to export all frames from an arxml in one dbc independently if it is referenced from a can-cluster...

p.s: If I have one guess free: is it a GM system description?

nostar commented 8 years ago

It is, but provided by a supplier and very alpha. Also ultimately im looking to go to DBF since Im outfitting our lab with busmaster and Peak hardware to replace Vector. For now im converting to dbc for ease of browsing the file with vectors databse editor.

Heres the latest output:

Read arxml ... Done

Build arTree ... Done

DEBUG 0 frames in arxml... Busname: CAN_2_Cluster Error - PHYSICAL-CHANNELS not found Error - PHYSICAL-CHANNEL not found Error - FRAME-TRIGGERINGS not found 0 frames found in arxml

done

Exporting gb.dbc ... CAN_2_Cluster 0 Frames found done

ebroecker commented 8 years ago

Hmm - your .arxml seems to be very buggy. Do you have any "CAN-FRAME-TRIGGERING" elements in your arxml? This is tested with the newest update. If no there are no "CAN-FRAME-TRIGGERING" elements in the arxml, I have no Idea how to get the CAN-ID. But if they are in, I could add the former mentioned option to export all can-frames to one dbc.

So, lets try further on, and if we have a solution, I'm going to implement some feature to work with this buggy .arxmls.

ebroecker commented 8 years ago

I did some update in the "issue23"-brach -- please test once again

2016/01/04 Edit: Did you have time testing the patches?

nostar commented 8 years ago

Happy new year! The output from the latest issue23 branch is pasted below. FWIW the only thing I am interested in at this point (for the output of a dbc/dbf file) is message frames with signal definitions. Any message / signal attributes, PDU Groups, sending / receiving nodes, etc arent necessary at this point. Soon I will be well versed in the arxml format, but I'm not there yet:

~/src/canmatrix-issue23 $ canconvert GB_ASR_TCM.arxml gb.dbc no yaml-export-support, some dependencys missing ... try pip install pyyaml no yaml-import-support, some dependencys missing ... , try pip install yaml Importing GB_ASR_TCM.arxml ... Read arxml ... Done

Build arTree ... Done

DEBUG 0 frames in arxml... DEBUG 85 can-frame-triggering in arxml... Busname: CAN_2_Cluster Error - PHYSICAL-CHANNELS not found Error - PHYSICAL-CHANNEL not found Error - FRAME-TRIGGERINGS not found 0 frames found in arxml

done

Exporting gb.dbc ... CAN_2_Cluster 0 Frames found done

ebroecker commented 8 years ago

Happy new year!

The debug output was helpful. There are 85 frames in the arxml. Please try converting with following (new) command line option: --arxmlIgnoreClusterInfo This should hopefully work for your needs.

canconvert --arxmlIgnoreClusterInfo GB_ASR_TCM.arxml gb.dbc

nostar commented 8 years ago

So close... I have a dbc file with all 85 frames with correct name & ID but no signals.

ebroecker commented 8 years ago

ok, the info you want is still missing. I added some new debug-info. Could you please try once more and post the result of the run?

The triggering info and CAN-ID info is stored in another place than the pdu signal mapping.
It seems the reference from the triggering info to the signal mapping is missing. If this is the root cause of this issue, maybe I could solve this by name-matching. Your next run of canconvert could help verifying this assumption.

nostar commented 8 years ago

~/src/canmatrix-issue23 $ canconvert --arxmlIgnoreClusterInfo GB_ASR_TCM.arxml gb.dbc no yaml-export-support, some dependencys missing ... try pip install pyyaml no yaml-import-support, some dependencys missing ... , try pip install yaml Importing GB_ASR_TCM.arxml ... Read arxml ... Done

Build arTree ... Done

DEBUG 0 frames in arxml... DEBUG 85 can-frame-triggering in arxml... DEBUG 0 SIGNAL-I-PDU in arxml... DEBUG: Frame xxx_MSG no SIGNAL-TO-PDU-MAPPINGS found DEBUG: Frame xxx_MSG no I-SIGNAL-TO-I-PDU-MAPPING found (This last 2 lines repeated for all frames)

ebroecker commented 8 years ago

I added further debug info. could you try one more? Note: You need following command line option additionally: -vv

canconvert -vv --arxmlIgnoreClusterInfo GB_ASR_TCM.arxml gb.dbc

nostar commented 8 years ago

~/src/canmatrix-issue23 $ canconvert -vv --arxmlIgnoreClusterInfo GB_ASR_TCM.arxml gb.dbc WARNING - exportall - no yaml-export-support, some dependencies missing ... try pip install pyyaml WARNING - exportall - no yaml-export-support, some dependencies missing ... (probably yaml) WARNING - importall - no yaml-import-support, some dependencies missing ... , try pip install yaml INFO - convert - Importing GB_ASR_TCM.arxml ... DEBUG - importarxml - Read arxml ... DEBUG - importarxml - Done

DEBUG - importarxml - Build arTree ... DEBUG - importarxml - Done

DEBUG - importarxml - DEBUG 0 frames in arxml... DEBUG - importarxml - DEBUG 85 can-frame-triggering in arxml... DEBUG - importarxml - DEBUG 0 SIGNAL-TO-PDU-MAPPINGS in arxml... DEBUG - importarxml - DEBUG 621 I-SIGNAL-TO-I-PDU-MAPPING in arxml... DEBUG: Frame xxx_MSG no SIGNAL-TO-PDU-MAPPINGS found DEBUG: Frame xxx_MSG no I-SIGNAL-TO-I-PDU-MAPPING found (Last 2 lines repeated for all frames, 170 lines) ... INFO - convert - done

INFO - convert - Exporting gb.dbc ... INFO - convert - INFO - convert - 85 Frames found INFO - convert - done

ebroecker commented 8 years ago

this is strange: I-SIGNAL-TO-I-PDU-MAPPING without SIGNAL-TO-PDU-MAPPINGS... I think your arxml is corrupt.

I need your help: What is the parent xml tag of one I-SIGNAL-TO-I-PDU-MAPPING tag? Perhaps it would help to know also the parent of the parent xml tag...

nostar commented 8 years ago 
<I-SIGNAL-TO-PDU-MAPPINGS>
  <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxxx">
   <SHORT-NAME>Signal01</SHORT-NAME>
   <LONG-NAME>
    <L-4 L="EN">Signal01</L-4>
   </LONG-NAME>
   <I-SIGNAL-REF DEST="I-SIGNAL">/ISignalContainer/ISignal01</I-SIGNAL-REF>
   <PACKING-BYTE-ORDER>OPAQUE</PACKING-BYTE-ORDER>
   <START-POSITION>0</START-POSITION>
 </I-SIGNAL-TO-I-PDU-MAPPING>
</I-SIGNAL-TO-PDU-MAPPINGS>

the next level up from this is ELEMENT

nostar commented 8 years ago

I would not put any effort into this until I get word back from the supplier that provided this file. I noticed that all the start bits of all signals in all messages are 0 as well, so I contacted them and come to find out they had the same problem and had it fixed for them. I'm waiting for an updated file, I'll report my results afterwards. Thanks for all the help so far!

ebroecker commented 8 years ago

I'd expect something like:

<ELEMENTS>
        <SIGNAL-I-PDU>
          <SHORT-NAME>SomeName</SHORT-NAME>
          <LENGTH>64</LENGTH>
          <SIGNAL-TO-PDU-MAPPINGS>
            <I-SIGNAL-TO-I-PDU-MAPPING>
              <SHORT-NAME>Signal1</SHORT-NAME>
              <START-POSITION>1</START-POSITION>
              <PACKING-BYTE-ORDER>MOST-SIGNIFICANT-BYTE-FIRST</PACKING-BYTE-ORDER>
              <SIGNAL-REF DEST="I-SIGNAL">/ISignal/Signal1</SIGNAL-REF>
            </I-SIGNAL-TO-I-PDU-MAPPING>

Nevertheless, if your ARXML has wrong "START-POSITIONS" we are not able to generate a dbc... Lets wait for an updated arxml.

ebroecker commented 8 years ago

I close this issue now. I integrated all issues that came up during analysing issue23 and merged to master. As showed up your .arxml does currently not include all informations needed for correct import and conversion. Please come back with creating a new issue as soon as you have new arxml and still problems converting it.

ebroecker commented 8 years ago

good news: I talked to a college who gave me some good hints. Maybe you could once more try the issue23 branch...

nostar commented 8 years ago 
~/src/canmatrix-issue23 $ canconvert -vv --arxmlIgnoreClusterInfo GB_ASR_TCM_fixed.arxml gb.dbc
WARNING - exportall - no yaml-export-support, some dependencies missing ...  try pip install pyyaml
WARNING - exportall - no yaml-export-support, some dependencies missing ... (probably yaml)
WARNING - importall - no yaml-import-support, some dependencies missing ... , try pip install yaml
INFO - convert - Importing GB_ASR_TCM_fixed.arxml ...
DEBUG - importarxml - Read arxml ...
DEBUG - importarxml -  Done

DEBUG - importarxml - Build arTree ...
DEBUG - importarxml -  Done

DEBUG - importarxml - DEBUG 0 frames in arxml...
DEBUG - importarxml - DEBUG 85 can-frame-triggering in arxml...
DEBUG - importarxml - DEBUG 0 SIGNAL-TO-PDU-MAPPINGS in arxml...
DEBUG - importarxml - DEBUG 621 I-SIGNAL-TO-I-PDU-MAPPING in arxml...
DEBUG - importarxml - DEBUG: Frame Frame00 no I-SIGNAL-TO-I-PDU-MAPPING found
DEBUG - importarxml - DEBUG: Frame Frame01 no I-SIGNAL-TO-I-PDU-MAPPING found
DEBUG - importarxml - DEBUG: Frame Frame02 no I-SIGNAL-TO-I-PDU-MAPPING found
DEBUG - importarxml - DEBUG: Frame Frame03 no I-SIGNAL-TO-I-PDU-MAPPING found
DEBUG - importarxml - DEBUG: Frame Frame04 no I-SIGNAL-TO-I-PDU-MAPPING found
INFO - convert - done

INFO - convert - Exporting gb.dbc ...
INFO - convert -
INFO - convert - 85 Frames found
INFO - convert - done

Now I have a dbc file with frames and signals, but the signal start bits are mangled. Some of them are negative numbers which cause syntax errors when trying to open in vector dbc editor.

The supplier confirmed that the original arxml file I was using was corrupt, so they gave me this replacement which i called 'fixed'. I still get the unhandled exception error upon opening in vector autosar explorer along with 0 for start bits on all signals as before, but they do have correctly mapped signals to frames. Right now all we know is that I have autosar explorer 1.3.x and they have 1.7.x.

ebroecker commented 8 years ago

did you try without "--arxmlIgnoreClusterInfo" also? Should also work now. Maybe the startbits are just wrong in arxml?

nostar commented 8 years ago

So in the suppliers version of autosar explorer, all of the signal start positions are correct. Using MSG_615 as an example, here is what the dbc output would be if it were correct:

BO_ 615 MSG_615 : 7 Vector__XXX
 SG_ Sig01 : 39|12@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig02 : 43|1@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig03 : 7|27@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig04 : 49|2@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig05 : 28|5@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig06 : 42|8@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig07 : 50|1@0+ (1.0,0) [0|1] "" Vector__XXX

Here is what is in the generated dbc file:

BO_ 615 MSG_615 : 7 Vector__XXX
 SG_ Sig01 : 34|12@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig02 : 43|1@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig03 : -15|27@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig04 : 50|2@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig05 : 32|5@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig06 : 49|8@0+ (1.0,0) [0|1] "" Vector__XXX
 SG_ Sig07 : 50|1@0+ (1.0,0) [0|1] "" Vector__XXX

Now here is a snippet of the arxml file that defines the signals and their start positions, all of which match the 'correct' layout for this message. Notice the weird naming. All of the short names are Sig01, with an incremented number appended after actual Sig01. The long names are all simply 'Sig01' and the actual signal name that appears in AE of both versions is the name after /ISignalContainer/ in I-SIGNAL-REF

Finally, the 8th I-SIGNAL-TO-I-PDU-MAPPING is actually the PDU that all of these signals belong to according to the suppliers version of AE, where in mine, the PDU is blank with a red exclamation and all signals with start bit of 0 with yellow exclamation next to each.

<I-SIGNAL-TO-PDU-MAPPINGS>
    <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxx">
        <SHORT-NAME>Sig01</SHORT-NAME>
        <LONG-NAME>
            <L-4 L="EN">Sig01</L-4>
        </LONG-NAME>
        <I-SIGNAL-REF DEST="I-SIGNAL">/ISignalContainer/ISig01</I-SIGNAL-REF>
        <PACKING-BYTE-ORDER>MOST-SIGNIFICANT-BYTE-FIRST</PACKING-BYTE-ORDER>
        <START-POSITION>39</START-POSITION>
    </I-SIGNAL-TO-I-PDU-MAPPING>
    <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxx">
        <SHORT-NAME>Sig01_1</SHORT-NAME>
        <LONG-NAME>
            <L-4 L="EN">Sig01</L-4>
        </LONG-NAME>
        <I-SIGNAL-REF DEST="I-SIGNAL">/ISignalContainer/ISig02</I-SIGNAL-REF>
        <PACKING-BYTE-ORDER>MOST-SIGNIFICANT-BYTE-FIRST</PACKING-BYTE-ORDER>
        <START-POSITION>43</START-POSITION>
    </I-SIGNAL-TO-I-PDU-MAPPING>
    <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxx">
        <SHORT-NAME>Sig01_2</SHORT-NAME>
        <LONG-NAME>
            <L-4 L="EN">Sig01</L-4>
        </LONG-NAME>
        <I-SIGNAL-REF DEST="I-SIGNAL">/ISignalContainer/ISig03</I-SIGNAL-REF>
        <PACKING-BYTE-ORDER>MOST-SIGNIFICANT-BYTE-FIRST</PACKING-BYTE-ORDER>
        <START-POSITION>7</START-POSITION>
    </I-SIGNAL-TO-I-PDU-MAPPING>
    <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxx">
        <SHORT-NAME>Sig01_3</SHORT-NAME>
        <LONG-NAME>
            <L-4 L="EN">Sig01</L-4>
        </LONG-NAME>
        <I-SIGNAL-REF DEST="I-SIGNAL">/ISignalContainer/ISig04</I-SIGNAL-REF>
        <PACKING-BYTE-ORDER>MOST-SIGNIFICANT-BYTE-FIRST</PACKING-BYTE-ORDER>
        <START-POSITION>49</START-POSITION>
    </I-SIGNAL-TO-I-PDU-MAPPING>
    <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxx">
        <SHORT-NAME>Sig01_4</SHORT-NAME>
        <LONG-NAME>
            <L-4 L="EN">Sig01</L-4>
        </LONG-NAME>
        <I-SIGNAL-REF DEST="I-SIGNAL">/ISignalContainer/ISig05</I-SIGNAL-REF>
        <PACKING-BYTE-ORDER>MOST-SIGNIFICANT-BYTE-FIRST</PACKING-BYTE-ORDER>
        <START-POSITION>28</START-POSITION>
    </I-SIGNAL-TO-I-PDU-MAPPING>
    <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxx">
        <SHORT-NAME>Sig01_5</SHORT-NAME>
        <LONG-NAME>
            <L-4 L="EN">Sig01</L-4>
        </LONG-NAME>
        <I-SIGNAL-REF DEST="I-SIGNAL">/ISignalContainer/ISig06</I-SIGNAL-REF>
        <PACKING-BYTE-ORDER>MOST-SIGNIFICANT-BYTE-FIRST</PACKING-BYTE-ORDER>
        <START-POSITION>42</START-POSITION>
    </I-SIGNAL-TO-I-PDU-MAPPING>
    <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxx">
        <SHORT-NAME>Sig01_6</SHORT-NAME>
        <LONG-NAME>
            <L-4 L="EN">Sig01</L-4>
        </LONG-NAME>
        <I-SIGNAL-REF DEST="I-SIGNAL">/ISignalContainer/ISig07</I-SIGNAL-REF>
        <PACKING-BYTE-ORDER>MOST-SIGNIFICANT-BYTE-FIRST</PACKING-BYTE-ORDER>
        <START-POSITION>50</START-POSITION>
    </I-SIGNAL-TO-I-PDU-MAPPING>
    <I-SIGNAL-TO-I-PDU-MAPPING UUID="xxxxxxxxxxxxxxxxxxxxxxx">
        <SHORT-NAME>Sig01_7</SHORT-NAME>
            <LONG-NAME>
                <L-4 L="EN">Sig01</L-4>
            </LONG-NAME>
        <I-SIGNAL-GROUP-REF DEST="I-SIGNAL-GROUP">/ISignalContainer/INameOfPDU</I-SIGNAL-GROUP-REF>
    </I-SIGNAL-TO-I-PDU-MAPPING>
</I-SIGNAL-TO-PDU-MAPPINGS>
<UNUSED-BIT-PATTERN>0</UNUSED-BIT-PATTERN>
    ...
    ...
ebroecker commented 8 years ago

Thanks for these Infos. I think this a 'Motorola'-Byte-Order-Issue. This is the first arxml which has "MOST-SIGNIFICANT-BYTE-FIRST" signals in. I'll solve this today evening. Seems to be easy, because I did this already for other formats.

ebroecker commented 8 years ago

Byteorder / Startbit should be fixed

nostar commented 8 years ago

That did the trick. Now I have a dbc file with all messages and signals!

Just an FYI: Initially when trying to open the dbc file in vector dbc editor I got an error 'Database: Illegal segmentation description" and the file would not open. I found a message with DLC of 1 byte, 4 signals, that have start positions of 3, 5, 6, 7 in the arxml file. the first 3 are 1 bit signals, the one at start bit 7 is defined as a 0 bit signal, which is an error I'm sure. The generated DBC file had that signal defined at start bit 23, 0 bit length. Apparently vector dbc editor will not open a file with any 0 bit size signals. Changing it to 1 allowed it to open, then just showed the warning in the browser about the start bit of 23 being out of range for a 1 byte message.

ebroecker commented 8 years ago

I close this issue now, because I created the zero-length issue (#28)