adding support for Python code.

[kdschlosser]

OK so here we are, I think this is a better place to continue our conversation. and there may be users that have some suggestions/thoughts that can also chime in if they want.

I want to bring everyone up to speed on what @collin80 and I have discussed.

I have a pretty decent amount of knowledge on the Vector CANdb++ DBC file format. I have updated a Python library called cantools so using it is similiar to the CANdb++ app. the relationships between the objects are pretty close to what you see in the CANdb++ app. I also added support for 103 Vector defined attributes as well. There are some bugs that I also fixed and improved the overall performance.

I have written a few scripts that connect to a CAN interface using the dynamic libraries that typically get provided with the interface drivers.

I also wrote a program that will calculate the timing registers that need to get passed to an interface (using the dynamic library) so the interface will be able to connect to the CAN network. This program is able to calculate the registers for all of the major CAN chipset manufacturers and it can calculate the registers using only the target bitrate as an input.

These things can be ported from Python to C code, doing this would require a whole lot of work. I raised the question of leaving the python code as is and making a bridge instead. @collin80 's response was this is something that he has thought about and was interested in doing. This can be done really easily using an exported C function and 2 structures. The exported C function would get called from the python code and a python function (wrapped to look like a C function) would get passed to it. This callback would get used when wanting to carry out a task that is in Python code. one of the structures would hold the information on what needs to be carried out and any information that may be needed to carry out that task. and the other is for returned data if any. It would be really easy to expand onto it in the future if needs be, just add more fields to the structures and the relevant code needed to access those fields on either end.

When Savvy CAN starts it simply has to instruct the OS to run the Python code. On windows this would be in the form of an executable and the Python interpreter would be compiled into that executable along with the Python code that needs to be run. on posix systems Python usually comes preinstalled and Savvy CAN would simply run python passing the script as an argument.

This is the most stable type of a connection that can be made and it is also one of the easiest. I am able to write the code needed to do the data type conversions on the python side of things There isn't anything that would need to be done on the Savvy CAN end of things in order for it to handle the data types coming over the connection. the conversion that is done on the Python end is proper C data types.

I can hammer out the Python end of things pretty quickly someone that knows their way around the Savvy CAN code is what is needed. come up with a game plan starting where the 2 pieces connect and work outwards from there.

I am suggesting this kind of a connection because it is far quicker and easier to develop in Python then it is in C. Python is more dynamic in nature, Scripts for additional interfaces can easily be added simply by dropping a script that is written for an interface into a "script" directory and it will not need to be compiled prior to it being added and code changes to the script can be made while Savvy CAN is running, you would just have to instruct it to reload the script. it makes development a whole lot faster and also adds the ability for a user to easily add support for an interface that is not supported. The interface script would define a class that is a subclass of an "Interface" class and it would override the methods of the parent class. the methods would be read, write, available, start, stop, etc... those kinds of things.

This would be a really cool thing when it is running and would provide an easy way to add support for new interfaces. the DBC file reader and writer can also be added in the same manner if wanted.

[kdschlosser]

I just updated the Vector DBC file parser. I added an example script and also a DBC file I made that encodes OBDII requests and decodes the responses.

https://github.com/kdschlosser/vector_dbc

I did want to mention that the dbc parser does properly handle multiplexed messages/signals.. and You can see that in action in the example using the OBDII dbc file.

[MatinF]

Hi Kevin,

I tried loading a log file with OBD2 data responses (e.g. for speed, RPM etc) as well as your OBD2 DBC file, but I'm unable to display physical values when using the interpret frames. Are you making some specific setting changes to make it do so?

In comparison, loading J1939 data with a J1939 DBC yields physical values next to each signal:

Maybe I'm missing something basic.

best, Martin

[kdschlosser]

I believe that Savvy CAN doesn't correctly support multiplexed messages/signals currently. If you shoot me over a copy of your log I can test it with my Python library and see if it properly decodes the frames.

It looks like the key component that piece that is multiplexed is what is missing, you see the pid: IntakeManiAbsPress.. there should also be an IntakeManiAbsPress: value. This is the piece that is multiplexed. The pid is supposed to redirect to a signal named IntakeManiAbsPress and that signal is what does the decoding. Savvy CAN is not making the redirection.

[MatinF]

Hi again,

Once our latest PR is supported to SavvyCAN, you can load our plugin below into the plugins folder and then load the attached OBD2 log file.

best, Martin

clx000plugin.zip

31CB1F25_00000022_00000002-57614CEA891D1DF3651E4328D5EEE37E043458B4CC99E380FCA4A8DD835AC139_CAN_1.txt

[collin80]

It does support multiplexed signals. But, the "correctly" part is likely the issue. I know for a fact that several multiplexed signals from Tesla Model S and Model 3 DBC files are working. However, I can't say that I had any sort of certainty while writing the DBC decoder so I probably have made some wrong assumptions and/or not handled everything properly. So, I'll have to look at Kevin's parser to help me to figure out where mine went wrong.

[kdschlosser]

here is the output..

log_output.txt

There is an issue with the IntakeManiAbsPress signal. all the rest of them appear to be working as they should.

[kdschlosser]

@collin80

I said that only because of #268 and #271.. so there are some glitches in the multiplexed portion of the dbc parser, I just didn't know to what extent. From #271 it seems like there is an issue with the signal mapping.

[kdschlosser]

I fixed the issues in the OBDII.dbc file.

Here is your log output. I did have it do unit conversions just as an example of what the program can do.

log_output.txt

[kdschlosser]

@collin80

If you have any questions please fire away,... The parser is pretty confusing with how it operates and this is one of the things I am trying to correct. It's a pa to make changes because I have to change/move something and then test it. make any changes needed if it is not working right and then test it again. you know the drill.

I want to move the parsing into the classes that represent the the dbc objects this will make it much easier to understand and also easier to hunt down bugs. right now the parsing is dumped into a single 2000 or so line file with functions nested inside other functions . you can go cross eyed looking at it.

It was really confusing to look at before I cleaned it up some. originally there were function parameters that had the same name as variables in the same scope as the function.. and there were 5 layer deep nested for loops, It was really hard to follow. I have cleaned up quit a bit of it, but there is still a lot more to be done.

Moving the parsing to the classes that represent the dbc objects is what is going to enable me to merge files together.

I did want to let you know that anything you see that is wrapped in [ ]'s and there is a for loop inside of that bracket, it runs like a typical "for" iteration except it gets run in C code using the python back end, the iteration and processing of whatever is in that loop gets processed some 200 times faster. the pythonic term for it is "list comprehension" if you didn't know this.

I do not know what you level of knowledge is with Python, I would imagine probably similiar to my knowledge with C/CPP.. enough to get you into trouble.

[kdschlosser]

I just pushed another code change. I got the loading of the OBDII.dbc file down to 2 seconds which isn't that bad considering there are 896 signals and the file is 2,491 lines long.

To write the OBDII data to a new file takes 0.45 seconds, This is great, it is also where I spent some time to optimize the code. I just started the optimization for the parsing of the file. The 2 seconds is only the first round of changes