Export encoding vs. comment encoding?

[altendky]

I am working over in altendky/canmatrix with a couple short term goals.

• Support Python3
• Support at least importing of PEAK/PCAN Explorer .sym files

In the context of the Python3 situation, encode() now returns bytes rather than a string which write() doesn't like. I noticed that if there were only a single encoding specified then it could simply be specified via open() rather than encoding each line individually. But, there are two encodings specified.

In what situation would it make sense to encode the content and comments differently and does that really result in a valid file?

Thanks for any explanation you can provide.

[ebroecker]

well - first of all I am happy to see anybody using this code.

Python3-support is a very good idea, which was planed far in future :+1:

About different encodings in dbc: I don't think this makes any sense, but the support came in, because some days ago a user with a chinese dbc had different encodings for comments and the remaining file. The dbcs I saw had only one encoding per file. While editing a dbc with Vector-Tools usually only one decoding is in the file.

Peak-Sym-File: Do you have any description for the .sys-file-format?

[altendky]

Python3 'support' by way of futurize was easy after the indentation and UTF-8 cleanup (20539dd4773f2f917f87370fc8577a15a016447f, see command line in commit message), though I haven't really tested it. I have a preliminary parser in place and was able to re-export to .dbc the basic non-multiplexor messages. I expect my parser will get committed in the next few days. I will also see about getting a .sym file that I can share for your reference. It's basic structure is similar to .ini files plus another layer of grouping above and another layer of details within each variable.

Another point I would expect to do would be to implement a Setuptools configuration for distribution.

Presently I just quickly deleted all the encode() calls and set the single encoding via open(). I will plan to undo that and try to pick a clean way to support multiple encodings in the export within Python 3 so as to maintain that feature.

Overall, I am working on a CAN based service tool to be coded in PyQt5 with python-can. I am exploring CanMatrix for both import and possibly as a base for internal objects holding the data. I haven't gotten too far into it but I imagine inheriting from both the CanMatrix Frame class and the python-can listener to implement a bus interface which is configurable by any of the CanMatrix supported input file types.

[altendky]

The .sym is the PCAN Explorer native file format. The .dbc is exported from PCAN Explorer using their DBC plugin.

AFE_CAN_ID247_FACTORY.dbc.txt AFE_CAN_ID247_FACTORY.sym.txt

[ebroecker]

Thanks a lot for THW examples. I just startend with the sym-importer. You can expect Thema first release oft pcan-importer tomorrow. Export is even more simple to implement maybe first release in Monday...

[altendky]

I was expecting to finish the importer, but thanks! I was looking around and didn't find any references to supporting enumerations (ENUMS). Did I miss the code or is that not implemented yet?

[ebroecker]

the ENUMS are Valuetables in canmatrix. Valuetables are attributes of signals. I should perhaps perhaps spend more time in documentation...

first import-support is done. some issues are open:

• I did not test to import a intel-coded signal
• currently no encoding/decoding is done.

[altendky]

In regard to Python3 support, do you have any test suites setup? I would figure that if you have a set of example files for each format, each one could be translated to all other format types and then used as a reference for the same process with a Python3 version. With a little luck the same code could be made both Python3 and 2 compatible.

Also, would you be interested in a Setuptools friendly directory structure? I don't necessarily know the details of what that means but I need to learn and start setting up my applications and libraries that way anyways.

From the looks of it I will be continuing to use canmatrix but want to make sure the changes I'm looking for are things you are interested in as well.

Just as a note, it looks like the .dbc file that I provided as an example is messed up. It has a 32-bit signal starting at bit 39! Not really plausible unless you have > 8 byte messages... I'll probably explore that next week and see if it's a bug in PCAN Explorer or what.

[ebroecker]

I'm sorry, I have currently no open test-files. I have some good files here, but I can't publish them. I think, it would be a great idea to have some open test-files in various formats. Whereby the key-formats are dbc and arxml which are the standard-formats in the automotive industry. As first attempt I could create a .dbc-file and a .arxml-file with several features and export it to all formats. With these files we could check, if something has changed while switching to python3. In my opinion python2-support can be abandoned if it makes trouble.

Another filestructure is also no problem for me. I definitively have no idea how a setuptools-friendly structure looks like. I think: everything, that makes it more easy to use this tool is good.

your dbc has some problems. Seems, that the export does not support multiplexed signals also. I think, the export of canmatrix is even better

[altendky]

If you would like to have direct contact (rather than abusing this issue thread :] ), my email is listed on my contact page or you can find me on google hangouts. Otherwise, we'll continue here.

I will create a new fork for doing test script development. After this we can worry about Python3 since we will then be able to check for regressions (even if only in private due to concerns about file content).

Just for reference, you can see example Python3 and Setuptools commits in my original fork (altendky/canmatrix), as well as some other cleanup. Since they are either small or automated changes, I will expect to reimplement them after setting up the tests.

[ebroecker]

I close this issue now, because Issue #3 shows why the title of this Issue is a feature someone needs.

All other things in this Issue will be discussed outside this Issue.