Closed MatinF closed 5 years ago
danielhrisca
commented
5 years ago Hello Martin,
please confirm if items 1 to 4 are fixed in development.
Scrambling CAN bus logging is not working since handling CAN bus logging requires specific channel names. There is no work-around for this.
MatinF
commented
5 years ago Hi Daniel,
I tried with the latest GUI version just now, testing with CSV - but I'm getting a "Not Responding" issue when I try to export with any type of settings. I don't see any details in the console on this.
Best, Martin
danielhrisca
commented
5 years ago Hello Martin,
I've accidentally introduced an error. You should use the new code
MatinF
commented
5 years ago Hi again Daniel,
I tested a bit with CSV, I seem to still encounter a couple of issues - I hope the below helps a bit in the troubleshooting. Happy to do further tests!
1) Testing CSV with none of the settings enabled works - but the dialog box stays at 0% for a few seconds, then disappears. This intuitively indicates that the export is done - yet the file is not completed until ~10 seconds more after this. I think this is an issue with the dialog box more than the export.
2) When exporting to CSV as above, the tool currently provides 4 output files for a raw bus logging sample. However, in e.g. the OBD2 sample data we have, only the first channel group (channel group 0) actually contains relevant data. This is correctly shown when e.g. browsing the Channel view - but the CSV export produces a large CSV file for Channel 1 and 3 in our case, which I don't think happened before. Channels 1 and 3 are pseudo channels related to variable length data bytes in channel group 0 and 2 respectively. From an end user perspective the most logical export in the case where there's only data on CAN bus channel 1 would be that the data of Channel Group 0 is exported.
If this part is unclear, please let me know then I'll try to provide a better example - I think this part has worked as intended before.
3) When exporting with Single time base enabled, I get a NoneType error and the resulting CSV is 0 kb.
Could not parse stylesheet of object ChannelGroupInfoDialog(0x1d6604be370) File "site-packages\asammdf\gui\widgets\file.py", line 1376, in export File "site-packages\asammdf\gui\utils.py", line 100, in run_thread_with_progress
<class 'Exception'>: Traceback (most recent call last):
File "site-packages\asammdf\gui\utils.py", line 133, in run
File "site-packages\asammdf\mdf.py", line 1260, in export
File "site-packages\numpy\lib\function_base.py", line 2091, in __call__
File "site-packages\numpy\lib\function_base.py", line 2167, in _vectorize_call
File "site-packages\asammdf\blocks\utils.py", line 1493, in csv_bytearray2hex
AttributeError: 'int' object has no attribute 'tobytes'4) When exporting to CSV with Time as date enabled (but the others disabled), I get a noneType error and no file is produced:
<class 'Exception'>: Traceback (most recent call last):
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\gui\utils.py", line 133, in run
self.output = self._target(*self._args, **self._kwargs)
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\mdf.py", line 1337, in export
pd.to_datetime(df.index + self.header.start_time.timestamp(), unit='s')
UnboundLocalError: local variable 'df' referenced before assignment
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\gui\widgets\file.py", line 1375, in export
progress=progress,
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\gui\utils.py", line 100, in run_thread_with_progress
raise Exception(thr.error)
<class 'Exception'>: Traceback (most recent call last):
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\gui\utils.py", line 133, in run
self.output = self._target(*self._args, **self._kwargs)
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\mdf.py", line 1337, in export
pd.to_datetime(df.index + self.header.start_time.timestamp(), unit='s')
UnboundLocalError: local variable 'df' referenced before assignment4) When exporting to CSV with Time as Date + Single Time Base enabled (but the other disabled), I get a NoneType error and no file is produced:
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\gui\widgets\file.py", line 1375, in export
progress=progress,
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\gui\utils.py", line 100, in run_thread_with_progress
raise Exception(thr.error)
<class 'Exception'>: Traceback (most recent call last):
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\gui\utils.py", line 133, in run
self.output = self._target(*self._args, **self._kwargs)
File "c:\users\marti\appdata\local\programs\python\python37\lib\site-packages\asammdf\mdf.py", line 1337, in export
pd.to_datetime(df.index + self.header.start_time.timestamp(), unit='s')
UnboundLocalError: local variable 'df' referenced before assignmentQuick update:
1) CSV: I tried the latest GUI ("Fix export" commit):
<class 'Exception'>: Traceback (most recent call last):
File "site-packages\asammdf\gui\utils.py", line 133, in run
File "site-packages\asammdf\mdf.py", line 1148, in export
KeyError: 'CAN_DataFrame.CAN_DataFrame.BusChannel'
File "site-packages\asammdf\gui\widgets\file.py", line 1376, in export
File "site-packages\asammdf\gui\utils.py", line 85, in run_thread_with_progress
<class 'TypeError'>: 'NoneType' object is not subscriptable2) Excel: This seems to be removed in the latest version, which I think is a good choice.
4) Parquet: This seems to work now with all setting combinations - I've not scrutinized the data, but just concluded that the files seem to be exported without error.
3) HDF5: This seems to also work in all cases where single time base is not enabled. Enabling Single Time Base seems to result in a NoneType error as below.
File "site-packages\asammdf\gui\widgets\file.py", line 1376, in export
File "site-packages\asammdf\gui\utils.py", line 100, in run_thread_with_progress
<class 'Exception'>: Traceback (most recent call last):
File "site-packages\asammdf\gui\utils.py", line 133, in run
File "site-packages\asammdf\mdf.py", line 1148, in export
KeyError: 'CAN_DataFrame.CAN_DataFrame.BusChannel'Hi Daniel, hope all is well.
I did some more testing on this. When I try to export a DBC converted file, everything seems to work - i.e. all combinations yield the expected result (from what I can see).
As such, I think most of the remaining challenges are specific to the bus logging format.
Prioritization suggestion To prioritize, I think a solution for now could be to do the following changes: 1) When a bus logging MDF is loaded, gray out the "Single time base" export option 2) When a channel group contains no data, exclude it from the export (e.g. CG2 in our sample) 3) When a channel group has a "VLSD" flag, exclude it from the export (e.g. CG1 and CG3 in our sample)
Example of VLSD flag in sample file:
I would assume this would be a simpler fix than trying to find some way of handling Single time base for bus logging data (which I think will be hard). Further, the above would help a ton in simplifying things for users (and potentially speed up the export as the VLSD groups can be skipped entirely).
PS: One observation I made during my tests: When I now convert a raw file with a DBC, the resulting scaled file shows up as if it has no channels available. When I then change from Natural sort to Internal file structure, the channels appear again (and now also appear if I re-select natural sort). It seems to be a specific issue to the case where Natural sort is the default selected option - it does not happen when Internal file structure is used, which is why I've not seen this before. Not a major issue, but just an FYI.
danielhrisca
commented
5 years ago @MatinF
3) HDF5: This seems to also work in all cases where single time base is not enabled. Enabling Single Time Base seems to result in a NoneType error as below.
This does not raise exceptions for me
3. When a channel group has a "VLSD" flag, exclude it from the export (e.g. CG1 and CG3 in our sample)
This is done now more generally by checking if the channel group has channels
I would assume this would be a simpler fix than trying to find some way of handling Single time base for bus logging data (which I think will be hard). Further, the above would help a ton in simplifying things for users (and potentially speed up the export as the VLSD groups can be skipped entirely).
I'm not sure how this would help performance.
MatinF
commented
5 years ago Hi Daniel, thanks for the update - great!
I checked the updated version - the following are fixed: 1) The HDF5 seems to work on my end now as well 2) The VLSD channels are no longer exported, which is nice - I think your more general solution makes sense here
Only minor remaining observations from my end are below:
1) When using Single time base on the raw bus logging file, it yields an odd result where the #rows is double the #cycles. I think the simplest would be the disabling of Single time base for raw bus logging files - but it's not critical for us (we can guide users to not enable this)
2) The export currently still creates a file for a channel group even if the channel group has no actual data. This is probably fairly unique to our case where we do have channels in the group (in case somebody would connect a CAN bus to our bus Channel 2). If an end user has e.g. 2 x CAN bus channels + 2 x LIN bus channels, yet only logs on a single CAN channel, he'll be getting 4 output files instead of 1. Perhaps the export function could determine if the file contains data or not - and if no data is contained, choose not to export it. Again, it's not vital, but just from a perspective of making the export as clean as possible. I've attached an example of the empty output file below: Basic.ChannelGroup_2.zip
The pending issues here are not critical and I think the vast majority of the things have been fixed, which is highly appreciated! :-) As such, I'll also be closing this ticket later today. Thanks again for great work Daniel.
Hi Daniel,
Thanks again for all the great work in getting the latest GUI up and running as an executable as well.
As mentioned, I did some basic tests with the latest GUI version (Windows executable, v1.0.448).
I've listed below some scenarios where I encounter errors - I think most are consistent across both raw bus logging and the converted data, though the error messages listed are taken for the raw bus logging OBD2 data sample: CBC40C04_00000086_00000002.zip
Happy to provide further details where possible!
EXPORT TESTS
1) Export as csv (date as time enabled, other disabled) Note: Same issue for physical values
This leads to a none type error and the following console output:
Note that the error happens also when Time from 0s is enabled - but it does not happen if Single time base is enabled.
3) Export as hdf5 (any setting combination) Gives a nonetype error again, but with below console output:
4) Export as parquet (any setting combination) Same as hdf5, but with below console output:
2) Export as excel (any setting combination) While I don't think Excel export is the most vital (we'd go for CSV for speed anyway), I did notice that it seems to lead to some errors for the raw data. Again, the Nonetype error occurs, this time for any combination.
The following is output in the console:
FILTER TESTS
For the raw OBD2 data, I tried to filter e.g. so as to include only Channel Group 0 and 1 (for the sake of excluding the 2nd bus channel), which could be a practical use case for bus logging data. However, the resulting filtered mdf seems to be empty (no console output, though).
SCRAMBLE TESTS
I get a nonetype error when I try to scramble the text - but maybe this is because scrambling doesn't make a lot of sense for anonymous bus logging data in the first place. Perhaps it would make sense to disable the tab in this case, but not critical.