search

danielhrisca / asammdf

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files
GNU Lesser General Public License v3.0
625 stars 221 forks source link

ASAMMDF GUI v7.1.1 GPS Map Not Displaying #751

Closed fcol95 closed 10 months ago

fcol95 commented 2 years ago

Python version

('python=3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 ' 'bit (AMD64)]') 'os=Windows-10-10.0.19044-SP0' 'numpy=1.23.1' 'asammdf=7.1.1'

MDF version

v4.11

Description

Have a CAN log with GPS latitude and longitude signals. When trying to create a GPS window in the GUI, the leaflet maps doesn't seem to load up and display. See below the error message showing up in the console in the first screenshot. See the latitude and longitude signals trying to be displayed on the map but on a plot in the second screenshot.

image

image

MatinF commented 2 years ago

I can confirm observing this in the 7.1.2.dev4 version:

image

allaboutjeeps commented 2 years ago

Confirmed. Seeing the same issue:

asammdf 7.1.1

GPS_no_map

danielhrisca commented 1 year ago

There seems to be a bug if PySide6==6.3.1 is used. I've installed PySide6==6.2.0 and the maps are loading correctly

MatinF commented 1 year ago

Correct Daniel, this solves it on my end with the latest dev version and forcing PySide6==6.2.0. I believe this issue can be closed.

MatinF commented 1 year ago

Hi again,

After some more digging, I have found out that using PySide6==6.2.0 indeed solves the GPS plot issue.

However, using that version of PySide6 also unfortunately causes the Bus Logging tabular display and CAN bus trace views to "crash" if I click a cell/row, as mentioned in another issue.

In contrast, using PySide==6.3.1 fixes the crash in the Bus Logging trace/tabular view - but does not work with the GPS plots.

Not sure why PySide==6.2.0 causes the crash in the checksum.

danielhrisca commented 1 year ago

@MartinF PySide6 6.2.4 is also available. Do you get the crash with this version as well?

MatinF commented 1 year ago

I don't get the crash with 6.2.4, but with that version the GPS map does not work unfortunately.

Below are my current dependencies related to this:

PyQt5==5.15.4
PyQt5-Qt5==5.15.2
PyQt5-sip==12.9.0
pyqtgraph==0.12.4
pyqtlet==0.3.3
pyqtlet2==0.8.0
PyQtWebEngine==5.15.6
PyQtWebEngine-Qt5==5.15.2
pyserial==3.5
PySide6==6.2.4
pytest==6.2.5
python-dateutil==2.8.2
pytz==2022.1
pyvat==1.3.15
pywin32==303
QtPy==2.1.0
MatinF commented 1 year ago

Here is the error output by the way:

qt.webenginecontext:

GLImplementation: desktop
Surface Type: OpenGL
Surface Profile: CompatibilityProfile
Surface Version: 4.5
Using Default SG Backend: yes
Using Software Dynamic GL: no
Using Angle: no

Init Parameters:
  *  application-name py-asammdf
  *  browser-subprocess-path C:\users\marti\appdata\local\programs\python\python38\lib\site-packages\PySide6\.\QtWebEngineProcess.exe
  *  create-default-gl-context
  *  disable-es3-gl-context
  *  disable-features DnsOverHttpsUpgrade,ConsolidatedMovementXY,InstalledApp,BackgroundFetch,WebOTP,WebPayments,WebUSB,PictureInPicture
  *  disable-speech-api
  *  enable-features NetworkServiceInProcess,TracingServiceInProcess
  *  enable-threaded-compositing
  *  in-process-gpu
  *  use-gl desktop

js: Uncaught TypeError: Cannot read property 'mapObject' of null
js: Uncaught TypeError: Cannot read property 'l2Object' of null
MatinF commented 1 year ago

Some additional inputs:

When using PySide6==6.2.0, I only seem to encounter the 'crash' when using the 'Bus Logging' mode, i.e. reviewing the raw data. Here, the crash occurs both in the 'CAN bus trace' view and in the 'tabular display' of individual Channel Groups.

However, if I DBC decode the data and then try to use the 'tabular display' to show data for a Channel Group, I can select cells without causing the crash.

This leads me to believe the issue with 6.2.0 is isolated to the way Bus Logging data is handled. Maybe this can further help identify how it might be solved with that version?

MatinF commented 1 year ago

One additional update on this:

After some more testing, it appears the issue is not necessarily linked to Bus Logging vs. Decoded data. Rather, it appears to be an issue related to the size of the file.

If I wait for a very extended period of time, I am able to select cells across any files. I.e. to clarify, there is not a "crash" per se, but rather some very heavy and extended processing that appears to take place when I click a cell for the first time. The processing time is linked to the amount of data in the Channel Group, which also explains why the issue may at first appear more prevalent for Bus Logging (where all CAN data is in the same Channel Group), vs. e.g. a decoded CAN ID Channel Group.

In short, the "bug" appears to be some excessive processing that is initiated when a cell is clicked while using PySide6==6.2.0. And as mentioned, this processing is either much faster or non-existent with 6.3.1 for some reason.

danielhrisca commented 1 year ago

@MatinF can you send the file that shows this delay?

MatinF commented 1 year ago

@danielhrisca Please see attached below examples.

To clarify, this is not related to a specific MF4 file - it happens across all our log files (unfinalized, finalized). The 'stalling' is directly related to the file size and my hypothesis is that there is some 'unintended' refresh/reload of the entire dataset when a cell is clicked. What is surprising, perhaps, is that this "refresh issue" takes far longer than loading the file and loading the CAN bus trace.

With the small file fw-010605-demo.MF4 (0.3 MB), it takes ~1 second to load the CAN bus trace, while "clicking a cell" causes a stall for ~6 seconds.

With the larger file, j1939-example.mf4 (~5 MB), it takes ~1-3 seconds to load the CAN bus trace, while "clicking a cell" causes a delay of 200+ seconds.

During the stalling, asammdf is competely unresponsive and my PC sounds as if it's doing some intensive work.

data.zip

MatinF commented 1 year ago

Also, if there is any console print output I can provide that may help clarify this let me know. The default console does not seem to yield info directly related to the clicking of a cell.

best, Martin

MatinF commented 1 year ago

One extra update: It appears that after the "loading/stalling" is done, I am able to freely select cells for a while. But if I change tabs (e.g. to my browser and back again), it seemingly has to again go through the "loading" process, causing the stalling again.

MatinF commented 1 year ago

It also looks like the "loading" is CPU-intensive only, not RAM intensive: image

danielhrisca commented 1 year ago

I am unable to replicate your problem. What specs does your PC have? can_trace

MatinF commented 1 year ago

I've added them below. My guess is maybe that there is some dependency issue in that case. Are you able to share a pip freeze of your dependencies? Then I can try in a venv to see if I can test it there.

image

danielhrisca commented 1 year ago 
alabaster==0.7.12
altgraph==0.17.2
arrow==1.2.2
astroid==2.11.7
asttokens==2.0.5
atomicwrites==1.4.1
attrs==21.4.0
autopep8==1.6.0
Babel==2.10.3
backcall==0.2.0
bcrypt==3.2.2
beautifulsoup4==4.11.1
binaryornot==0.4.4
bitstruct==8.15.1
black==22.8.0
bleach==5.0.1
canmatrix==0.9.5
cchardet==2.1.5
certifi==2022.6.15
cffi==1.15.1
chardet==5.0.0
charset-normalizer==2.1.0
click==8.1.3
cloudpickle==2.1.0
clr-loader==0.1.7
colorama==0.4.5
construct==2.10.68
cookiecutter==2.1.1
coverage==6.4.2
cramjam==2.5.0
cryptography==37.0.4
Cython==0.29.30
debugpy==1.6.2
decorator==5.1.1
defusedxml==0.7.1
diff-match-patch==20200713
dill==0.3.5.1
docutils==0.19
entrypoints==0.4
et-xmlfile==1.1.0
executing==0.9.1
fastjsonschema==2.16.1
fastparquet==0.8.1
flake8==4.0.1
fsspec==2022.5.0
future==0.18.2
h5py==3.7.0
hdf5storage==0.1.18
idna==3.3
imagesize==1.4.1
inflection==0.5.1
intervaltree==3.1.0
ipykernel==6.15.1
ipython==7.34.0
ipython-genutils==0.2.0
isal==0.11.1
isort==5.10.1
jedi==0.18.1
jellyfish==0.9.0
Jinja2==3.1.2
jinja2-time==0.2.0
jsonschema==4.7.2
jupyter-client==7.3.4
jupyter-core==4.11.1
jupyterlab-pygments==0.2.2
keyring==23.7.0
lark==1.1.2
lazy-object-proxy==1.7.1
ldfparser==0.14.0
lxml==4.9.1
lz4==4.0.2
Mako==1.2.1
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mccabe==0.6.1
mdfreader==4.1
mistune==0.8.4
mpmath==1.2.1
mypy-extensions==0.4.3
natsort==8.1.0
nbclient==0.6.6
nbconvert==6.5.0
nbformat==5.4.0
nest-asyncio==1.5.5
nidaqmx==0.6.3
numexpr==2.8.3
numpy==1.23.1
numpydoc==1.4.0
objprint==0.2.2
openpyxl==3.0.10
packaging==21.3
palanteer==0.6
pandas==1.4.3
pandocfilters==1.5.0
paramiko==2.11.0
parso==0.8.3
pathspec==0.9.0
pefile==2022.5.30
pexpect==4.8.0
pickleshare==0.7.5
pip-licenses==3.5.4
platformdirs==2.5.2
pluggy==1.0.0
prompt-toolkit==3.0.30
psutil==5.9.1
PTable==0.9.2
ptyprocess==0.7.0
pure-eval==0.2.2
py-spy==0.3.12
pyaml==21.10.1
pybind11==2.10.0
pycodestyle==2.8.0
pycparser==2.21
pydocstyle==6.1.1
pyflakes==2.4.0
Pygments==2.12.0
pyinstaller==5.2
pyinstaller-hooks-contrib==2022.8
pyinstrument==4.2.0
pylint==2.14.5
pyls-spyder==0.4.0
PyNaCl==1.5.0
pyparsing==3.0.9
pypiwin32==223
PyQt5==5.15.7
PyQt5-Qt5==5.15.2
PyQt5-sip==12.11.0
pyqtgraph==0.12.4
pyqtlet==0.3.3
pyqtlet2==0.8.0
PyQtWebEngine==5.15.6
PyQtWebEngine-Qt5==5.15.2
pyrsistent==0.18.1
pyserial==3.5
PySide6==6.2.0
python-can==4.0.0
python-dateutil==2.8.2
python-lsp-black==1.2.1
python-lsp-jsonrpc==1.0.0
python-lsp-server==1.5.0
python-slugify==6.1.2
python-snappy==0.6.1
pythonnet==3.0.0rc4
pytoolconfig==1.2.2
pytz==2022.1
pyusb==1.2.1
pywin32==304
pywin32-ctypes==0.2.0
PyYAML==6.0
pyzmq==23.2.0
QDarkStyle==3.0.3
qstylizer==0.2.1
QtAwesome==1.1.1
qtconsole==5.3.1
QtPy==2.1.0
requests==2.28.1
rope==1.3.0
Rtree==1.0.0
scipy==1.9.0
setuptools-cpp==0.1.0
shiboken6==6.3.0
six==1.16.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
spark-parser==1.8.9
Sphinx==5.1.1
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
spyder==5.3.2
spyder-kernels==2.3.2
stack-data==0.3.0
sympy==1.10.1
text-unidecode==1.3
textdistance==4.3.0
three-merge==0.1.1
tinycss2==1.1.1
toml==0.10.2
tomli==2.0.1
tomlkit==0.11.1
tornado==6.2
traitlets==5.3.0
typing_extensions==4.3.0
udsoncan==1.14
ujson==5.4.0
uncompyle6 @ https://github.com/rocky/python-uncompyle6/archive/master.zip
uptime==3.0.1
urllib3==1.26.10
viztracer==0.15.3
watchdog==2.1.9
wcwidth==0.2.5
webencodings==0.5.1
whatthepatch==1.0.2
windows-curses==2.3.0
wrapt==1.14.1
xdis==6.0.4
xlrd==2.0.1
XlsxWriter==3.0.3
xlwt==1.3.0
yapf==0.32.0
yappi==1.3.6
MatinF commented 1 year ago

So I tried the below steps: 1) Create a new virtual environment 2) Run pip install asammdf[gui] 3) Run pip uninstall asammdf 4) Run pip install git+https://github.com/danielhrisca/asammdf.git@development 5) Open the GUI via env/scripts/asammdf.exe

In this setup, the "cell loading" issue does not exist, but the GPS plot also does not work. This is expected as the default dependency requirement for asammdf is as per below:

Pip freeze for default installation in virtual environment

asammdf @ git+https://github.com/danielhrisca/asammdf.git@7d6ca57ad92756c69f6e7107c63e465b00d47811
attrs==22.1.0
canmatrix==0.9.5
click==8.1.3
colorama==0.4.5
future==0.18.2
isal==1.0.1
lxml==4.9.1
lz4==4.0.2
natsort==8.2.0
numexpr==2.8.3
numpy==1.23.3
packaging==21.3
pandas==1.4.4
psutil==5.9.2
pyparsing==3.0.9
pyqtgraph==0.12.4
pyqtlet2==0.8.0
PySide6==6.3.2
PySide6-Addons==6.3.2
PySide6-Essentials==6.3.2
python-dateutil==2.8.2
pytz==2022.2.1
QtPy==2.2.0
scipy==1.7.3
shiboken6==6.3.2
six==1.16.0
typing-extensions==4.3.0

6) After this I tried changing my PySide6 installation as below:

pip uninstall pyside6-addons
pip uninstall pyside6-essentials
pip uninstall pyside6
pip install PySide6==6.2.0

7) After doing the above, I now have again the issue as before - i.e. loading time when clicking a cell, but the GPS plot works.

Pip freeze after changing to PySide6==6.2.0

asammdf @ git+https://github.com/danielhrisca/asammdf.git@7d6ca57ad92756c69f6e7107c63e465b00d47811
attrs==22.1.0
canmatrix==0.9.5
click==8.1.3
colorama==0.4.5
future==0.18.2
isal==1.0.1
lxml==4.9.1
lz4==4.0.2
natsort==8.2.0
numexpr==2.8.3
numpy==1.23.3
packaging==21.3
pandas==1.4.4
psutil==5.9.2
pyparsing==3.0.9
pyqtgraph==0.12.4
pyqtlet2==0.8.0
PySide6==6.2.0
python-dateutil==2.8.2
pytz==2022.2.1
QtPy==2.2.0
scipy==1.7.3
shiboken6==6.2.0
six==1.16.0
typing-extensions==4.3.0

Other things I tried

The issue seems to be linked to PySide6, as well as potentially PySide6-Addons, PySide6-Essentials and shiboken6. See also below comparison that isolates the changes between a setup where the cell-loading is fixed (but no GPS map), vs one where the cell loading is not fixed (but the GPS map works):

Cell loading fixed (but no GPS map)

PySide6==6.3.2
PySide6-Addons==6.3.2
PySide6-Essentials==6.3.2
shiboken6==6.3.2

Cell loading not fixed (but GPS map works)

PySide6==6.2.3

shiboken6==6.2.3
danielhrisca commented 1 year ago

I'm sorry Martin, I get no slowdown with a new environment and this pip freeze:

attrs==22.1.0
canmatrix==0.9.5
click==8.1.3
colorama==0.4.5
future==0.18.2
isal==1.0.1
lxml==4.9.1
lz4==4.0.2
natsort==8.2.0
numexpr==2.8.3
numpy==1.23.3
packaging==21.3
pandas==1.4.4
psutil==5.9.2
pyparsing==3.0.9
pyqtgraph==0.12.4
pyqtlet2==0.8.0
PySide6==6.2.0
python-dateutil==2.8.2
pytz==2022.2.1
QtPy==2.2.0
shiboken6==6.2.0
six==1.16.0
typing_extensions==4.3.0
MatinF commented 1 year ago

Hi again Daniel,

can you let me know what OS environment you're testing this on? Maybe we can replicate your setup on our end on a Linux PC and see if we then find an issue or not.

danielhrisca commented 1 year ago

Windows 10 x64, cpython 3.10 x64

MatinF commented 1 year ago

Hi again Daniel,

So I upgraded from Python 3.8 to Python 3.10 (x64, like my Windows 10 x64), installed via the Windows Installer from here: https://www.python.org/downloads/windows/

I then did a virtual environment and installed your dependencies (and double checked that they match exactly yours). But the issue persists. I've recorded a video of it below and attached the sample file I'm loading.

https://user-images.githubusercontent.com/26184427/189976608-b9a18042-f934-477c-982e-6ba2f8b04c0c.mp4

test-file.zip

I simply do not understand why this would differ.

MatinF commented 1 year ago

As an additional observation: In my virtual environment, I'm able to install asammdf 7.0.7, which I believe is the "latest" version that allows me to both click around cells without issues in the tabular display or CAN bus trace - while also supporting the GPS map. All works with below dependencies:

asammdf==7.0.7
attrs==22.1.0
canmatrix==0.9.5
click==8.1.3
colorama==0.4.5
future==0.18.2
isal==1.0.1
lxml==4.9.1
lz4==4.0.2
natsort==8.2.0
numexpr==2.8.3
numpy==1.23.3
packaging==21.3
pandas==1.4.4
psutil==5.9.2
pyparsing==3.0.9
PyQt5==5.15.7
PyQt5-Qt5==5.15.2
PyQt5-sip==12.11.0
pyqtgraph==0.12.4
pyqtlet==0.3.3
pyqtlet2==0.8.0
PyQtWebEngine==5.15.6
PyQtWebEngine-Qt5==5.15.2
PySide6==6.2.0
python-dateutil==2.8.2
pytz==2022.2.1
QtPy==2.2.0
scipy==1.9.1
shiboken6==6.2.0
six==1.16.0
typing_extensions==4.3.0

However, we would of course want to get it working also with the more recent asammdf releases.

If I use the above dependencies and update to asammdf 1.0.1, it reintroduces the issue with clicking cells causing loading.

MatinF commented 1 year ago

Another from my team tested this and on his PC it works OK. So it may be something specific to my PC, though I have no idea what it might be ..

MatinF commented 1 year ago

Suggest closing this as it appears to be something specific to my PC

danielhrisca commented 10 months ago

With the latest release 7.3.16 running pip install asammdf[gui] will install the correct pinned version and multiple GPS windows are working