ratal / mdfreader

Read Measurement Data Format (MDF) versions 3.x and 4.x file formats in python
Other
169 stars 74 forks source link

Reading Error Using "ChannelList" #153

Closed proedig closed 5 years ago

proedig commented 5 years ago

Python version

'3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]'

Platform information

'Windows-10-10.0.14393-SP0'

Numpy version

'1.14.3'

mdfreader version

mdfreader 2.7.8

Description

Hi! I have problem with reading in mdffiles using the optional argument "channellist".

When I try

data = mdfreader.mdf(file)

there is no problem. All channels are read in. When I try the same file with

data = mdfreader.mdf(file,channelList=['n'])

I get the following output:

_C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py:552: UserWarning: Unexpected error: (<class 'ModuleNotFoundError'>, ModuleNotFoundError("No module named 'dataRead'",), <traceback object at 0x00000258E3768EC8>) warn('Unexpected error: {}'.format(excinfo())) C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py:553: UserWarning: dataRead crashed, back to python data reading warn('dataRead crashed, back to python data reading')

However, there is reasonable output:

_In [29]: data Out[29]: {'time_2': {'unit': 's', 'description': '', 'master': 'time_2', 'masterType': 1, 'data': array([5.60005003e-03, ..., 4.75285915e+03])}, 'n': {'unit': 'rpm', 'description': 'Engine speed', 'master': 'time2', 'masterType': 1, 'data': array([0, ..., 0], dtype=uint16)}}

So this means for me there is no problem with the variable 'n'. Is it a problem with cython? I have installed cython version 0.28.5.

data.MDFVersionNumber = 300

Best wishes

Philip package_list.txt

ratal commented 5 years ago

Hi, When reading whole file content, mdfreader uses efficient numpy methods to read the channels. When a channel list is specified, these numpy methods are not performing anymore, consuming too much memory and computation for only one channel for instance --> so I developped a cython module dataRead that is doing this one by one channel reading. However, this cython module has to be properly compiled for you platform --> not done properly done in your environment, the warning you get. I see you use python3.6 and win10, I think there is wheel available from pip ? Cython could be an issue for some users therefore I implemented a less computationnaly performing fallback in pure python (but still saving memory) --> you can still read your channel.

proedig commented 5 years ago

Hi ratal,

thanks for your reply. I deinstalled cython and then reinstalled it using pip:

(base) C:\Users\proedig>pip install Cython
Collecting Cython
  Downloading https://files.pythonhosted.org/packages/b3/b1/50ae389b0cf0f63e0e988df2162593e1dbfd1807c92a5d90c03c8c249c3f/Cython-0.28.5-cp36-cp36m-win_amd64.whl (2.5MB)
    100% |████████████████████████████████| 2.5MB 2.2MB/s
Installing collected packages: Cython
Successfully installed Cython-0.28.5

(base) C:\Users\proedig>python
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cython
>>>

So, I assume that cython is installed properly on my system. However, I still get the same warning messages as printed above. Do you have an idea what else I can try to get the fast cython read in running?

Best wishes Philip

ratal commented 5 years ago

Hi Philip, What error do you get while installing mdfreader ? From pip (with or without wheel ?) or from source/git with python setup.py install (develop or build_ext could be used) ? Along with cython, you need to have the proper compiler installed, usual pitfall (the one used to compile the python version you use). Nowadays, it seems Microsoft brings directly support with anaconda from visual studio, so you could install everything correctly (anaconda+cython+compiler).

proedig commented 5 years ago

Hi ratal,

installing mdfreader works fine, without errors. I used pip:

(base) D:\Python\MOP4-17005\Modellabweichung>pip install mdfreader
Collecting mdfreader
  Using cached https://files.pythonhosted.org/packages/28/96/db18dffe2b8d8a2d265fb3bc16f307425905364b2c26dc422dde7945c825/mdfreader-2.7.8-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.14 in c:\programdata\anaconda3\lib\site-packages (from mdfreader) (1.14.3)
Requirement already satisfied: lxml in c:\programdata\anaconda3\lib\site-packages (from mdfreader) (4.2.5)
Requirement already satisfied: bitarray in c:\programdata\anaconda3\lib\site-packages (from mdfreader) (0.8.1)
Requirement already satisfied: sympy in c:\programdata\anaconda3\lib\site-packages (from mdfreader) (1.1.1)
Installing collected packages: mdfreader
Successfully installed mdfreader-2.7.8

Now, I have also installed a working compiler, namely Visual Studio Build Tools: https://blogs.msdn.microsoft.com/vcblog/2017/11/02/visual-studio-build-tools-now-include-the-vs2017-and-vs2015-msvc-toolsets

For testing, I ran the "hello world" example from cython (https://cython.readthedocs.io/en/latest/src/quickstart/build.html). After installing Visual Studio Build Tools, cython works fine now. I updated cython to version 0.29.

However, I still get the following messages when running mdfreader together with channellist:

C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py:552: UserWarning: Unexpected error: (<class 'ModuleNotFoundError'>, ModuleNotFoundError("No module named 'dataRead'",), <traceback object at 0x00000227FE0BD408>)
  warn('Unexpected error: {}'.format(exc_info()))
C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py:553: UserWarning: dataRead crashed, back to python data reading
  warn('dataRead crashed, back to python data reading')

So, there seems to be another problem. Do you have any idea what else I can try? It says that the module 'dataRead' is not found. Do I need to add some directory to my source path?

Best wishes Philip

danielhrisca commented 5 years ago

@proedig

did you try the fix from PR #154 ?

proedig commented 5 years ago

did you try the fix from PR #154 ?

Thanks, now it's working!

proedig commented 5 years ago

Hi again!

I think I found a bug when using "channelList". In my MDF-3-file I have a binary signal, which is 0 at the beginning and jumps to 1 after 1083 seconds. When I read in all channels, the data is correctly imported. When I read in only one channel, the signal is loaded in as a flat 0 for all times, which is not the true signal.

mdfdict1 = mdfreader.mdf(file,channelList=[label])
mdfdict2 = mdfreader.mdf(file)

y1 = mdfdict1[label]['data']
t1 = mdfdict1['time_7']['data']

y2 = mdfdict2[label]['data']
t2 = mdfdict2['time_7']['data']

plt.plot(t1,y1,label='with channelList')
plt.plot(t2,y2,label='without channelList')
plt.legend()

This code results in the following plot:

datfile_test

ratal commented 5 years ago

Hi, I detected a bug in dataRead.pyx for int32, nBytes considered was 2 instead of 4. dataRead is used when you use channelList. Can you try to recompile you dataRead.pyx and try again ? (python setup.py develop in place)

proedig commented 5 years ago

Hi ratal, thanks for the fast response. I have just tried with the recent dev-version including your bugfix, but the problem still exists. Please let me know if you need further information.

ratal commented 5 years ago

Hi, Are you sure you are using recompiled .pyx (cython dataRead.pyx to generate new C and then compile with setup.py) ? Otherwise, I will have diffculties to check what is going wrong without file. But maybe a snapshot of the channel description/headers (CN, CC) from mdvalidator would be a good start for me to try reproducing your issue.

proedig commented 5 years ago

Hi, yes, I could find your changes in dataRead.pyx and then recompiled it. There are some warnings regarding the conversion of certain data types. Unfortunately, those warnings are in German. There is another warning at the end which I do not understand because a compiler is installed and working.

(base) D:\Python\mdfreader\mdfreader-dev>python setup.py develop
[<setuptools.extension.Extension('dataRead') at 0x1eb402679b0>]
running develop
running egg_info
writing mdfreader.egg-info\PKG-INFO
writing dependency_links to mdfreader.egg-info\dependency_links.txt
writing entry points to mdfreader.egg-info\entry_points.txt
writing requirements to mdfreader.egg-info\requires.txt
writing top-level names to mdfreader.egg-info\top_level.txt
reading manifest file 'mdfreader.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'docs\build\html'
warning: no files found matching 'mdfreader.pdf' under directory 'docs\build\latex'
warning: no files found matching 'README'
writing manifest file 'mdfreader.egg-info\SOURCES.txt'
running build_ext
building 'dataRead' extension
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\ProgramData\Anaconda3\lib\site-packages\numpy\core\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\cppwinrt" /TcdataRead.c /Fobuild\temp.win-amd64-3.6\Release\dataRead.obj
dataRead.c
c:\programdata\anaconda3\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
dataRead.c(7622): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(7862): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8102): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8342): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8597): warning C4334: "<<": Das Ergebnis der 32-Bit-Verschiebung wurde implizit in 64 Bits konvertiert. (War eine 64-Bit-Verschiebung vorgesehen?)
dataRead.c(8831): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(8917): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8982): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9145): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9231): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9296): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9459): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9545): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9610): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9773): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9859): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9924): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(15515): warning C4244: "Initialisierung": Konvertierung von "double" in "float", möglicher Datenverlust
dataRead.c(15521): warning C4244: "Initialisierung": Konvertierung von "double" in "float", möglicher Datenverlust
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\ProgramData\Anaconda3\libs /LIBPATH:C:\ProgramData\Anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17134.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17134.0\um\x64" /EXPORT:PyInit_dataRead build\temp.win-amd64-3.6\Release\dataRead.obj /OUT:build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib
   Bibliothek "build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib" und Objekt "build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.exp" werden erstellt.
Code wird generiert.
Codegenerierung ist abgeschlossen.
copying build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd ->
Unexpected error: (<class 'SystemExit'>, SystemExit("error: could not delete 'dataRead.cp36-win_amd64.pyd': Zugriff verweigert",), <traceback object at 0x000001EB40543D48>)
running develop
running egg_info
writing mdfreader.egg-info\PKG-INFO
writing dependency_links to mdfreader.egg-info\dependency_links.txt
writing entry points to mdfreader.egg-info\entry_points.txt
writing requirements to mdfreader.egg-info\requires.txt
writing top-level names to mdfreader.egg-info\top_level.txt
reading manifest file 'mdfreader.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'docs\build\html'
warning: no files found matching 'mdfreader.pdf' under directory 'docs\build\latex'
warning: no files found matching 'README'
writing manifest file 'mdfreader.egg-info\SOURCES.txt'
running build_ext
Creating c:\programdata\anaconda3\lib\site-packages\mdfreader.egg-link (link to .)
mdfreader 2.8 is already the active version in easy-install.pth
Installing mdfconverter-script.py script to C:\ProgramData\Anaconda3\Scripts
Installing mdfconverter.exe script to C:\ProgramData\Anaconda3\Scripts

Installed d:\python\mdfreader\mdfreader-dev
Processing dependencies for mdfreader==2.8
Searching for bitarray==0.8.1
Best match: bitarray 0.8.1
Adding bitarray 0.8.1 to easy-install.pth file

Using c:\programdata\anaconda3\lib\site-packages
Searching for lxml==4.2.5
Best match: lxml 4.2.5
Adding lxml 4.2.5 to easy-install.pth file

Using c:\programdata\anaconda3\lib\site-packages
Searching for sympy==1.1.1
Best match: sympy 1.1.1
Adding sympy 1.1.1 to easy-install.pth file

Using c:\programdata\anaconda3\lib\site-packages
Searching for numpy==1.14.3
Best match: numpy 1.14.3
Adding numpy 1.14.3 to easy-install.pth file

Using c:\programdata\anaconda3\lib\site-packages
Finished processing dependencies for mdfreader==2.8
setup.py:121: UserWarning: It is strongly advised to install Cython along with compilation environment for performance and robustness purpose
  warn('It is strongly advised to install Cython along with compilation environment '
proedig commented 5 years ago

But maybe a snapshot of the channel description/headers (CN, CC) from mdvalidator would be a good start for me to try reproducing your issue.

Data type of numpy-array created by mdfreader is dtype=uint8

The header of my mdf file says:

image

And these are the information for my particular signal:

mdf_validator

ratal commented 5 years ago

The bug I fixed should not have improved the situation. However, looking at your setup trace, it seems you are using the python fallback of dataRead, there should be a bug there I will investigate in the case of a 1 bit signal. In the mean time, ready at the trace that dataRead can not be overwritten : Unexpected error: (<class 'SystemExit'>, SystemExit("error: could not delete 'dataRead.cp36-win_amd64.pyd': Zugriff verweigert",), <traceback object at 0x000001EB40543D48>) You should remove all build part to be regenerated with correct access right. I think using dataRead should solve your issue.

proedig commented 5 years ago

Ok, I uninstalled mdfreader by execution of pip uninstall mdfreader and deleted the old mdfreader directory. In contrast to last time, I opened a command window as administrator to have full access rights. I then reinstalled the latest development version of mdfreader:

(base) D:\Python\mdfreader\mdfreader-dev>python setup.py develop
[<setuptools.extension.Extension('dataRead') at 0x2214dd73c18>]
running develop
running egg_info
creating mdfreader.egg-info
writing mdfreader.egg-info\PKG-INFO
writing dependency_links to mdfreader.egg-info\dependency_links.txt
writing entry points to mdfreader.egg-info\entry_points.txt
writing requirements to mdfreader.egg-info\requires.txt
writing top-level names to mdfreader.egg-info\top_level.txt
writing manifest file 'mdfreader.egg-info\SOURCES.txt'
reading manifest file 'mdfreader.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'docs\build\html'
warning: no files found matching 'mdfreader.pdf' under directory 'docs\build\latex'
warning: no files found matching 'README'
writing manifest file 'mdfreader.egg-info\SOURCES.txt'
running build_ext
building 'dataRead' extension
creating build
creating build\temp.win-amd64-3.6
creating build\temp.win-amd64-3.6\Release
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\ProgramData\Anaconda3\lib\site-packages\numpy\core\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\cppwinrt" /TcdataRead.c /Fobuild\temp.win-amd64-3.6\Release\dataRead.obj
dataRead.c
c:\programdata\anaconda3\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
dataRead.c(7565): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(7805): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8045): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8285): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8540): warning C4334: "<<": Das Ergebnis der 32-Bit-Verschiebung wurde implizit in 64 Bits konvertiert. (War eine 64-Bit-Verschiebung vorgesehen?)
dataRead.c(8777): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(8863): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8928): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9091): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9177): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9242): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9405): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9491): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9556): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9719): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9805): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9870): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(15360): warning C4244: "Initialisierung": Konvertierung von "double" in "float", möglicher Datenverlust
dataRead.c(15366): warning C4244: "Initialisierung": Konvertierung von "double" in "float", möglicher Datenverlust
creating D:\Python\mdfreader\mdfreader-dev\build\lib.win-amd64-3.6
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\ProgramData\Anaconda3\libs /LIBPATH:C:\ProgramData\Anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17134.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17134.0\um\x64" /EXPORT:PyInit_dataRead build\temp.win-amd64-3.6\Release\dataRead.obj /OUT:build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib
   Bibliothek "build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib" und Objekt "build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.exp" werden erstellt.
Code wird generiert.
Codegenerierung ist abgeschlossen.
copying build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd ->
Creating c:\programdata\anaconda3\lib\site-packages\mdfreader.egg-link (link to .)
Adding mdfreader 2.8 to easy-install.pth file
Installing mdfconverter-script.py script to C:\ProgramData\Anaconda3\Scripts
Installing mdfconverter.exe script to C:\ProgramData\Anaconda3\Scripts

Installed d:\python\mdfreader\mdfreader-dev
Processing dependencies for mdfreader==2.8
Searching for lxml==4.2.5
Best match: lxml 4.2.5
Adding lxml 4.2.5 to easy-install.pth file

Using c:\programdata\anaconda3\lib\site-packages
Searching for sympy==1.1.1
Best match: sympy 1.1.1
Adding sympy 1.1.1 to easy-install.pth file

Using c:\programdata\anaconda3\lib\site-packages
Searching for numpy==1.14.3
Best match: numpy 1.14.3
Adding numpy 1.14.3 to easy-install.pth file

Using c:\programdata\anaconda3\lib\site-packages
Finished processing dependencies for mdfreader==2.8

The error message you refered to is not printed anymore. However, the binary signal which I would like to analyze is still incorrectly displayed. There is no warning message which says that I am falling back to the python version of dataRead (there is no warning message at all).

proedig commented 5 years ago

I noticed that the same bug also occurs when using noDataLoading=True:

mdfdict2=mdfreader.mdf(file)
mdfdict3=mdfreader.mdf(file,noDataLoading=True)

y2 = mdfdict2[label]['data']
y3 = mdfdict3.getChannelData(label)

plt.plot(y2,label='noDataLoading=False')
plt.plot(y3,label='noDataLoading=True')
plt.legend()

datfile_test2

ratal commented 5 years ago

Good for the setup. It means that dataRead or fallback, you still have same error as noDataLoading=True will trigger dataRead or fallback usage. But as these 2 are different codes, it is unlikely there is error here, I would rather look at the input argument of dataRead and check if it fits with MDFValidator description. In readSortedRecord, can you tell what are dataRead arguments for this channel when using channelList ? Looking at your screenshots, I think it is first time I see such long data, cn_val_sbit=2799 ! Data is 1 bit in a record that has at least 350 bytes... Maybe there is something wrong happening at loadInfo because of this.

proedig commented 5 years ago

Please tell me in which file and which line I should set a breakpoint to check the input arguments.

danielhrisca commented 5 years ago

@proedig would you mind sharing the file via email? I would like to check it against my lib

ratal commented 5 years ago

in mdf3reader.py, record class, readSortedRecord, line 533

proedig commented 5 years ago

@proedig would you mind sharing the file via email? I would like to check it against my lib

Hi Daniel, unfortunately I am not allowed to share this data file with others outside my company. Please let me know if I can provide other useful information about this problem.

proedig commented 5 years ago

in mdf3reader.py, record class, readSortedRecord, line 533

Ok, when executing

mdfdict3=mdfreader.mdf(file,noDataLoading=True)
y3 = mdfdict3.getChannelData(label)

I come along line 533 for two times. Since I have only one label I assume the first time is for the time vector and the second time for the signal values?

These are the input argument of dataRead for the first time:

chan.bitCount = 64 convertDataType3to4[chan.signalDataType] = 4 chan.nativedataFormat = 'f8' nrecord_chunk = 12931 self.CGrecordLength = 351 chan.bitOffset = 0 chan.posByteBeg = 0 chan.nBytes = 8

For the second time:

chan.bitCount = 1 convertDataType3to4[chan.signalDataType] = 0 chan.nativedataFormat = 'u1' nrecord_chunk = 12931 self.CGrecordLength = 351 chan.bitOffset = 7 chan.posByteBeg = 349 chan.nBytes = 1

danielhrisca commented 5 years ago

@proedig

you can use this to scramble the file (randomizes all texts in the file; be sure to use a copy of the original measurement) before sending.

ratal commented 5 years ago

Yes, indeed if master channel data of the channel is not yet loaded, mdfreader load it. At first look: bitcount is ok dataFormat is ok first bit 2799 = 349 (posByteBeg) * 8 + 7 (bitOffset) --> ok nBytes ok for nrecord_chunk and CGrecordLength , I would need to have CG block screenshot corresponding to these channels to confirm (there could be some recordid not considered). Can you send it please ? I will review in detail the dataRead code with these parameters in the meantime

proedig commented 5 years ago

Ok, this is the header of the corresponding channel group for this label:

image

So, the values for nrecord_chunk and self.CGrecordLength seem to be reasonable then.

In additon, this is the header for the corresponding data group:

image

proedig commented 5 years ago

@proedig

you can use this to scramble the file (randomizes all texts in the file; be sure to use a copy of the original measurement) before sending.

thanks for the hint. I will try this out

ratal commented 5 years ago

ok, there is no recordid, not the issue. I reviewed the dataRead Cython code but could not really find any hint. I lately introduced new attribute for channels, bit_masking_needed. Can you check line 959 in mdf4reader.py if it is False (dataRead shoudl already do bit extraction, no need to redo) ?

ratal commented 5 years ago

Most probably found, please confirm. You should change in md3freader: self[id].bit_masking_needed = False into self[rec_chan[id].channelNumber].bit_masking_needed = False Modified in dev branch but be carefull using this branch, I did major refactoring for pep8 compliance, method names and arguments might not match your code anymore.

proedig commented 5 years ago

Thank you very much for your efforts. Now it's working and the signal data is read in correctly.

As you said, the bug was in line 542 in mdf3reader.py. However, in my version I had to change it to self[recChan[id].channelNumber].bit_masking_needed = False. I do not have an object called rec_chan.