Closed proedig closed 5 years ago
Hi, When reading whole file content, mdfreader uses efficient numpy methods to read the channels. When a channel list is specified, these numpy methods are not performing anymore, consuming too much memory and computation for only one channel for instance --> so I developped a cython module dataRead that is doing this one by one channel reading. However, this cython module has to be properly compiled for you platform --> not done properly done in your environment, the warning you get. I see you use python3.6 and win10, I think there is wheel available from pip ? Cython could be an issue for some users therefore I implemented a less computationnaly performing fallback in pure python (but still saving memory) --> you can still read your channel.
Hi ratal,
thanks for your reply. I deinstalled cython and then reinstalled it using pip:
(base) C:\Users\proedig>pip install Cython
Collecting Cython
Downloading https://files.pythonhosted.org/packages/b3/b1/50ae389b0cf0f63e0e988df2162593e1dbfd1807c92a5d90c03c8c249c3f/Cython-0.28.5-cp36-cp36m-win_amd64.whl (2.5MB)
100% |████████████████████████████████| 2.5MB 2.2MB/s
Installing collected packages: Cython
Successfully installed Cython-0.28.5
(base) C:\Users\proedig>python
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cython
>>>
So, I assume that cython is installed properly on my system. However, I still get the same warning messages as printed above. Do you have an idea what else I can try to get the fast cython read in running?
Best wishes Philip
Hi Philip,
What error do you get while installing mdfreader ?
From pip (with or without wheel ?) or from source/git with python setup.py install
(develop or build_ext could be used) ?
Along with cython, you need to have the proper compiler installed, usual pitfall (the one used to compile the python version you use). Nowadays, it seems Microsoft brings directly support with anaconda from visual studio, so you could install everything correctly (anaconda+cython+compiler).
Hi ratal,
installing mdfreader works fine, without errors. I used pip:
(base) D:\Python\MOP4-17005\Modellabweichung>pip install mdfreader
Collecting mdfreader
Using cached https://files.pythonhosted.org/packages/28/96/db18dffe2b8d8a2d265fb3bc16f307425905364b2c26dc422dde7945c825/mdfreader-2.7.8-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.14 in c:\programdata\anaconda3\lib\site-packages (from mdfreader) (1.14.3)
Requirement already satisfied: lxml in c:\programdata\anaconda3\lib\site-packages (from mdfreader) (4.2.5)
Requirement already satisfied: bitarray in c:\programdata\anaconda3\lib\site-packages (from mdfreader) (0.8.1)
Requirement already satisfied: sympy in c:\programdata\anaconda3\lib\site-packages (from mdfreader) (1.1.1)
Installing collected packages: mdfreader
Successfully installed mdfreader-2.7.8
Now, I have also installed a working compiler, namely Visual Studio Build Tools: https://blogs.msdn.microsoft.com/vcblog/2017/11/02/visual-studio-build-tools-now-include-the-vs2017-and-vs2015-msvc-toolsets
For testing, I ran the "hello world" example from cython (https://cython.readthedocs.io/en/latest/src/quickstart/build.html). After installing Visual Studio Build Tools, cython works fine now. I updated cython to version 0.29.
However, I still get the following messages when running mdfreader together with channellist:
C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py:552: UserWarning: Unexpected error: (<class 'ModuleNotFoundError'>, ModuleNotFoundError("No module named 'dataRead'",), <traceback object at 0x00000227FE0BD408>)
warn('Unexpected error: {}'.format(exc_info()))
C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py:553: UserWarning: dataRead crashed, back to python data reading
warn('dataRead crashed, back to python data reading')
So, there seems to be another problem. Do you have any idea what else I can try? It says that the module 'dataRead' is not found. Do I need to add some directory to my source path?
Best wishes Philip
@proedig
did you try the fix from PR #154 ?
did you try the fix from PR #154 ?
Thanks, now it's working!
Hi again!
I think I found a bug when using "channelList". In my MDF-3-file I have a binary signal, which is 0 at the beginning and jumps to 1 after 1083 seconds. When I read in all channels, the data is correctly imported. When I read in only one channel, the signal is loaded in as a flat 0 for all times, which is not the true signal.
mdfdict1 = mdfreader.mdf(file,channelList=[label])
mdfdict2 = mdfreader.mdf(file)
y1 = mdfdict1[label]['data']
t1 = mdfdict1['time_7']['data']
y2 = mdfdict2[label]['data']
t2 = mdfdict2['time_7']['data']
plt.plot(t1,y1,label='with channelList')
plt.plot(t2,y2,label='without channelList')
plt.legend()
This code results in the following plot:
Hi,
I detected a bug in dataRead.pyx for int32, nBytes considered was 2 instead of 4. dataRead is used when you use channelList.
Can you try to recompile you dataRead.pyx and try again ? (python setup.py develop
in place)
Hi ratal, thanks for the fast response. I have just tried with the recent dev-version including your bugfix, but the problem still exists. Please let me know if you need further information.
Hi,
Are you sure you are using recompiled .pyx (cython dataRead.pyx
to generate new C and then compile with setup.py) ?
Otherwise, I will have diffculties to check what is going wrong without file. But maybe a snapshot of the channel description/headers (CN, CC) from mdvalidator would be a good start for me to try reproducing your issue.
Hi, yes, I could find your changes in dataRead.pyx and then recompiled it. There are some warnings regarding the conversion of certain data types. Unfortunately, those warnings are in German. There is another warning at the end which I do not understand because a compiler is installed and working.
(base) D:\Python\mdfreader\mdfreader-dev>python setup.py develop
[<setuptools.extension.Extension('dataRead') at 0x1eb402679b0>]
running develop
running egg_info
writing mdfreader.egg-info\PKG-INFO
writing dependency_links to mdfreader.egg-info\dependency_links.txt
writing entry points to mdfreader.egg-info\entry_points.txt
writing requirements to mdfreader.egg-info\requires.txt
writing top-level names to mdfreader.egg-info\top_level.txt
reading manifest file 'mdfreader.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'docs\build\html'
warning: no files found matching 'mdfreader.pdf' under directory 'docs\build\latex'
warning: no files found matching 'README'
writing manifest file 'mdfreader.egg-info\SOURCES.txt'
running build_ext
building 'dataRead' extension
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\ProgramData\Anaconda3\lib\site-packages\numpy\core\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\cppwinrt" /TcdataRead.c /Fobuild\temp.win-amd64-3.6\Release\dataRead.obj
dataRead.c
c:\programdata\anaconda3\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
dataRead.c(7622): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(7862): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8102): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8342): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8597): warning C4334: "<<": Das Ergebnis der 32-Bit-Verschiebung wurde implizit in 64 Bits konvertiert. (War eine 64-Bit-Verschiebung vorgesehen?)
dataRead.c(8831): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(8917): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8982): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9145): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9231): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9296): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9459): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9545): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9610): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9773): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9859): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9924): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(15515): warning C4244: "Initialisierung": Konvertierung von "double" in "float", möglicher Datenverlust
dataRead.c(15521): warning C4244: "Initialisierung": Konvertierung von "double" in "float", möglicher Datenverlust
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\ProgramData\Anaconda3\libs /LIBPATH:C:\ProgramData\Anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17134.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17134.0\um\x64" /EXPORT:PyInit_dataRead build\temp.win-amd64-3.6\Release\dataRead.obj /OUT:build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib
Bibliothek "build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib" und Objekt "build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.exp" werden erstellt.
Code wird generiert.
Codegenerierung ist abgeschlossen.
copying build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd ->
Unexpected error: (<class 'SystemExit'>, SystemExit("error: could not delete 'dataRead.cp36-win_amd64.pyd': Zugriff verweigert",), <traceback object at 0x000001EB40543D48>)
running develop
running egg_info
writing mdfreader.egg-info\PKG-INFO
writing dependency_links to mdfreader.egg-info\dependency_links.txt
writing entry points to mdfreader.egg-info\entry_points.txt
writing requirements to mdfreader.egg-info\requires.txt
writing top-level names to mdfreader.egg-info\top_level.txt
reading manifest file 'mdfreader.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'docs\build\html'
warning: no files found matching 'mdfreader.pdf' under directory 'docs\build\latex'
warning: no files found matching 'README'
writing manifest file 'mdfreader.egg-info\SOURCES.txt'
running build_ext
Creating c:\programdata\anaconda3\lib\site-packages\mdfreader.egg-link (link to .)
mdfreader 2.8 is already the active version in easy-install.pth
Installing mdfconverter-script.py script to C:\ProgramData\Anaconda3\Scripts
Installing mdfconverter.exe script to C:\ProgramData\Anaconda3\Scripts
Installed d:\python\mdfreader\mdfreader-dev
Processing dependencies for mdfreader==2.8
Searching for bitarray==0.8.1
Best match: bitarray 0.8.1
Adding bitarray 0.8.1 to easy-install.pth file
Using c:\programdata\anaconda3\lib\site-packages
Searching for lxml==4.2.5
Best match: lxml 4.2.5
Adding lxml 4.2.5 to easy-install.pth file
Using c:\programdata\anaconda3\lib\site-packages
Searching for sympy==1.1.1
Best match: sympy 1.1.1
Adding sympy 1.1.1 to easy-install.pth file
Using c:\programdata\anaconda3\lib\site-packages
Searching for numpy==1.14.3
Best match: numpy 1.14.3
Adding numpy 1.14.3 to easy-install.pth file
Using c:\programdata\anaconda3\lib\site-packages
Finished processing dependencies for mdfreader==2.8
setup.py:121: UserWarning: It is strongly advised to install Cython along with compilation environment for performance and robustness purpose
warn('It is strongly advised to install Cython along with compilation environment '
But maybe a snapshot of the channel description/headers (CN, CC) from mdvalidator would be a good start for me to try reproducing your issue.
Data type of numpy-array created by mdfreader is dtype=uint8
The header of my mdf file says:
And these are the information for my particular signal:
The bug I fixed should not have improved the situation.
However, looking at your setup trace, it seems you are using the python fallback of dataRead, there should be a bug there I will investigate in the case of a 1 bit signal.
In the mean time, ready at the trace that dataRead can not be overwritten : Unexpected error: (<class 'SystemExit'>, SystemExit("error: could not delete 'dataRead.cp36-win_amd64.pyd': Zugriff verweigert",), <traceback object at 0x000001EB40543D48>)
You should remove all build part to be regenerated with correct access right.
I think using dataRead should solve your issue.
Ok, I uninstalled mdfreader by execution of pip uninstall mdfreader
and deleted the old mdfreader directory. In contrast to last time, I opened a command window as administrator to have full access rights. I then reinstalled the latest development version of mdfreader:
(base) D:\Python\mdfreader\mdfreader-dev>python setup.py develop
[<setuptools.extension.Extension('dataRead') at 0x2214dd73c18>]
running develop
running egg_info
creating mdfreader.egg-info
writing mdfreader.egg-info\PKG-INFO
writing dependency_links to mdfreader.egg-info\dependency_links.txt
writing entry points to mdfreader.egg-info\entry_points.txt
writing requirements to mdfreader.egg-info\requires.txt
writing top-level names to mdfreader.egg-info\top_level.txt
writing manifest file 'mdfreader.egg-info\SOURCES.txt'
reading manifest file 'mdfreader.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'docs\build\html'
warning: no files found matching 'mdfreader.pdf' under directory 'docs\build\latex'
warning: no files found matching 'README'
writing manifest file 'mdfreader.egg-info\SOURCES.txt'
running build_ext
building 'dataRead' extension
creating build
creating build\temp.win-amd64-3.6
creating build\temp.win-amd64-3.6\Release
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\ProgramData\Anaconda3\lib\site-packages\numpy\core\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include -IC:\ProgramData\Anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\cppwinrt" /TcdataRead.c /Fobuild\temp.win-amd64-3.6\Release\dataRead.obj
dataRead.c
c:\programdata\anaconda3\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
dataRead.c(7565): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(7805): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8045): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8285): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8540): warning C4334: "<<": Das Ergebnis der 32-Bit-Verschiebung wurde implizit in 64 Bits konvertiert. (War eine 64-Bit-Verschiebung vorgesehen?)
dataRead.c(8777): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(8863): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(8928): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9091): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9177): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9242): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9405): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9491): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9556): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9719): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(9805): warning C4293: "<<": Umschaltanzahl ist negativ oder zu groß, Verhalten undefiniert.
dataRead.c(9870): warning C4244: "=": Konvertierung von "__int64" in "long", möglicher Datenverlust
dataRead.c(15360): warning C4244: "Initialisierung": Konvertierung von "double" in "float", möglicher Datenverlust
dataRead.c(15366): warning C4244: "Initialisierung": Konvertierung von "double" in "float", möglicher Datenverlust
creating D:\Python\mdfreader\mdfreader-dev\build\lib.win-amd64-3.6
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\ProgramData\Anaconda3\libs /LIBPATH:C:\ProgramData\Anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17134.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17134.0\um\x64" /EXPORT:PyInit_dataRead build\temp.win-amd64-3.6\Release\dataRead.obj /OUT:build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib
Bibliothek "build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib" und Objekt "build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.exp" werden erstellt.
Code wird generiert.
Codegenerierung ist abgeschlossen.
copying build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd ->
Creating c:\programdata\anaconda3\lib\site-packages\mdfreader.egg-link (link to .)
Adding mdfreader 2.8 to easy-install.pth file
Installing mdfconverter-script.py script to C:\ProgramData\Anaconda3\Scripts
Installing mdfconverter.exe script to C:\ProgramData\Anaconda3\Scripts
Installed d:\python\mdfreader\mdfreader-dev
Processing dependencies for mdfreader==2.8
Searching for lxml==4.2.5
Best match: lxml 4.2.5
Adding lxml 4.2.5 to easy-install.pth file
Using c:\programdata\anaconda3\lib\site-packages
Searching for sympy==1.1.1
Best match: sympy 1.1.1
Adding sympy 1.1.1 to easy-install.pth file
Using c:\programdata\anaconda3\lib\site-packages
Searching for numpy==1.14.3
Best match: numpy 1.14.3
Adding numpy 1.14.3 to easy-install.pth file
Using c:\programdata\anaconda3\lib\site-packages
Finished processing dependencies for mdfreader==2.8
The error message you refered to is not printed anymore. However, the binary signal which I would like to analyze is still incorrectly displayed. There is no warning message which says that I am falling back to the python version of dataRead (there is no warning message at all).
I noticed that the same bug also occurs when using noDataLoading=True
:
mdfdict2=mdfreader.mdf(file)
mdfdict3=mdfreader.mdf(file,noDataLoading=True)
y2 = mdfdict2[label]['data']
y3 = mdfdict3.getChannelData(label)
plt.plot(y2,label='noDataLoading=False')
plt.plot(y3,label='noDataLoading=True')
plt.legend()
Good for the setup. It means that dataRead or fallback, you still have same error as noDataLoading=True will trigger dataRead or fallback usage. But as these 2 are different codes, it is unlikely there is error here, I would rather look at the input argument of dataRead and check if it fits with MDFValidator description. In readSortedRecord, can you tell what are dataRead arguments for this channel when using channelList ? Looking at your screenshots, I think it is first time I see such long data, cn_val_sbit=2799 ! Data is 1 bit in a record that has at least 350 bytes... Maybe there is something wrong happening at loadInfo because of this.
Please tell me in which file and which line I should set a breakpoint to check the input arguments.
@proedig would you mind sharing the file via email? I would like to check it against my lib
in mdf3reader.py, record class, readSortedRecord, line 533
@proedig would you mind sharing the file via email? I would like to check it against my lib
Hi Daniel, unfortunately I am not allowed to share this data file with others outside my company. Please let me know if I can provide other useful information about this problem.
in mdf3reader.py, record class, readSortedRecord, line 533
Ok, when executing
mdfdict3=mdfreader.mdf(file,noDataLoading=True)
y3 = mdfdict3.getChannelData(label)
I come along line 533 for two times. Since I have only one label I assume the first time is for the time vector and the second time for the signal values?
These are the input argument of dataRead for the first time:
chan.bitCount = 64 convertDataType3to4[chan.signalDataType] = 4 chan.nativedataFormat = 'f8' nrecord_chunk = 12931 self.CGrecordLength = 351 chan.bitOffset = 0 chan.posByteBeg = 0 chan.nBytes = 8
For the second time:
chan.bitCount = 1 convertDataType3to4[chan.signalDataType] = 0 chan.nativedataFormat = 'u1' nrecord_chunk = 12931 self.CGrecordLength = 351 chan.bitOffset = 7 chan.posByteBeg = 349 chan.nBytes = 1
@proedig
you can use this to scramble the file (randomizes all texts in the file; be sure to use a copy of the original measurement) before sending.
Yes, indeed if master channel data of the channel is not yet loaded, mdfreader load it. At first look: bitcount is ok dataFormat is ok first bit 2799 = 349 (posByteBeg) * 8 + 7 (bitOffset) --> ok nBytes ok for nrecord_chunk and CGrecordLength , I would need to have CG block screenshot corresponding to these channels to confirm (there could be some recordid not considered). Can you send it please ? I will review in detail the dataRead code with these parameters in the meantime
Ok, this is the header of the corresponding channel group for this label:
So, the values for nrecord_chunk and self.CGrecordLength seem to be reasonable then.
In additon, this is the header for the corresponding data group:
@proedig
you can use this to scramble the file (randomizes all texts in the file; be sure to use a copy of the original measurement) before sending.
thanks for the hint. I will try this out
ok, there is no recordid, not the issue.
I reviewed the dataRead Cython code but could not really find any hint.
I lately introduced new attribute for channels, bit_masking_needed
. Can you check line 959 in mdf4reader.py if it is False (dataRead shoudl already do bit extraction, no need to redo) ?
Most probably found, please confirm.
You should change in md3freader:
self[id].bit_masking_needed = False
into
self[rec_chan[id].channelNumber].bit_masking_needed = False
Modified in dev branch but be carefull using this branch, I did major refactoring for pep8 compliance, method names and arguments might not match your code anymore.
Thank you very much for your efforts. Now it's working and the signal data is read in correctly.
As you said, the bug was in line 542 in mdf3reader.py. However, in my version I had to change it to self[recChan[id].channelNumber].bit_masking_needed = False
. I do not have an object called rec_chan
.
Python version
'3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]'
Platform information
'Windows-10-10.0.14393-SP0'
Numpy version
'1.14.3'
mdfreader version
mdfreader 2.7.8
Description
Hi! I have problem with reading in mdffiles using the optional argument "channellist".
When I try
data = mdfreader.mdf(file)
there is no problem. All channels are read in. When I try the same file with
data = mdfreader.mdf(file,channelList=['n'])
I get the following output:
_C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py:552: UserWarning: Unexpected error: (<class 'ModuleNotFoundError'>, ModuleNotFoundError("No module named 'dataRead'",), <traceback object at 0x00000258E3768EC8>) warn('Unexpected error: {}'.format(excinfo())) C:\ProgramData\Anaconda3\lib\site-packages\mdfreader\mdf3reader.py:553: UserWarning: dataRead crashed, back to python data reading warn('dataRead crashed, back to python data reading')
However, there is reasonable output:
_In [29]: data Out[29]: {'time_2': {'unit': 's', 'description': '', 'master': 'time_2', 'masterType': 1, 'data': array([5.60005003e-03, ..., 4.75285915e+03])}, 'n': {'unit': 'rpm', 'description': 'Engine speed', 'master': 'time2', 'masterType': 1, 'data': array([0, ..., 0], dtype=uint16)}}
So this means for me there is no problem with the variable 'n'. Is it a problem with cython? I have installed cython version 0.28.5.
data.MDFVersionNumber = 300
Best wishes
Philip package_list.txt