Stack Overflow when File is unfinalized

[tobiasandorfer]

Hello Daniel,

Python version

'python=3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]' 'os=Windows-10-10.0.17763-SP0' 'numpy=1.16.5' 'asammdf=5.20.1'

MDF version

'asammdf=5.20.1'

Traceback

Fatal Python error: Cannot recover from stack overflow.

Current thread 0x0000a214 (most recent call first):

a\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 67 in load File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 211 in load File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 726 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 lization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block File "C:\ProgramData\Anaconda3\lib\site-packages\asammdf\blocks\finalization_shim.py", line 734 in _load_block ...

Traceback (most recent call last): File "c:/svn/01_Python/02_Script/ScriptAutoAnalyseJ1.py", line 291, in for result in results: File "C:\ProgramData\Anaconda3\lib\concurrent\futures\process.py", line 483, in _chain_from_iterable_of_lists for element in iterable: File "C:\ProgramData\Anaconda3\lib\concurrent\futures_base.py", line 598, in result_iterator yield fs.pop().result() File "C:\ProgramData\Anaconda3\lib\concurrent\futures_base.py", line 435, in result return self.get_result() File "C:\ProgramData\Anaconda3\lib\concurrent\futures_base.py", line 384, in get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Description

I´m running a lot of mf4 files parallel with the concurrent.futures.ProcessPoolExecutor()

In one partikular dataset i always got the error above. After running it multiple times i found a file that was not finished

after removing the file from the dataset, there was no error and the calculations finished.

This time sadly i have no solution to offer :-) Take care and again thanks for the amazing work!

Tobias

[danielhrisca]

Hello Tobias,

sadly I don't have a solution either. The finalization shim is only working with some files. A more general an complete finalization mechanism need sto be implemented

[tobiasandorfer]

Hello Daniel,

to prevent the stackoverflow for the moment, would it be practical to implement somting like this in the code:

with open(path_to_mdf, 'rb') as f:
    if f.read()[0:8] == b'UnFinMF ':
        print("dont touch")

regards Tobias

[MatinF]

I think that would disable all support for unfinalized files, correct? Doing so would be problematic for our use cases at least where the current implementation of unfinalized files works as intended.

@tobiasandorfer Perhaps you can review the code for the finalization to see what parts are missing in order to support the unfinalized files you have. What tool are you recording these with?

[tobiasandorfer]

@MatinF you are absolutly right, I didnt reflect much on this proposal:-) The tool must have been CANoe I think. I will have a look at the code, and come back if i find a solution.

[danielhrisca]

@tobiasandorfer @MatinF I've made a rework for the finalization code. This should be more generic/complete now. Please check if it is working better

Edit: use the development branch code

[MatinF]

@tobiasandorfer Out of curiosity, what other programs are you able to load your unfinalized data in? And would you be able to test out if you can load the unfinalized MDF4 files from this zip?

sample-data.zip

@danielhrisca I'll check it out to see if it works as intended for ours as well.

[tobiasandorfer]

@MatinF besides the asammdf i only use CANape/vSignalyser for looking at MDF files. I can not open my unfinalized files with this software, i suspect, they use my initial aproach as well :-) besides the vector / python tools i know NiDiadem, they use a converter that works some time with i.O. files. What i can recommend for working with mdf is the AVL CONCERTO 5 i used it in a trial phase. No idea though how it handels broaken files.

[tobiasandorfer]

@danielhrisca I tryed the development branch code and it seams to work. I have no more Stack Overflow Error

Thanks again !!!