Opening fst file containing huge number of variables

[wa5wi5ou]

Hi Thomas,

I tried to open an fst file with 70000 variables (real), but impulse wasn't able to open it. Can you give me a hand please? test_real_70000.zip Find attached the fst file that I used.

ps: I could open it with gtkwave without problems

thanks,

[thomasAtToem]

Hi Klai,

thanks for sending the issue. We found a timeout problem. Will ping you when the update is available. The string problem is still under investigation.

thanks, thomas

On 07.09.2017 12:28, Klai wrote:

Hi Tomas,

I tried to open an fst file with 70000 variables (real), but impulse wasn't able to open it. Can you give me a hand please? test_real_70000.zip https://github.com/toem/impulse/files/1284347/test_real_70000.zip Find attached the fst file that I used.

ps: I could open it with gtkwave without problems

thanks,

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/toem/impulse/issues/50, or mute the thread https://github.com/notifications/unsubscribe-auth/AI6sTmQg5XUyoH7Ww_kCFTLzMu6Oi5Heks5sf8VrgaJpZM4PPnPu.

[wa5wi5ou]

Hi Thomas,

For the string issue, I fixed it and it works fine with me.

Actually I found 2 issues with the Impulse converter : 1- The string problem. 2- When I perform make with a huge file, (i.e : win32-x86_64/fst.exe ) I get a segmentation fault.

And for now, the problem of opening a huge file isn't solved.

Please find attached the patch below. impulse.zip

Thanks,

Klai

[thomasAtToem]

Hi Klai,

great , we will check.

For the 70000 float file we found 2 problems.

• Timeout value can not be set. (from configuration)
• Performance problems because of many signals in 1 scope (writer library checks for unique name) -> so timeout occured 70000 signals in many scopes would not make the problem.

thanks, thomas

On 08.09.2017 17:54, Klai wrote:

Hi Thomas,

For the string issue, I fixed it and it works fine with me.

Actually I found 2 issues with the Impulse converter : 1- The string problem. 2- When I perform make with a huge file, (i.e : win32-x86_64/fst.exe ) I get a segmentation fault.

And for now, the problem of opening a huge file isn't solved.

Please find attached the patch below. impulse.zip https://github.com/toem/impulse/files/1288409/impulse.zip

Thanks,

Klai

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/toem/impulse/issues/50#issuecomment-328142722, or mute the thread https://github.com/notifications/unsubscribe-auth/AI6sTna99ozwOl14HlNG2V7jrjE_AaKnks5sgWNJgaJpZM4PPnPu.

[thomasAtToem]

Hi Klai,

1 is fixed in our code base -- thanks! 2 could you explain what you mean with huge/make ?!? 3 what was the reason of your change from static memory to malloc for the buffers in your patch ?

thanks, thomas

On 08.09.2017 17:54, Klai wrote:

Hi Thomas,

For the string issue, I fixed it and it works fine with me.

Actually I found 2 issues with the Impulse converter : 1- The string problem. 2- When I perform make with a huge file, (i.e : win32-x86_64/fst.exe ) I get a segmentation fault.

And for now, the problem of opening a huge file isn't solved.

Please find attached the patch below. impulse.zip https://github.com/toem/impulse/files/1288409/impulse.zip

Thanks,

Klai

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/toem/impulse/issues/50#issuecomment-328142722, or mute the thread https://github.com/notifications/unsubscribe-auth/AI6sTna99ozwOl14HlNG2V7jrjE_AaKnks5sgWNJgaJpZM4PPnPu.

[wa5wi5ou]

Hi Thomas,

1- When I went to the plugins repository: ($eclipse repo$/plugins/de.toem.impulse.serializer.hdl_1.8.157/flux/fst/) and I perform make, after that I executed the fst.exe; generated ( for me in win32-x86_64 ), I had a segmentation fault. example: gdb --args ./fst.exe test_real_70000.fst So when I changed from static memory to malloc for the buffers, it's fixed.

2- I generated a 70000 variable fst file, with many scopes, and I still get same problem in Impulse. I could open it with gtkwave.

Please find attached the fst file below. test_real_70000_scopes.zip

thanks, Klai

[thomasAtToem]

With 1.8.16 the file can be loaded when setting the timeout to 30000 (fst serializer configuration). The slow reading is affected by many signals in 1 folder (impulse checks for unique name - this take time when many signals are in 1 folder). If you pack the 70000 signals into many folders (the usual scenario) there is no performance problem !

[wa5wi5ou]

Hi Thomas,

I still get the same problem when I tried to open the fst file with 70000 signals in many scopes.(file attached below) When I open it with Impulse, it shows nothing, there is no data in the viewer. I tried to compile impulse in eclipse plugin repository, with these file (i.e : 1) Go to <your eclipse repo>/plugins/de.toem.impulse.serializer.hdl_1.8.16/flux/fst/ 2) gdb -ex run -ex bt --args win32-x86_64/fst.exe test_70000_scopes.fst 3) I get :

Thread 1 received signal SIGSEGV, Segmentation fault.
0x000000000043ba36 in ___chkstk_ms ()
#0  0x000000000043ba36 in ___chkstk_ms ()
#1  0x000000000055ec70 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

So when I changed the static allocation memory to dynamic with malloc, I had no longer SIGSEGV, and I could open the file with Impulse. (See patch attached).

Impulse_1.8.16.patch.zip test_70000_scopes.zip

Thanks Klai

[thomasAtToem]

Hi Klai,

thanks for this. Yes, we were not sure about the heap patch, but its required to get around this stack Problem ( we havnt seen this on our platforms). Will fix this for next release (probably end of the week).

thanks, thomas

Am 2017-09-19 10:10, schrieb Klai:

Hi Thomas,

I still get the same problem when I tried to open the fst file with 70000 signals in many scopes.(file attached below) When I open it with Impulse, it shows nothing, there is no data in the viewer. I tried to compile impulse in eclipse plugin repository, with these file (i.e :

• Go to <your eclipse repo>/plugins/de.toem.impulse.serializer.hdl_1.8.16/flux/fst/
• gdb -ex run -ex bt -args win32-x86_64/fst.exe test_70000_scopes.fst
• I get :

Thread 1 received signal SIGSEGV, Segmentation fault. 0x000000000043ba36 in ___chkstk_ms ()

0 0x000000000043ba36 in ___chkstk_ms ()

1 0x000000000055ec70 in ?? ()

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

So when I changed the static allocation memory to dynamic with malloc, I had no longer SIGSEGV, and I could open the file with Impulse. (See patch attached).

Impulse_1.8.16.patch.zip [1] test_70000_scopes.zip [2]

Thanks Klai

-- You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub [3], or mute the thread [4].

*

Links:

[1] https://github.com/toem/impulse/files/1313465/Impulse_1.8.16.patch.zip [2] https://github.com/toem/impulse/files/1313466/test_70000_scopes.zip [3] https://github.com/toem/impulse/issues/50#issuecomment-330463389 [4] https://github.com/notifications/unsubscribe-auth/AI6sTqOwTXBIJiYgjAuNDYjAbQgvya0Wks5sj3cDgaJpZM4PPnPu

[thomasAtToem]

Now fixed, please check !