Open tomcl opened 3 months ago
Instrumentation shows that the heap used by the simulation, while quite large, is not the determining factor here.
See #447 for how to make the simulator handle long simulations better with less heap use, but in prcatice that has no relevance to this issue.
Steps To Reproduce Bug
cpu
in Demo Projects.DATAPATH
custom symbol on the diagram on the left.NB - Going forward or backwards in the waveform simulator will not result in crash.
Developer Tools Screenshot
Developer tools was turned on at start up, but did not show anything useful at time of crash. It showed that the devtool window was disconnected at time of crash. See screenshot below.
Next Steps
Not sure why this would happen or how to proceed. Can Dr. Clarke (@tomcl) provide pointers on how to verify whether this is caused by the aforementioned circular buffer issues?
Well done! It is a bit of a mystery to me.
One suggestion - it is a memory problem. perhaps keeping dev tools open and graphing memory use and recreating the bug would show this (not sure whether dev tools can provide continuous indication of memory use - rather than graphing it at end after which maybe it will have crashed?).
Another suggestion: trace SVG generation. That would seem to be the issue?
Finally - there is some code that puts up a progress control when SVG generation takes too long. Maybe that is going wrong - but the issue I do not understand is why no backtrace. Perhaps in some circumstances an infinite loop?
The heap size goes quite quickly up above 500MB in this situation and given dev tools itself uses heap it looks like the heap limit may be the problem. we could print out heap memory use to console to diagnose better because I do not trust dev tools heap profiling.
The bug happens when expanding a small window for up to 1000 clocks.
https://github.com/electron/electron/issues/2056
Could be followed to increase the heap size and see if that helps - but we probably also want to limit the number of waveforms we create - fi there is a heap size issue.
This is not conclusive. it could also be some weird bug dependent on number of waveforms and number of clocks - perhaps to do with rerunning the simulation (which happens sometimes). But if increasing heap size fixes it we can be pretty sure it is that.
I have tried implementing that fix. See commit on dev_tc
for what I have done. I am not even sure the heap size is being increased because I cannot easily get the v8.getHeapSatatistics() method to work - it causes an infinite loop. Possibly this requires "require v8
" - all this js stuff is a bit opaque to me, and because it is js via F# and fable it is obscure to a lot of people.
I am still not convinced this is a heap limit. Printing out memory usage after each simulation is crashed at 720MB. That is suspiciously close to the standard 32 bit memory limit of 700MB or so. But surely we run a 64 bit version of electron? And the fix to increase heap size to 7500MB should be working...
I guess it needs checking:
The backytrace displayed by the main process (which I guess comes from the renderer) make sit clear this is a heap overflow probelm - but presumably one in the svg display code, not issie itself.
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF6AB500D36 node::SetTracingController+80134
2: 00007FF6AB500FF5 node::OnFatalError+645
3: 00007FF6AE15B873 v8::Function::NewInstance+851
4: 00007FF6AE15B808 v8::Function::NewInstance+744
5: 00007FF6AE21C417 v8::CppHeap::CollectGarbageInYoungGenerationForTesting+67671
6: 00007FF6ACDFF0E6 v8::CppHeap::GetHeapHandle+86326
7: 00007FF6ACDFA526 v8::CppHeap::GetHeapHandle+66934
8: 00007FF6ACDEC626 v8::CppHeap::GetHeapHandle+9846
9: 00007FF6AD012613 v8::CppHeap::GetHeapHandle+2262627
10: 00007FF63FE9C3F9
Error sending from webFrameMain: Error: Render frame was disposed before WebFrameMain could be accessed
at s.send (node:electron/js2c/browserinit:2:86493)
at .send (node:electron/js2c/browser_init:2:72283)
at BrowserWindow.
To wrap up the heap size maze.
https://github.com/electron/electron/issues/31330
It turns out that a hard heap size is limited to 4GB by a v8 pointer compression feature that electron will not ever be able to switch off. Total memory size is not so limited - unfortunately in F# almost everything is allocated on heap.
https://www.electronjs.org/blog/v8-memory-cage
That is sad - but not disastrous since Issie does not use anything like that amount of memory. The heap out of memory message here must be related to the SVG rendering process.
The symptoms are that this error happens only when making a large number of long SVGs.
Given this, my best guess is that the problem is the total SVG polyline path size, or the total SVG font size. We have 4 path points and up to one text element with 10 chars per clock cycle. Currently 1000 clock cycles are always generated.
It is therefore plausible that even though most of this stuff is not displayed, the SVG object size is too large. Part of the polylines is displayed and it is possible that therefore they are all processed. For 200 waveforms we have 1,000,000 polyline points.
Fixes
checking the conditions for this crash to happen:
This shows memory use as waveforms are remade under conditions that will cause the crash at high zoom-in. The three levels of memory use correspond to making waveforms after pressing the zoom-in button twice. Each time memory usage goes up, and also the time taken to make the waveforms increases by a large amount.
Memory allocation trace shows most of the allocated memory is SVG text:
higher zoom allocations:
lower zoom allocations:
The difference here reflects the fact that at higher zooms more numbers are printed on the waveforms.
Making a circuit with 1000 binary waveforms - each changing every cycle - the system seems much more robust and SVG regeneration time is constant with zoom.
So we have three fixes:
Describe the bug White-screen crash with issie unresponsive after has been reported when doing long simulations of large designs.
To Reproduce Not yet enough information
Additional context This was reported Spring 2024 by an Issie user. It is hoped that they will be able to recreate the issue with dev tools open and report the stacktrace that will probably diagnose this problem.