Hearing cracking/popping, and noticing CSS animation framerate drop after running midio for awhile

[evangipson]

Expected Behavior

You'd never hear popping or crackling in the audio, and never notice a slowdown no matter how long you let midio run.

Actual Behavior

CRACK beautiful noises POPPLE POP POP gorgeous airy chord CRAAAAAAckckkckKCkkKKk

Solutions

1. the <span>s that i'm creating for the piano key effects might not be getting cleaned up correctly? i am using .removeChild and not assigning it to a variable... so it should just go away. Maybe i could only allow X number of spans to be created??
2. performance of many oscillators being created at a time with many filters each oscillator. i don't think this is the problem because if i load up midio and click a bunch it's fine and doesn't start clicking/crackling/slowing down til after a bunch of clicking. it's like the problem always takes around a minute to show up- and sometimes it never shows up at all.. .which leads me to my third theory
3. some filter being applied to some oscillator at some volume level sometimes. this might be due to any number of calculations in the code for ADSR, pretty much anytime i'm doing any filter or gain adjustments on any filter or oscillator- which is kind of a lot. i believe this is my strongest lead.

Steps to Reproduce the Problem

1. Load up midio
2. Click a bunch
3. Notice how you don't get crackling/popping/slow down all of the time
4. Reload midio
5. Let it run for 1-5 minutes
6. See if you hear crackling

Screenshots

Specifications

• Midio Version: current
• Operating System: Windows 10
• Node Version: v0.10.25

[robertpateii]

i think i figured this out and sent you messages (and gifs) on teams about it. i don't want to spoil the ending here

[robertpateii]

hint: you're really close with "using .removeChild and not assigning it to a variable... so it should just go away. "

it doesn't just go away. none of the objects you stop using do. They hang around until the garbage collector decides to trace out all the objects, figure out which ones aren't in use, and then remove them from memory in one big go.

This is why high performance games are not written in garbage collected languages (and if they are, the garbage collector's behavior is tip toed around and carefully managed and understood).

"Periodically, the garbage-collector will start from these roots, find all objects that are referenced from these roots, then all objects referenced from these, etc. Starting from the roots, the garbage collector will thus find all reachable objects and collect all non-reachable objects." https://developer.mozilla.org/en-US/docs/Web/JavaScript/Memory_Management

Since you can't allocate and deallocate your own memory, you need some way to not have to create a bunch of objects all the time.

[evangipson]

This was addressed in https://github.com/evangipson/midio/pull/29 (more specifically it began with commit https://github.com/evangipson/midio/commit/ad86ac080bbb0708f00d298e4e98546b05c49f7f and ended right after with commit https://github.com/evangipson/midio/commit/c7148d49f6bb836656c4d39e54f1bb245ee2d5d1)... buuut I'd like to test more before closing this issue.

Next steps:

1. ✅ Let midio run for an extended period of time and observe it
   • This is definitely still an issue. I get crackling after about 2-3 minutes of midio about 80% of the time.
2. Let midio run for awhile then play with generating a new song
3. Click around lots and observe as well

[robertpateii]

the new look is fantastic

[evangipson]

I don't think this is a JavaScript heap problem. I've done some research on Oscillators and seems like you can't .start() an oscillator that you've used .stop() on. The garbage collector will clean up the oscillator after a .stop() is called.

Then I took a few heap snapshots:

after some heavy usage and they were all the same as the heap when the program loaded... so I don't think I have any leaks on the JS side. I think it might be due to me using "opacity" and an animation on it, forcing a repaint perhaps.

I think I'm really onto something with this will-change property that I'm reading about. it's supported in most browsers and I don't mind this not working in IE11 (edge doesn't have that good of support for web audio API in the first place)- but it does suck that Edge won't get the optimizations if I do go that route.... but after testing "will-change" it didn't really work. I still got the CPU 100% spikes.

Tried to add some optimizations via commits https://github.com/evangipson/midio/commit/7ca92ef87f4355e0a74662d27cdec703ef84b990, https://github.com/evangipson/midio/commit/c0dc865556ebb155927add8fb70afc102a2b479e, https://github.com/evangipson/midio/commit/6ad890fe6380a9a6cbcec1c0d30d328ee95870f2, and https://github.com/evangipson/midio/commit/17cd690a9e90fae9323f8652cbee2cd2bb762e7f. Testing those out right now. getting less crackling in general so I am hopeful!

Here's what I observe in the performance profiler in chrome devtools, just sudden spikes to 100% CPU with no other spikes:

[evangipson]

After some more lengthy testing this morning - I've found out a couple things:

• When I display: none the note-active class... there isn't much performance difference. I'm beginning to think it's the JS again, and not the CSS because my animations are optimized now.
• I'm noticing some slowdown with user input notes after awhile... this could be related to the issue I'm having perhaps? Like you'll click around the on the visualizer and not see your notes appear for 5-10 seconds... but it's always super responsive for about a minute or so upon initial load. I believe this is due to my use of "requestAnimationFrame"

[robertpateii]

Yo canvas was designed for the problem you're trying to solve with rendering these notes. Learn it; use it

[robertpateii]

Disclaimer: I haven't used it much so I could be wrong, but it does say this on the tin: "used to draw graphics using scripting (usually JavaScript). This can, for instance, be used to draw graphs, make photo composition or simple (and not so simple) animations. "

[robertpateii]

https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial

[robertpateii]

Or by "it's the js" do you mean purely the music genration? What if you ripped out the visual elemets entirely as a test?

[evangipson]

yeah it's the music generation, or something else in the JS i believe, because I turned off everything relating to drawing the notes and was still getting performance issues (crackling/popping/lagging input/fan blowing like crazy in laptop) after awhile, even after disabling the drawNoteOnVisualizer() function , which handles updating the DOM.