search

mmontag / chip-player-js

Web-based music player for a variety of video game and chiptune music formats.
https://chiptune.app
GNU General Public License v3.0
324 stars 17 forks source link

Piano roll visualizer #91

Open mmontag opened 2 years ago

mmontag commented 2 years ago

Piano Roll

Implement a piano roll style visualizer for note data.

Examples:

Modizer uses custom OpenGL code to generate the 3D falling piano notes: https://github.com/yoyofr/modizer/blob/e57dd94aa891e148ea8376db2734f6620a93f729/src/DetailViewControllerIphone.mm#L6070 https://github.com/yoyofr/modizer/blob/c4338354ca93a2809dcd843f4042fc295dd576e5/utils/Misc/RenderUtils.mm#L5254

vpavlenko commented 9 months ago

Hi Matt,

Thank you for the fantastic player!

I'd like to think out load regarding making a piano roll visualizer. I see three options:

A) Getting a real-time frequency info from oscillators (the NFSPlay way) - which should work for NES, Genesis and others. Something like that: https://github.com/mmontag/chip-player-js/commit/9b59554c24e50632dcdb768ef5898fb5ea2ecb68 Do you think it's the optimal approach? (Update: I see now that you have an article on the period-to-note conversion)

B) Rendering the notes ahead of time - say, do core._gme_play(this.gmeCtx, this.bufferSize * 2, this.buffer); silently with a very long buffer and dump the oscillator states ahead of time. An upside will be that a user can see what's gonna happen next, and even get a bird's-eye view picture of a whole piece. A downside may be the initial freezing - although generating 30 seconds of the buffer (waveform samples, not oscillator stats) on my Mac M1 takes as little as 90 ms.

C) Pre-calculate the note content for each tune ahead of time and load them along the .nsf file - as an external analysis (.mid or the like). This approach may have even more downsides, but among potential upsides we can calculate certain music theory stuff ahead of time: measures, key, mode, chords - and display it in a nice way (think https://www.hooktheory.com/theorytab)

I don't really want to distract you, so I'm only asking for your thoughts if you have time to brainstorm about this.

Cheers!

mmontag commented 9 months ago

Interesting, I was just thinking about this last night. tl;dr: I would go with option A, the real-time frequency approach.

From what I've seen, these chip-based players almost always show a realtime note visualization - you can't look forward indefinitely to see what notes are coming. I am thinking of NSFPlay and X68000-style music player software (see below).

Like you said, with chip-based music, you're just seeing the channel frequency mapped onto piano notes. You can see the note wiggle between two semitones when it's actually just a vibrato. Or you will see a bass drum sweep quickly down the lower register of notes.

Based on this, I think it would be too ambitious to do read-ahead and render notes arbitrarily far ahead of playback. The shortcoming of the current spectrogram visualization is not that it doesn't look ahead, but that it has so much noise from harmonic overtones, making it harder to see the notes played.

Either way, it's a large amount of work to modify the player libraries to expose the channel states (note on/off + frequency + volume. Would probably make sense to do at the same time as exposing the discrete channel audio buffers (to visualize the channel waveforms).

MMDSP: image hoot: image LMZ2: image

mmontag commented 9 months ago

Also, my primary goal is a nice full-page viz that could be recorded and uploaded to YouTube.

vpavlenko commented 9 months ago

I'm currently exploring the approach when a prior music theory annotation is done for the piece to make a meaningful visualization. You can see the current state: https://vpavlenko.github.io/chiptheory/

The code: https://github.com/vpavlenko/chiptheory/tree/master/src/components/chiptheory

I'm not sure how to automatically generalize this approach to the majority of pieces, but this is not my goal so far. I care about making a handy tool for annotating pieces interactively.

mmontag commented 9 months ago

Wow, this is pretty cool!

vpavlenko commented 7 months ago

And then, regarding MIDI files, I also wanna share a super early prototype: https://rawl.rocks/

Repo: https://github.com/vpavlenko/rawl

Tonics aren't perfectly estimated as for now, I'm gonna tackle it soon

vpavlenko commented 2 days ago

Completely irrelevant to your original project, but I just wanted to share the state of visualizations I've achieved so far:

https://rawl.rocks/c/MIDI/The%20Beatles/Eleanor%20Rigby.4.mid https://rawl.rocks/f/chopin_military_polonaise https://rawl.rocks/f/Carol_of_the_Bells https://rawl.rocks/f/god_only_knows_bioshock_infinite_2013 https://rawl.rocks/f/chopin_ballade_no_4_piano_solo