How do I keep the eventCallback in sync with playback?

[mssever]

Hello,

I'm trying to use ABCJS.TimingCallbacks to highlight each note as it's played. Playback is handled by ABCJS.synth.SynthController. I'm struggling to keep the timing callback in sync with playback. I've tried to attach an event listener to the play button, but that only partially works. If I stop/pause playback before the song ends, seek, or do anything else other than playing from start to finish, the event callback gets out of sync. Once the playback finishes and I restart it, the event callback doesn't get called until after I pause.

I'm sure that I'm missing something here and doing something fundamentally wrong. What's the correct way to synchronize the event callbacks with playback?

What I've tried so far:

1. Reading the documentation
2. Doing internet searches
3. Asking AI and getting completely wrong answers (which suggests that my approach is fundamentally wrong)
4. Tracing the event handlers in the browser's dev tools (and only finding my own event handlers; is it possible to respond to click events without an event handler?)
5. Reading the ABCJS source code without being able to find what I need.

In case it helps, here's my TypeScript class. The interesting methods are init_timing_callbacks, init, and play:

import ABCJS from "abcjs";
import 'abcjs/abcjs-audio.css';
import { crEl, qs } from "../../lib/dom";
import { listen, unlisten } from "../../lib/events";
import type { Maybe, Nullable } from "../../lib/types/meta";

export class ScorePlayback {
  score: Maybe<ABCJS.TuneObjectArray>;
  initialized: boolean;
  controller: ABCJS.SynthObjectController;
  container: HTMLElement;
  warp: Maybe<number>;
  highlighted_elements: HTMLElement[] = [];
  timing_callbacks: Maybe<ABCJS.TimingCallbacks>;
  buffer: Maybe<ABCJS.MidiBuffer>;

  constructor(controller_selector: string) {
    this.initialized = false;
    this.container = qs(controller_selector);
    this.controller = new ABCJS.synth.SynthController()
    this.controller.load(controller_selector, {}, {
      displayRestart: true,
      displayPlay: true,
      displayProgress: true,
      displayWarp: true,
    });
    const text = crEl("span", { text: "Tempo: ", class: "label" });
    const label = qs('.abcjs-tempo-wrapper label', this.container);
    label.insertBefore(text, label.firstChild);
  }

  set rendered_score(score: ABCJS.TuneObjectArray) {
    this.score = score;
    this.initialized = false;
  }

  get rendered_score(): Maybe<ABCJS.TuneObjectArray> {
    return this.score;
  }

  delete_container(): void {
    this.container.parentElement?.removeChild(this.container);
  }

  init_timing_callbacks(): void {
    console.debug(`Initializing timing callbacks. Current time: ${new Date().toLocaleTimeString()}`);
    this.timing_callbacks = new ABCJS.TimingCallbacks(
      (this.rendered_score as ABCJS.TuneObjectArray)[0],
      { eventCallback: this.note_play_callback }
    );
  }

  async init(): Promise<void> {
    if (!this.rendered_score) {
      throw new Error("No score to play.");
    }
    console.debug("Initializing score playback.");
    this.init_timing_callbacks();
    const midi_buffer = new ABCJS.synth.CreateSynth();
    await midi_buffer.init({
      visualObj: this.rendered_score[0],
      // debugCallback: console.debug,
    });
    await this.controller.setTune(this.rendered_score[0], false);
    await midi_buffer.prime();
    this.buffer = midi_buffer;
    if (this.warp) {
      this.controller.setWarp(this.warp);
    }
    const play_button = qs<HTMLButtonElement>('button.abcjs-midi-start', this.container);
    listen({
      event: "click",
      target: play_button,
      callback: () => this.play(play_button),
      name: "play",
      passive: true,
    });
    // if(midi_buffer.getIsRunning()) {
    //   this.play(play_button);
    // }
    this.initialized = true;
    play_button.click();
  }

  play(button: HTMLButtonElement): void {
    // this.timing_callbacks?.reset();
    this.timing_callbacks?.start();
    // unlisten({ event: "click", target: button, name: "play" });
    // listen({
    //   event: "click",
    //   target: button,
    //   callback: () => this.pause(button),
    //   name: "pause",
    //   passive: true,
    // });
  }

  pause(button: HTMLButtonElement): void {
    this.timing_callbacks?.pause();
    this.clear_highlighted_notes();
    unlisten({ event: "click", target: button, name: "pause" });
    listen({
      event: "click",
      target: button,
      callback: () => this.play(button),
      name: "play",
      passive: true,
    });
  }

  note_play_callback = (event: Nullable<ABCJS.NoteTimingEvent>): ABCJS.EventCallbackReturn => {
    console.debug(`note_play_callback(): ${event?.milliseconds}`);
    this.clear_highlighted_notes();
    if (event) {
      // console.log(event);
      event.elements?.forEach(wrapper => {
        wrapper.forEach(elem => {
          this.highlight_notes(elem);
        });
      });
    } // else {
    //   this.buffer?.seek(0);
    //   this.init_timing_callbacks();
    // }
    return;
  }

  // clear_listeners(selectors: string[], root = this.container): void {
  //   selectors.forEach(selector => {
  //     const item = qs(selector, root);
  //     item.parentElement?.replaceChild(item.cloneNode(true), item);
  //   });
  // }

  highlight_notes = (elem: HTMLElement): void => {
    elem.setAttribute("fill", "#4d9900");
    elem.classList.add("abcjs-note_selected");
    this.highlighted_elements.push(elem);
  }

  clear_highlighted_notes = (): void => {
    this.highlighted_elements.forEach((elem) => {
      elem.setAttribute("fill", "currentColor");
      elem.classList.remove("abcjs-note_selected");
    });
    this.highlighted_elements = [];
  }
}

[asjl]

I added playback to my WordPress plugin, choon-player. The code is at: • https://codeberg.org/isnz/choon-player/src/branch/main/js/choon-player.js See from line 755 - I'm not using typescript and I'm using the ABCJS Editor but there should be some useful stuff in there.

You can see an example in action at: • https://lpnz.org/ballymagan-fairies/

On Mon, Jun 17, 2024, at 01:16, Scott Severance wrote:

Hello,

I'm trying to use ABCJS.TimingCallbacks to highlight each note as it's played. Playback is handled by ABCJS.synth.SynthController. I'm struggling to keep the timing callback in sync with playback. I've tried to attach an event listener to the play button, but that only partially works. If I stop/pause playback before the song ends, seek, or do anything else other than playing from start to finish, the event callback gets out of sync. Once the playback finishes and I restart it, the event callback doesn't get called until after I pause.

I'm sure that I'm missing something here and doing something fundamentally wrong. What's the correct way to synchronize the event callbacks with playback?

What I've tried so far:

1. Reading the documentation https://paulrosen.github.io/abcjs/animation/timing-callbacks.html
2. Doing internet searches
3. Asking AI and getting completely wrong answers (which suggests that my approach is fundamentally wrong)
4. Tracing the event handlers in the browser's dev tools (and only finding my own event handlers; is it possible to respond to click events without an event handler?)
5. Reading the ABCJS source code without being able to find what I need. In case it helps, here's my TypeScript class. The interesting methods are init_timing_callbacks, init, and play:

import ABCJS from "abcjs"; import 'abcjs/abcjs-audio.css'; import { crEl, qs } from "../../lib/dom"; import { listen, unlisten } from "../../lib/events"; import type { Maybe, Nullable } from "../../lib/types/meta";

export class ScorePlayback { score: Maybe; initialized: boolean; controller: ABCJS.SynthObjectController; container: HTMLElement; warp: Maybe; highlighted_elements: HTMLElement[] = []; timing_callbacks: Maybe; buffer: Maybe;

constructor(controller_selector: string) { this.initialized = false; this.container = qs(controller_selector); this.controller = new ABCJS.synth.SynthController() this.controller.load(controller_selector, {}, { displayRestart: true, displayPlay: true, displayProgress: true, displayWarp: true, }); const text = crEl("span", { text: "Tempo: ", class: "label" }); const label = qs('.abcjs-tempo-wrapper label', this.container); label.insertBefore(text, label.firstChild); }

set rendered_score(score: ABCJS.TuneObjectArray) { this.score = score; this.initialized = false; }

get rendered_score(): Maybe { return this.score; }

delete_container(): void { this.container.parentElement?.removeChild(this.container); }

init_timing_callbacks(): void { console.debug(Initializing timing callbacks. Current time: ${new Date().toLocaleTimeString()}); this.timing_callbacks = new ABCJS.TimingCallbacks( (this.rendered_score as ABCJS.TuneObjectArray)[0], { eventCallback: this.note_play_callback } ); }

async init(): Promise { if (!this.rendered_score) { throw new Error("No score to play."); } console.debug("Initializing score playback."); this.init_timing_callbacks(); const midi_buffer = new ABCJS.synth.CreateSynth(); await midi_buffer.init({ visualObj: this.rendered_score[0], // debugCallback: console.debug, }); await this.controller.setTune(this.rendered_score[0], false); await midi_buffer.prime(); this.buffer = midi_buffer; if (this.warp) { this.controller.setWarp(this.warp); } const play_button = qs('button.abcjs-midi-start', this.container); listen({ event: "click", target: play_button, callback: () => this.play(play_button), name: "play", passive: true, }); // if(midi_buffer.getIsRunning()) { // this.play(play_button); // } this.initialized = true; play_button.click(); }

play(button: HTMLButtonElement): void { // this.timing_callbacks?.reset(); this.timing_callbacks?.start(); // unlisten({ event: "click", target: button, name: "play" }); // listen({ // event: "click", // target: button, // callback: () => this.pause(button), // name: "pause", // passive: true, // }); }

pause(button: HTMLButtonElement): void { this.timing_callbacks?.pause(); this.clear_highlighted_notes(); unlisten({ event: "click", target: button, name: "pause" }); listen({ event: "click", target: button, callback: () => this.play(button), name: "play", passive: true, }); }

note_play_callback = (event: Nullable): ABCJS.EventCallbackReturn => { console.debug(note_play_callback(): ${event?.milliseconds}); this.clear_highlighted_notes(); if (event) { // console.log(event); event.elements?.forEach(wrapper => { wrapper.forEach(elem => { this.highlight_notes(elem); }); }); } // else { // this.buffer?.seek(0); // this.init_timing_callbacks(); // } return; }

// clear_listeners(selectors: string[], root = this.container): void { // selectors.forEach(selector => { // const item = qs(selector, root); // item.parentElement?.replaceChild(item.cloneNode(true), item); // }); // }

highlight_notes = (elem: HTMLElement): void => { elem.setAttribute("fill", "#4d9900"); elem.classList.add("abcjs-note_selected"); this.highlighted_elements.push(elem); }

clear_highlighted_notes = (): void => { this.highlighted_elements.forEach((elem) => { elem.setAttribute("fill", "currentColor"); elem.classList.remove("abcjs-note_selected"); }); this.highlighted_elements = []; } }

— Reply to this email directly, view it on GitHub https://github.com/paulrosen/abcjs/issues/1022, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABL6VKMEBLMYNR7KMWFJK43ZHY2HBAVCNFSM6AAAAABJNCEC26VHI2DSMVQWIX3LMV43ASLTON2WKOZSGM2TMMJSGE3DKNY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[paulrosen]

Thanks for the detailed issue. I'll try out your code. It's possible that it is a bug in abcjs and not yours.