New mechanism for frequency aligning tracks on a spectrogram

[FredrikKarlssonSpeech]

The current convention of frequency aligning of tracks on top of a spectrogram creates a couple of hard to solve problems since only one track can have the name set to FORMANTS at the time for an entire database. So, if you for instance want to compare outputs of two formant extraction settings, you could have set up two different perspectives for that and easily switch between them, but you can't, since only one can be overlayed (since they have to have the name FORMANTS).

And, you cant overlay say harmonics on a narrow band spectrogram, unless you define them as FORMANTS. It just feels awkward and constraining.

Could not all definitions of

”assign": [{
    "signalCanvasName": "SPEC",
    "ssffTrackName":  <whatever track name>
}],”

be assumed by the web app to be frequency aligned? If non-frequency aligned signals still really needs to be overlayed on a spectrogram, I suggest that you could disable the frequency aligned default behavior like this

”assign": [{
    "signalCanvasName": "SPEC",
    "ssffTrackName":  <whatever track name>,
    "frequencyAlign": false
}],”

But I think this would very seldom be used...

[raphywink]

Getting rid of the very hacky "FORMANTS" track name for frequency alignment has been on my todo list for quite some time now. I actually like your idea of having it be true by default for any track name but you can disable it by setting "frequencyAlign" to false. Will def. try to get around to this soonish.... have got a big bug to fix in the query engine of emuR first though. Will leave the issue open to track this feature

[samgregory]

I have a related question: The FORMANTS track name overplayed on the SPEC canvas also provides a feature to hand correct the formant tracks. Is there an equivalent feature for F0 tracks?

[raphywink]

I have a related question: The FORMANTS track name overplayed on the SPEC canvas also provides a feature to hand correct the formant tracks. Is there an equivalent feature for F0 tracks?

Unfortunatelly not. Correcting is currently only possible on the FORMANTS track that is overlayed over the SPEC canvas.

[samgregory]

Any idea if this would be considered for a future update?

[raphywink]

Yeah possibly as part of implementation of the feature mentioned in this GitHub issue. But can't promise anything...

[samgregory]

Could you also address an idiosyncrasy in the SsffParserWorker that the ssff2js() is able to convert columns from DOUBLE, FLOAT, LONG (despite an error message that suggests that LONG is unsupported), SHORT or BYTE but the jso2ssff() can only covert back to SHORT?

https://github.com/IPS-LMU/EMU-webApp/blob/f872cdc73c14b38d2c999545ae5f899bb5ee8934/src/app/workers/ssff-parser.worker.js#L423-L472

Possibly something like this?

jso.Columns.forEach((col) => {
    if (col.ssffdatatype === 'BYTE') {
        bytePerTime += col.length;
    } else if (col.ssffdatatype === 'SHORT') {
        bytePerTime += 2 * col.length;
    } else if (col.ssffdatatype === 'LONG') {
        bytePerTime += 4 * col.length;
    } else if (col.ssffdatatype === 'FLOAT') {
        bytePerTime += 4 * col.length;
    } else if (col.ssffdatatype === 'DOUBLE') {
        bytePerTime += 8 * col.length;
    } else {
        failed = true;
    }
});

// check if failed
if (failed) {
    return ({
        'status': {
            'type': 'ERROR',
            'message': 'Unsupported column type! Only DOUBLE, FLOAT, LONG, SHORT, BYTE columns supported for now!'
        }
    });
}

var byteSizeOfDataBuffer = bytePerTime * jso.Columns[0].values.length;

var dataBuff = new ArrayBuffer(byteSizeOfDataBuffer);
var dataBuffView = new DataView(dataBuff);

// convert buffer to header
var ssffBufView = new Uint8Array(global.stringToUint(headerStr));

// loop through vals and append array of each column to ssffBufView
var byteOffSet = 0;
jso.Columns[0].values.forEach((curArray, curArrayIDX) => {
    jso.Columns.forEach((curCol) => {
        if (curCol.ssffdatatype === 'BYTE') {
            curCol.values[curArrayIDX].forEach((val) => {
                dataBuffView.setInt8(byteOffSet, val, true);
                byteOffSet += 1;
            });
        } else if (curCol.ssffdatatype === 'SHORT') {
            curCol.values[curArrayIDX].forEach((val) => {
                dataBuffView.setInt16(byteOffSet, val, true);
                byteOffSet += 2;
            });
        } else if (curCol.ssffdatatype === 'LONG') {
            curCol.values[curArrayIDX].forEach((val) => {
                dataBuffView.setInt32(byteOffSet, val, true);
                byteOffSet += 4;
            });
        } else if (curCol.ssffdatatype === 'FLOAT') {
            curCol.values[curArrayIDX].forEach((val) => {
                dataBuffView.setFloat32(byteOffSet, val, true);
                byteOffSet += 4;
            });
        } else if (curCol.ssffdatatype === 'DOUBLE') {
            curCol.values[curArrayIDX].forEach((val) => {
                dataBuffView.setFloat64(byteOffSet, val, true);
                byteOffSet += 8;
            });
        } else {
            failed = true;
        }
    });
});

// check if failed
if (failed) {
    return ({
        'status': {
            'type': 'ERROR',
            'message': 'Unsupported column type! Only DOUBLE, FLOAT, LONG, SHORT, BYTE columns supported for now!'
        }
    });
}

[FredrikKarlssonSpeech]

Yeah possibly as part of implementation of the feature mentioned in this GitHub issue. But can't promise anything...

Fantastic! The ability to make small adjustments on tracks is a really great feature, and if just any track by that is made an overlay on the spectrogram would behave exactly like the magic FORMANTS track does now, then the issue is solved.

And, I just want to say that I don't consider implementing editing of any track worth implementing. Personally, I cannot see a use case for it, at least, unless it is placed on top of a spectrogram for reference. If you need to check the formant tracker, you can overlay the formants onto a wide band spectrogram and redraw the formants. If you need to adjust pitch you would only be able to do that in a scientifically sound way if you place the f0 track on top of a narrow band spectrogram.

Just my view, of course, but I thought that I would point the problems of general editing of tracks (accidental editing which then needs to be handled, and of course what basis you could actually have for drawing a track freehand)

[FredrikKarlssonSpeech]

Could you also address an idiosyncrasy in the SsffParserWorker that the ssff2js() is able to convert columns from DOUBLE, FLOAT, LONG (despite an error message that suggests that LONG is unsupported), SHORT or BYTE but the jso2ssff() can only covert back to SHORT?

https://github.com/IPS-LMU/EMU-webApp/blob/f872cdc73c14b38d2c999545ae5f899bb5ee8934/src/app/workers/ssff-parser.worker.js#L423-L472

Possibly something like this?

jso.Columns.forEach((col) => {
  if (col.ssffdatatype === 'BYTE') {
      bytePerTime += col.length;
  } else if (col.ssffdatatype === 'SHORT') {
      bytePerTime += 2 * col.length;
  } else if (col.ssffdatatype === 'LONG') {
      bytePerTime += 4 * col.length;
  } else if (col.ssffdatatype === 'FLOAT') {
      bytePerTime += 4 * col.length;
  } else if (col.ssffdatatype === 'DOUBLE') {
      bytePerTime += 8 * col.length;
  } else {
      failed = true;
  }
});

// check if failed
if (failed) {
  return ({
      'status': {
          'type': 'ERROR',
          'message': 'Unsupported column type! Only DOUBLE, FLOAT, LONG, SHORT, BYTE columns supported for now!'
      }
  });
}

var byteSizeOfDataBuffer = bytePerTime * jso.Columns[0].values.length;

var dataBuff = new ArrayBuffer(byteSizeOfDataBuffer);
var dataBuffView = new DataView(dataBuff);

// convert buffer to header
var ssffBufView = new Uint8Array(global.stringToUint(headerStr));

// loop through vals and append array of each column to ssffBufView
var byteOffSet = 0;
jso.Columns[0].values.forEach((curArray, curArrayIDX) => {
  jso.Columns.forEach((curCol) => {
      if (curCol.ssffdatatype === 'BYTE') {
          curCol.values[curArrayIDX].forEach((val) => {
              dataBuffView.setInt8(byteOffSet, val, true);
              byteOffSet += 1;
          });
      } else if (curCol.ssffdatatype === 'SHORT') {
          curCol.values[curArrayIDX].forEach((val) => {
              dataBuffView.setInt16(byteOffSet, val, true);
              byteOffSet += 2;
          });
      } else if (curCol.ssffdatatype === 'LONG') {
          curCol.values[curArrayIDX].forEach((val) => {
              dataBuffView.setInt32(byteOffSet, val, true);
              byteOffSet += 4;
          });
      } else if (curCol.ssffdatatype === 'FLOAT') {
          curCol.values[curArrayIDX].forEach((val) => {
              dataBuffView.setFloat32(byteOffSet, val, true);
              byteOffSet += 4;
          });
      } else if (curCol.ssffdatatype === 'DOUBLE') {
          curCol.values[curArrayIDX].forEach((val) => {
              dataBuffView.setFloat64(byteOffSet, val, true);
              byteOffSet += 8;
          });
      } else {
          failed = true;
      }
  });
});

// check if failed
if (failed) {
  return ({
      'status': {
          'type': 'ERROR',
          'message': 'Unsupported column type! Only DOUBLE, FLOAT, LONG, SHORT, BYTE columns supported for now!'
      }
  });
}

Perhaps a bit off topic? Maybe you could make it a separate issue?

[samgregory]

Hi @FredrikKarlssonSpeech

Yes - sorry I did get a bit off topic didn't I.

I agree that the ability to make small changes to a track is needed. The kind of scenarios I want to fix for an f0 track are where the pitch tracker got lost and so the value is set to 0 and this just happened to occur at the 0.5 mid-point of a segment - very inconvenient. This is technically possible with the current build of Emu-webApp by setting the SSFF responsible for pitch tracking to "FORMANTS", making the corrections and then putting it back to it's original name once you are done.

My off-topic post pre-empted the next problem with this approach which is that the f0 SSFF is unlikely to be in the SHORT INT format. Files exported from VoiceSauce, for example, are FLOATS and so Emu-webApp throws the 'Unsupported column type' error when you try to save the corrections.