[Feature Request]: TrackEncoder

[Rmkrs]

Describe your issue in as much detail as possible

Would it be possible to get the opposite of the TrackDecoder, being a TrackEncoder.

I've been playing around with having custom playlists, that do not originate from a search done with LavaLink. I can populate all fields of a LavaTrack, but the only one I'm missing is the Hash. And LavaTrack's without the Hash will not be played unfortunately/expectedly.

So would be nice to have a TrackEncoder, with a method that returns a string (the hash) and accepts a LavaTrack. Said method would then build the hash based on the properties.

I saw that the DefaultAudioPlayerManager of LavaPlayer has a method to do this. But there are a lot of JS dependencies that I could not find a .NET equivalent for, and the MUTF8 spec used is pretty specific. So I've had no luck in creating a working hash myself using c#.

Calling the lavalink endpoint to do the encoding could also work of course, but that would be very slow compared to a local method. Especially if the playlist has a few hundred songs.

I would love to read your thoughts on this.

Many thanks.

Version

v5 (Latest)

Version

GitHub's latest

Relevant log output

No response

[Yucked]

Send me a rough concept of how DefaultAudioPlayerManager does it or where you say encoding being done, maybe I can find a way to replicate it.

[Rmkrs]

The DefaultAudioPlayerManager source can be found here

The encodeTrack method starts at line 243:

 @Override
  public void encodeTrack(MessageOutput stream, AudioTrack track) throws IOException {
    DataOutput output = stream.startMessage();
    output.write(TRACK_INFO_VERSION);

    AudioTrackInfo trackInfo = track.getInfo();
    output.writeUTF(trackInfo.title);
    output.writeUTF(trackInfo.author);
    output.writeLong(trackInfo.length);
    output.writeUTF(trackInfo.identifier);
    output.writeBoolean(trackInfo.isStream);
    DataFormatTools.writeNullableText(output, trackInfo.uri);

    encodeTrackDetails(track, output);
    output.writeLong(track.getPosition());

    stream.commitMessage(TRACK_INFO_VERSIONED);
  }

That was the easy part, but the DataOutput is the working horse here. And that's what I could not find a .NET equivalent for.

You can find the source code for the DataOutput class here

I think it's pretty well documented, but I just suck at bit shifting and endianness :/

[Rmkrs]

And I guess I suck at placing comments too without closing the issue immediately...

[RofLBotZ94]

Hello

I also thought about having a track encoder, as I too had been having some cases lately, where Lavalink doesn't seem to be able to find some tracks that I want (may be that I am just not using/setting it properly), so I came here to suggest it being implemented as well and stumbled on this issue.

I tried implementing something myself and it seems to be working, at least with a few dozen tracks I've tested it out with. Here's what I came up with, with comments on the more relevant parts:

JavaBinaryWritter.cs

using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Text;

namespace Victoria.Encoder {
    // Based on JavaBinaryReader struct but for writting and loosely follows the concept of a stream
    // with the main difference being that instead of overwriting the previous contents (if there are any)
    // of the current write position it just shifts the contents to the right of it, kind of like the "Insert(Range)" method from a List 
    internal ref struct JavaBinaryWriter {
        private readonly Span<byte> _bytes;
        private int _currentWritePosition;

        public int Position // Current write position
        {
            get => _currentWritePosition;
            set
            {
                if (value < 0) {
                    throw new ArgumentOutOfRangeException(nameof(Position),
                        "An attempt was made to move the position before the beginning.");
                }

                // If we try to set the position past the length set it to the length
                // otherwise set it to the given position
                _currentWritePosition = Math.Min(value, Length);
            }
        }

        // Amount of bytes written
        public int Length { get; private set; }

        public JavaBinaryWriter(Span<byte> bytes) {
            _bytes = bytes;
            _currentWritePosition = 0;
            Length = 0;
        }

        // Write a string
        public void Write(string value) {
            var bytesToWrite = Encoding.UTF8.GetByteCount(value);

            // Write the length of the string in bytes as a short
            Write((short)bytesToWrite);

            // If we are writing to a position that already has contents, shift the existing contents to the right
            // of the current write position by the amount of bytes we need to write
            if (Position < Length) {
                _bytes.Slice(Position, Length - Position).CopyTo(_bytes.Slice(Position + bytesToWrite, Length - Position));
            }

            // Write the UTF8 encoded string to the "stream"
            Encoding.UTF8.GetBytes(value, _bytes.Slice(Position, bytesToWrite));
            Length += bytesToWrite;
            Position += bytesToWrite;
        }

        // Write to the "stream"
        // Note: Should be a primitive type!
        public void Write<T>(T value) where T : struct {
            // Get the byte contents of the value we want to write to the stream as a span
            var bytes = MemoryMarshal.AsBytes(MemoryMarshal.CreateSpan(ref value, 1));

            // Ensure it's Big Endian
            if (BitConverter.IsLittleEndian) {
                bytes.Reverse();
            }

            Write(bytes);
        }

        public void Write(Span<byte> bytes) {
            // If we are writing to a position that already has contents, shift the existing contents to the right
            // of the current write position by the amount of bytes we need to write
            if (Position < Length) {
                _bytes.Slice(Position, Length - Position).CopyTo(_bytes.Slice(Position + bytes.Length, Length - Position));
            }

            // Write to bytes to the "stream"
            bytes.CopyTo(_bytes.Slice(Position, bytes.Length));
            Length += bytes.Length;
            Position += bytes.Length;
        }

        // Just how it works with streams essentially
        public int Seek(int offset, SeekOrigin origin) {
            return Position = origin switch {
                SeekOrigin.Begin => offset,
                SeekOrigin.Current => Position + offset,
                SeekOrigin.End => Length + offset,
                _ => Position
            };
        }
    }
}

TrackEncoder.cs

using System;
using System.Linq;
using System.Text;

namespace Victoria.Encoder {
    public readonly struct TrackEncoder {
        // Stuff required for the encoding
        private const int TRACK_VERSIONED = 1;
        private const int TRACK_VERSION = 2;

        // Takes in a Track and sets its hash property in place
        public static void Encode(LavaTrack track) {
            Span<byte> bytes = stackalloc byte[GetByteCount(track)];

            var javaWriter = new JavaBinaryWriter(bytes);
            javaWriter.Write<byte>(TRACK_VERSION);
            javaWriter.Write(track.Title);
            javaWriter.Write(track.Author);
            javaWriter.Write((long)track.Duration.TotalMilliseconds);
            javaWriter.Write(track.Id);
            javaWriter.Write(track.IsStream);
            javaWriter.WriteNullableText(track.Url); // Extension method
            javaWriter.Write(track.Source);
            javaWriter.Write((long)track.Position.TotalMilliseconds);
            javaWriter.WriteVersioned(TRACK_VERSIONED); // Extension method

            track.Hash = Convert.ToBase64String(bytes);
        }

        // Calculate the number of bytes needed to encode the track
        // in part to make sure we allocate just the right amount of bytes
        private static int GetByteCount(LavaTrack track) {
            // The value 23 was derived from:
            // 8 (a long for position) + 8 (a long for duration) + 4 (an int representing the "versioned")
            // + 1 (a byte for the version) + 1 (a bool indicating if it's a stream) + 1 (a bool that will be written before the url)
            // = 23 bytes.
            //
            // After that we need to calculate the number of bytes needed for the string properties (except of course the "Hash" property)
            // plus 2 for each of them, since every string has its length prepended to it encoded as a short (2 bytes)
            return 23 + typeof(LavaTrack).GetProperties()
                .Where(p => p.PropertyType == typeof(string) && p.Name != "Hash")
                .Sum(p => 2 + Encoding.UTF8.GetByteCount(p.GetValue(track).ToString()));
        }
    }
}

Extension methods placed in VictoriaExtensions.cs

        internal static void WriteNullableText(this ref JavaBinaryWriter writer, string value) {
            var isValidString = !string.IsNullOrWhiteSpace(value);

            writer.Write(isValidString);

            if (isValidString) {
                writer.Write(value);
            }
        }

        internal static void WriteVersioned(this ref JavaBinaryWriter writer, int flags) {
            var value = writer.Length | flags << 30;
            writer.Seek(0, System.IO.SeekOrigin.Begin);
            writer.Write(value);
            writer.Seek(0, System.IO.SeekOrigin.End);
        }

Usage

var track = new LavaTrack(/*fields*/)
TrackEncoder.Encode(track);
// Now "track" has the hash

I placed both the JavaBinaryWritter.cs and TrackEncoder.cs in a new folder called Encoder, just like the namespace says. I also opted to set the hash in place instead of returning it as a string, because the Hash property can only be set internally or through the constructor, so I thought it would be a bit convulated to just return the string, since it would imply that first we would need to create the track, call the encode method on it, and then create a new track to set the hash through the constructor. When it comes to the MUTF8 from the JNI, from what I gathered by looking it up, C#'s UTF8 encoding should work just fine in the vast majority of cases, either way I reckon we can figure out how to implement their MUTF8, if it's necessary.

It's probably still a bit rough around the edges and I haven't really tested it extensively but so far from the few dozen tracks I mentioned that I tested, they all work and matches 100% with the hashes from the tracks that Lavalink returns when searching for them. It's also possible that @Yucked already implemented this and better and will commit it soon, since this issue has already been up for quite a while.

[Yucked]

Sheeeessh, do you want to open a PR. I'm ready to merge it and push it out today @RofLBotZ94

[Rmkrs]

Nice work @RofLBotZ94 :)

[RofLBotZ94]

I just wanted to post it here to see what you thought about it first and since your last commit said something about working on major/minor, it could be that you were making some significant changes and maybe this could break something. Or like I mentioned before, you could have already implemented it. But anyways, if it's good enough for you, it's good enough for me :). I will send it in just a minute.

[Yucked]

Thank you so much