search

Lovrom8 / RSMods

383 stars 34 forks source link

Use Rocksmith as a guitar to midi converter #114

Closed stereokai closed 2 years ago

stereokai commented 2 years ago

This library is amazing!

It looks like you were really able to tap into the core of the game engine.

Do you think it's possible to output the guitar notes detected by Rocksmith as midi?

ffio1 commented 2 years ago

Hi!

The game parses every strum as a midi note (note that I said strum, and not note). This means that if you play a single note, you get the correct midi value out, but playing a chord does not give the same result. There are two mods that use those functionality, GuitarSpeak, and Show current Note.

GuitarSpeak listens to your guitar when you're in the menus and when you play a note you specified (like E2 | open low E string), we will press a defined key on your keyboard. This means that when you strum the note, you can move within the menus without taking your hands off of your guitar / bass.

Show current note is somewhat self explanatory. We just show the name of the midi note on screen.

As for outputting the notes played into a midi file, that is far outside the scope of this project.

stereokai commented 2 years ago

but playing a chord does not give the same result.

What result does playing a chord give?

ffio1 commented 2 years ago

Playing a chord gives whatever the last note the game heard. So when you play a chord, it will either show the highest note or the lowest note, though there may be some variations to that.

stereokai commented 2 years ago

Interesting. So what you're saying is that the game doesn't actually recognize chords?

ffio1 commented 2 years ago

Uh. No? The memory address we use to get the current note only gives us one note but it comes in midi notation so it's the best to use.

I don't know how the game actually does note detection, I wouldn't be surprised if they look for each note, but I can't say definitely that it's one way or another.

stereokai commented 2 years ago

Uh. No?

I reread what you said, "it will either show the highest note or the lowest note". You were talking about GuitarSpeak, but I'm asking about how the game works internally.

ffio1 commented 2 years ago

We think it's some variation of Fast Fourier Transform, where it looks for the notes to expect. The address of all the other notes might be in memory, but that's outside the scope of both mods and of this project since we don't need that much detail to press a key for the user.

My realm of knowledge on this topic is very small, so I don't want to say something that could be completely wrong.

However, you may find the recent article about the audio system in RS+ interesting: https://www.ubisoft.com/en-us/game/rocksmith/plus/news-updates/1Zcwvdcw0IwhBEYoCGCQtt/rocksmith-dev-diary-february-2022-audio-updates

stereokai commented 2 years ago

Thanks a lot! That was a very interesting read!