Newer Lipsync animations bug eyes

[C0Assassin]

With the lipsync update, the mouth tracking has been done amazingly well. Not even the C3/Magma one can compare.

However, the update came at a cost. Anytime I render a new lipsync file and insert it into the game, I notice the character's eyes are in a 'default wide eyed' state. It is horribly uncanny looking, and looks bugged when in close ups. It didn't use to be like this in the previous version.

Peering into the expression sheet in the song.lipsync file, I can see that the newer versions typically start with: Cage_hi Cage_lo Church_hi Church_lo Earth_hi etc etc

Which is all necessary to track the phonemes correctly. However, peering into the older versions (and some of the ones created thru Magma/and the official Harmonix ones), I can see there's expression calls for eyes and eyesbrows. As an example: Blink Brow_aggressive Brow_down
Brow_pouty Brow_up Bump_hi Bump_lo

At the very least the older version in Onyx would at least put Blink and Brows, so I'm unsure as to why the newer ones always render without any of them. I didn't know whether it was a bug or if it was intentional, as if it was necessary to get a better mouth track or something.

Either way, I thought it was something that could potentially be addressed in the future.

[DioGiorno]

my question is how do you make lipsync works? I finished my lipsync stuff, was it suppose to be in the same midi track as charts? or the lipsync had to be its own midi file? do I convert the song into an con file or a live file?

[C0Assassin]

Go under lipsync tab in Onyx, give it the midi, and press the Turn Vocal tracks into LIPSYNC* tracks for RB3 button.

Now open the midi up again in Reaper, there is a new Lipsync midi track(s). All the automated mouth movements should be there already. Just paste your lipsync stuff on top of it, so that theres both eye movements or whatever along with the mouth movements.

After that save and export the midi, you feed it back to onyx, and press the Update RB3 milo button. You can extract a milo using c3 tools, just by dragging the con into c3 tools and digging into the con explorer. Once you extract the milo and update it, you can then just put it back in the con using the replace function; just like you did with extract.

[DioGiorno]

the files in the song folder is the same layout as clone hero, so I don't get the milo thing at all, I don't use Magma to convert, I just use Onyx to convert to RB2con. I don't have a milo, i just make one and update it but this is confusing me.

[DioGiorno]

I couldn't find replace button on extract

[C0Assassin]

Every song has a milo, otherwise it'd probably crash in game. During the conversion Onyx generates a milo for the song.

Drag your RB2con into C3 tools, and a con explorer window should appear. Go to contents, and there you should see your song's files.

Extract and replace are in one context menu, shown in this picture. https://cdn.discordapp.com/attachments/336311461506711554/881654919034187816/unknown.png

[DioGiorno]

so what do i do after replacing those? is there save button or something? sorry i didn't get back at ya because i was at work

[C0Assassin]

https://cdn.discordapp.com/attachments/336311461506711554/881729454064414740/unknown.png

Save button is back on the informations tab of the con.

[DioGiorno]

i think i mess it up, the xbox froze when i load up the song. so do i make new milo from onyx, use the notes.mid to update the milo, was the lipsync midi is suppose to be seperate than charts?

[C0Assassin]

Your reaper project should have everything in it.

https://cdn.discordapp.com/attachments/336311461506711554/881731892402094082/unknown.png

Your project needs to look like this. Once you put the text events from the Audrey track into the lipsync track, you export the midi and then put this one into Onyx, then update the milo.

[DioGiorno]

ok so, after i convert the song into con but how do i update new milo that i don't even have a milo yet, do i make rb3 milo from milo tab on Onyx?

[C0Assassin]

Yes, you can generate a new milo with that button. It'll use the midi you gave it (with the updated Lipsync1 with the audrey notes) and put it into a fresh milo, which you can rename to match your song's filenames.

[DioGiorno]

it's still freezing my xbox

[C0Assassin]

Honestly a brand new converted con by Onyx should already have a milo included, are you positive yours doesnt have one

[DioGiorno]

sorry for the longest replied possible but yes, but it still makes buggy eyes and lipsyncing animation isn't still good

[DioGiorno]

so here's what I did, my song folder contains album.png/guitar.ogg/notes.mid (with lipsync track)/song.ini the normal Clone Hero Song type

s

I did add lipsync button before this. This is my tracks look like

then I convert that song folder in RB2Con (batch process) and then load it in my Xbox 360. It still have same effects, still bugged eye, the lipsyncing still looks terrible.

Updating Milo thing is still confusing me because I check on the con, it has milo already in it so I don't know how to really update it

[C0Assassin]

The bug eye is fine, it just means you have no expressions or lipsync visemes added. The lipsync is in its default state.

When you add the lipsync events to the midi via onyx, you need to then edit the midi via reaper. You'll see the Lipsync1 midi track and in it, you'll see a lot of text events that look confusing.

These are mouth viseme events that onyx generates. Go ahead and add a new text event near the beginning of the song in the lipsync1 track, and write "Brow_aggressive 200 hold"

Now export this new midi, and load it into reapers lipsync tab again and update the milo with it. Then send the milo back into the con. And check the song again, you're going to see a very angry expression in your game for the song.

[DioGiorno]

So I did export the new midi, and you said "load it into reapers lipsync tab again" are you talking about inside Onyx or Reaper? And also, was I suppose to convert the song into con first, then extract the milo and then update the milo with the new midi? can you sent the screenshot on how you update it then replace function or am I puzzled as ever?

[C0Assassin]

Sorry, when I typed "load it into reapers lipsync tab again" I meant Onyx. Load the new exported midi into Onyx's lipsync tab and update the milo you extracted from the con. Then put updated milo back into the con.

You do not touch the midi in the con, ever.

[DioGiorno]

I GOT IT WORKING! Angry expression & lipsyncing is so much better. Thank you so much man, I should get into facial expressions more since I had audrey script on Reaper. You got any good tips on facial expressions?

[DioGiorno]

Like eye closing when they scream or something that can mix together?

[C0Assassin]

There's no real documentation on how to use the lipsync expressions properly, so everything that I've gathered has been my own personal notes.

Some of my notes:

Blink to lower eyelids down. 255 = Lowest point (Eyes Close) 0 = Highest point (Eyes Open)

Most events blend into the next, kinda like lighting calls. Need to place the same event right before a new event to get a snappier change. !!Edit: Use Hold parameter to create sustains, for both 'held' emotions and 'clearing/0ing' them out Use Ease parameter to blend the emotion into the next call.

Example: [Brow_aggressive 0] [Brow_aggressive 128] [Brow_aggressive 0] -- Don't Do [Brow_aggressive 0 ease] [Brow_aggressive 128 hold] [Brow_aggressive 128 ease] [Brow_aggressive 0 hold] -- Do

Blinks do not need this since they're so quick. [Blink 0] [Blink 255] [Blink 255] [Blink 0 hold] is enough to suffice a good blink.

Do not mix Blink 255 with Dramatic Pouty 255, at most use Blink 10. Eyes will clip out of sockets.

Do not mix Blink with Squint 255, remove the blink parameter altogether in the place. Eyelids will wrap downwards and over innercheek.

Rocker Smile Mellow gives you a nice soft smile, think Brown eyed Girl.

Brow Pouty 255 is good for really sad looking emotions. Avoid mixing Brow Pouty and Dramatic Pouty, its usable and creates very sad or pouty expressions, but Blinks will be in an unusable state. Only use if blinks aren't necessary. (Eyelids will clip into upper cheek)

[DioGiorno]

so the numbers like 255 (close) or 0 (open) would be the number after the name like [Brow_aggressive 0]?

[DioGiorno]

so which one that will make the eyes closed, is it the number after the name or is it blink itself?

[DioGiorno]

after i turn blink to close, do I had to make it back to 0 after closing the eyes?

[C0Assassin]

The number/velocity determines the weight of the expression.

The higher the number, the more prominent the expression. The lower the number, the less prominent. Also blink is in the notes.

Blink 255 is full close, Blink 0 is full open. I even gave examples in the notes. Read it carefully.

To close eyes: [Blink 0] [Blink 255]

To open eyes: [Blink 255] [Blink 0]

To hold eyes closed: [Blink 255 hold]

To hold eyes open: [Blink 0 hold]

[DioGiorno]

thanks bro