[Feature request] Ability to move camera

[mariomadproductions]

Feature request: Ability to move/rotate camera (and optionally extend boundaries of screen). The former feature is known as "free look" in Dolphin.

[zeromus]

This is more complex than it is in dolphin.

1. There's no dedicated sense of a perspective matrix which we can use as a hint for when to modify the camera, so we would require more advanced logic to try to distinguish the projection type. (although this may not be extremely advanced logic, as you will see later)
2. We can't use the mouse for looking all around because it would interfere with the normal gameplay. Don't say "Oh, you know, just dragging on the 3d screen is freelook and dragging on the touch screen is touch input", of course it isn't that simple.
3. Using the mouse would also be hard to do portably due to desmume's bolloxed up frontend<->core architecture; using hotkeys would be hard too, and not pleasant at all really
4. Extending boundaries of the screen!!!!!!! LOL!!!!! not gonna happen. It's too deeply woven with the compositing with 2d.
5. I don't want to put any of this junk in the already overburdened codebase with 1/20th of the maintainers dolphin has

So, what I decided to do is make it require use of a lua script. This way people can work out gizmos to control when to activate freelook, and libraries to determine which matrices to modify, and how.

I'll attach a minimal demo script, but here's how it works in a nutshell (see attached script for more info on the parameters)

1. emu.set3dtransform(2,matrix) will cause the given matrix to be combined with the game's
2. emu.register3devent(callback) will get called whenever the game changes the projection matrix. the newly set matrix is in _G.registered3devent_arg

I tested this in FF3 and mario kart and it worked as expected. In pokemon heartgold it worked OK outside, but inside it seemed flaky, I think due to a really eccentric camera setup. FF3 was an interesting case because I did need to detect the ortho matrix there or else the GUI would go nuts.

In case you're confused about what's going on when you test it in some other game first: It's supposed to slowly swivel the yaw and pitch of the camera, to kind of "look around" the scene.

As for "extending boundaries of the screen": I suggest you increase GPU scaling and make a major lua hack to squish the game into your HD aspect ratio and drop letterbox bars on top of it, and hope the game doesn't put any 2d engine gui on top synchronized with the position of 3d graphics.

future work: we may need to control whether the modified freelook matrix is used for box tests, because some games may be using those for object culling. However, many more games should be using software CPU geometry checks for the same thing, and there's nothing we can do about that.

freelook.lua.txt

(By the way, this is a really shoddy interface and I reserve the right to redo it once we realize more how it should work.)

[windwakr]

The modified matrix persists when the Lua script is stopped and the game is closed/changed.

[zeromus]

Does drawn lua graphics persist when the script is stopped, unless intentionally cleared?

[mariomadproductions]

Thanks for making this lua script. Even if its limited, its still useful to a degree.

[zeromus]

You're supposed to improve the script so it's not so limited.

[huckleberrypie]

Can anyone shed some light into this? Been trying to get freelook to work, but I couldn't get the LUA scripting feature to work as even if I copy lua51.dll to the exact same folder where Desmume resides, it is still complaining that the DLL is missing.

[zeromus]

use process monitor to discover which files are being looked for but can't be found; make sure you're not using the mismatching 32/64 size of dlls for the build of desmume you're using (that will make it say it can't find the DLL file just as if it were missing)

[huckleberrypie]

Never mind, looks like using lua5.1.dll from "lua-5.1.5_Win64_bin.zip" which I downloaded off Sourceforge did the trick.

[windwakr]

EDIT: @ everyone, please don't reply to discuss this script. This is not the place for that. Take any comments to https://forums.desmume.org/viewtopic.php?id=11217

@mariomadproductions Don't know if you've got your own thing working, but here's the script I've been toying with. Honestly don't know what I'm doing. Maybe I'm going about this the wrong way and overcomplicating it, but it's getting results. -see thread linked above-

Some webms: https://files.catbox.moe/zfni2x.webm https://files.catbox.moe/0fn3dt.webm

You'll need a controller like an xbox 360 controller with two sticks and analog triggers to fully use the script. Or you can attempt to modify the controls in the script.

You'll want to set the pitch offset before exploring around. Especially if the camera starts angled downards. It's done by aiming at the horizon and hitting X on your controller(or equivalent on w/e controller you use). Here's a webm showing what you get when you just start moving around without setting the pitch offset: https://files.catbox.moe/mmbcxh.webm And after: https://files.catbox.moe/s8hev7.webm

This isn't a universal solution, but it works for most games.

[huckleberrypie]

@mariomadproductions Don't know if you've got your own thing working, but here's the script I've been toying with. Honestly don't know what I'm doing. Maybe I'm going about this the wrong way and overcomplicating it, but it's getting results. https://my.mixtape.moe/ptuobt.zip

Some webms: https://my.mixtape.moe/tkxvho.webm https://my.mixtape.moe/ypdlzx.webm

You'll need my fork here that adds the joystick analog sticks to the lua controller interface: https://github.com/windwakr/desmume/tree/luajoyaxes Or you can attempt to modify the controls in the script.

You'll want to set the pitch offset before exploring around. Especially if the camera starts angled downards. It's done by aiming at the horizon and hitting X on your controller(or equivalent on w/e controller you use). Here's a webm showing what you get when you just start moving around without setting the pitch offset: https://my.mixtape.moe/cuwjgu.webm And after: https://my.mixtape.moe/hwtkso.webm

Could you do one for controlling the camera using the number pad?

[windwakr]

@zeromus Would you know why some scenes in some games have polygon breakup as I move around using this feature? It's like some of the polygons have a delay before being moved. Here's a minor example of what I'm talking about. It starts about halfway through: https://my.mixtape.moe/wvybuo.webm When I stop moving they stop being broken. And if I just zoom out and move my character around in game there is nothing wrong when the game itself moves the camera. It gets really bad in some games.

[stranno]

This isn't a universal solution, but it works for most games.

I'm getting 105: bad argument #1 to 'abs' (number expected, got nil) with your script. Latest build wth LUA controller support.

[zeromus]

@windwakr I don't immediately know why that happens. Send me an in-game save at that point in FF4HOL and I'll investigate it

[windwakr]

That's actually Final Fantasy 3. I've noticed in the time since I posted that turning off the JIT mostly solves the issue. It is still there somewhat in FF3 when it's off, but the issue disappears completely in most other games that were affected. Final Fantasy III.zip

[zeromus]

@stranno @windwakr I too have problems with the script; however as requested I'll take it to the forum

[zeromus]

The jit involvement in this was really weird. The actual problem was just a conceptual error of some kind... I can't figure out what I was thinking, but I had to fix the script to make more sense. The basic problem was, parts of the geometry would draw with different matrices or in different "frames" and the script was doing something inbetween "frames". I put "frames" in quotes to call out that it's a desmume "frame" which is the LCD frame that governs all the desmume and lua timing but NOT the 3d frame. Anyway, a proper freelook script should be governed entirely by 3d events, I believe. I'll attach a new script that works that way without the ff3 problems.

Turns out FF3 is using an ortho matrix for the world, so that needed fixing. I'm adding a thing you can use to get game codes so you can put per-game hacks in your script. Unfortunately, in order to properly transform the world, the HUD gets transformed too--annoyingly, it draws with the same matrix. This means, for the best results, I may need to add something that lets you adjust the matrix per vertex, so that you can detect (in this case) verts drawn at a particular Z value and use an unmolested matrix.

Thinking about this also made me realize we could make logic to synthesize vertices from lua too--that is, you could draw stuff using in-game resources. The reason for that would be so you can batch certain things as the game draws them, and then draw them yourself (transforming them, or not, any way you wish). In essence this is an even more flexible approach.

Let me describe it from scratch again in a more succinct way: what if we made lua into a vertex shader?

freelook2.lua.txt

[windwakr]

I think the problems we're seeing are different. I don't use the 3d event callback at all in my script and only adjust the matrix between frames.

a proper freelook script should be governed entirely by 3d events, I believe

Try your new script out in games like Mario Kart DS or Super Mario 64 DS. The callback is only called a couple of times when entering a new area and then never again and you don't get any camera movement.

But like I said in my last post, all of the issues I had with my script before have gone away with the JIT being turned off. Compare these two webms of Zelda Spirit Tracks. Notice in the first what happens after I bring up the game's menu then see how that doesn't happen in the second, which has the JIT disabled. https://my.mixtape.moe/acieiz.webm https://my.mixtape.moe/cqmfgx.webm I don't care about the hud elements in some games moving around with the camera so I'm satisfied with how it is now, tbh.

[zeromus]

The jit can't do this unless youre mixing up lcd and 3d frames. I'm 100% sure. I might believe it for one game due to a freak coincidence of timing something like a box test, but not more than one game. I haven't inspected your script in detail.

As for MKDS, when done the "right" way using 3d frames only, the problem is caused by the projection never getting set again.

I've decided maybe the next to last entry is best for determining ortho vs perspective, but clearly I'm not an expert.

freelook3.lua.txt

[windwakr]

OK yeah, I see what you mean now about the "frames".

Honestly don't care to update my script, though. It's a disgusting hacked together mess and I don't feel like toying with it anymore.

[bnthomason]

Hey, it appears freelook is broken in some of the builds. Maybe it's just my pc, but whenever I activate it, the camera will go in circles. I've tried modifying the script, but the camera still goes in circles.

https://www.youtube.com/watch?v=R15ryZQcehk&feature=youtu.be

[zeromus]

There is no possible way windows changed something which affected this. It's purely done in software. Especially if you're using softrasterizer, but anyway, no, there's no possible way, in any case.

In principle the freelook depends on the game and even the scene; that's why it's done by lua and not built into desmume. The most likely explanation is you have forgotten what you did in windows 8 and mistakenly believe something's changed between then and now; but nothing's changed except the validity of your expectations. You may need to revise the lua script for your particular application, because in general you MUST revise the lua script for a particular application.

[windwakr]

That game is Fossil Fighters? I've loaded it up and freelook works fine, although I wasn't able to test it at the location you show in the video. It looks like the script's detecting your right stick as being held to one side. Try restarting your computer or unplugging and replugging in your controller. If you have more than one controller plugged in, try to unplug all of the other ones.

[Malik177]

can there be a keyboard version of this free look script be created? I would love have the ability to move the camera around in certain ds games. I can't use an xbox controller since i don't have an xbox nor the emulator works with nintendo switch pro controller either.

[Malik177]

is this coment section dead?

[zeromus]

are you going to keep pinging it trying to wake the dead?

[Malik177]

Well not really it would look stupid for me to do. But, I'm guessing no one is going to make a keyboard compatible version of this lua script. Is there any chance you or windwakr can make this lua script? I'm sure over 200 people would enjoy having the ability to move the camera around. Plus, I really want to have the chance to exprience it

[Malik177]

I would like to freely move the camera around while playing mario and sonic at the 2008 olympic games for the ds

[zeromus]

fixing the script to use keyboard is middle school level computer programming homework assignment. ask on a gamer's forum. or mow lawns until you've saved up enough to buy a gamepad.

[huckleberrypie]

fixing the script to use keyboard is middle school level computer programming homework assignment. ask on a gamer's forum. or mow lawns until you've saved up enough to buy a gamepad.

Based comment. Will someone plz close this issue tho?

[zeromus]

why

[huckleberrypie]

why

idk, you're wasting time on that guy.

[zeromus]

nah. we were all that guy once. and look at us now.

[windwakr]

@Malik177 https://forums.desmume.org/viewtopic.php?pid=31042#p31042

[Malik177]

thanks windwakr do I have to put number values were the -- marks are at to make it work?

[windwakr]

do I have to put number values were the -- marks are at to make it work?

The controls are the buttons listed in the quotes. U, J, H, K move the camera around, numpad 8, 5, 4, 6 rotate the camera, etc. The -- just indicates the start of a comment in the programming language used here, which is Lua.

For a list of valid key names check this page near the bottom https://tasvideos.org/LuaScripting/TableKeys

[Malik177]

alright thanks