Physics and position types / SpriteBuilder

[vlidholt]

Physics needs to play nicely with SpriteBuilder and position types. This needs to be decided how to go about fixing.

SpriteBuilder & Cocos2d support different position types to layout interface objects, but also to scale up game play for phones vs tablets. This allows users to use the same logical coordinates when doing their game, by just changing the position type to CCPositionTypeScaled their whole game play will work on both phones and tablets. Currently this doesn’t work with the physics and this is something that really need to figure out how to do (physics simulation should be the same on phones and tablets).

I created a very simple sample project to illustrate how the you can use the scaled positions to scale up your gameplay. The physics simulation works perfectly on phones, but is scaled on tablets. I uploaded it to Google drive here (it also includes a compatible version of SpriteBuilder, you can preview in SpriteBuilder and run the Xcode project on iPhone/iPad): https://drive.google.com/file/d/0BzFSRgoEwrfXOWRmc010T2NFRGc/edit?usp=sharing

[slembcke]

Position types are implemented, but are not dynamic (nor should they be). Scale types are not implemented since it's a bad idea to rescale the size of physics objects ever.

[vlidholt]

Agreed, the physics should be the same on phones and tablets. That's the whole idea. Did you have a look at the samples?

[vlidholt]

Also, a reasonable limitation to a solution could be if it only supported the absolute and scaled position types – it could even assume that all children has the same position types. Physics are generally used for the gameplay so this should cover 99% of use cases.

[slembcke]

Ok. So I've been mulling on this for a few days.

The only viable solution I could think of was what we had talked about before, adding a scale factor to all of the inputs and outputs of the physics. This has significant downsides and I'm still really unhappy with it as an option:

• There are a lot of inputs and outputs to worry about. Shape/joint/body construction and positioning, queries, callbacks, etc.
• It adds a third coordinate system to the mix. (SpriteBuilder's, Cocos2D's and the scaled physics) Because of the way that content sizes and repositioning work, this seems very overcomplicated.
• It makes it that much harder to write correct, custom Objective-Chipmunk code. (It's already more complicated than I wanted it to be since so many operations need to be deferred)
• Determinism: Not something that is encouraged for anything using floating point numbers, but adding in the scaling factor will lose some determinism.

It's currently hardcoded that tablets have a hard-coded 2x position scale factor. This applies to everything instantiated by SpriteBuilder, but not CCNodes created in code using default settings. So there are two different coordinate systems involved, and the SpriteBuilder coordinates don't actually match the content sizes. I had to dig around quite a bit in the code to even figure out how this was working at all. It was especially confusing since what SpriteBuilder calls absolute coordinates is actually scaled coordinates in the code.

What if as a compromise between our two ideas, the SpriteBuilder tablet scaling is implemented using a projection matrix instead. That way the SpriteBuilder coordinates will match content sizes and regular Cocos coordinates. It will also match the physics coordinates. Instead of 3 coordinate systems, there is only 1. It doesn't rule out using the other positioning types such as the normalized or inset ones (which should work correctly now), but SpriteBuilder doesn't need to rely so heavily on the scaled coordinate type.

The changes necessary to make that work would be to update the projection for SpriteBuilder projects, and remove the interdependency of content scale and the existence of "retina" screens. (Which I've also argued doesn't really make sense anymore anyway.)

[vlidholt]

I've been also thinking a bit on how to best solve this. First some notes on things that may be misconceptions.

1) With v3 Cocos2d and SpriteBuilder uses the exact same coordinate system. 2) The 2x positionScaleFactor is not hard coded, you can set it either in your SpriteBuilder project settings (this may not have been implemented yet though, need to check!) or programmatically in your code using the CCDirector. By default the value is 2x for tablets though. 3) For many game studios, designing for both phones and tablets require much more than just scaling things up. E.g. buttons look big and ugly on an iPad if they are designed for iPhone and vice versa. Depending on the game this may or may not be limited to the user interface. Having options for positioning provides a huge advantage over competitors that hasn't got solutions for handling different device types.

Using a projection matrix to scale things up, is much less flexible than the current positioning options. Also, it will require updates on very many levels (e.g. textures need to be scaled down when higher resolution assets are used) which will require work throughout the whole pipeline and make things more complicated everywhere (with the exception of the physics implementation).

As I see it, there are two options that may be acceptable:

1) Adding a scaling factor on the physics. This would potentially add some extra complexity, and also potentially lose some determinism. It's the most complete solution though. 2) Impose some restrictions on the nodes with physics bodies below a physics node. For instance say they can only use position and rotation (and maybe scale?). They all have to use the CCPositionTypePoint or CCPositionTypeScaled and they all need to use the same type. This way it would be possible to just calculate the positions and rotations and then set them, the physics engine can safely use the same coordinates regardless of which type were used. Exactly the same coordinate systems are used for phones and tablets. The downside of this solution is that there are more restrictions to which position types can be used and also restrictions to which transformations are allowed. For almost all kinds of games this would be sufficient though.

[slembcke]

1) They don't use the same coordinate system. Regular Cocos nodes by default use your "points" positioning type. As far as I can tell, it's not possible to use this positioning type from SpriteBuilder since the "absolute" position type in SpriteBuilder maps to the "scaled" position type in Cocos. The content size of sprites do not match their scaled positions set from SpriteBuilder. This is very non-obvious too until you start digging into the code. The SpriteBuilder and Cocos terminology doesn't match and until you notice that SpriteBuilder sets the scaled positioning type for everything it instantiates, it's not obvious why the coordinate systems are different. 2) What is this if not a hard coded value being set into a global variable? -> https://github.com/cocos2d/cocos2d-iphone/blob/develop-v3/cocos2d/Platforms/iOS/CCDirectorIOS.m#L118 It's not even a per-scene value, it's a global. 3) What stops somebody from using your other positioning modes when using a projection matrix. I'm only saying to drop the scaled position type. What downside is there for that?

The thing is that Cocos already does all of the resolution handling stuff you mention, but it's hardcoded to be tied to whether or not the device is "retina" or not, which at this point is almost entirely worthless. Non-retina iPhones are rare, and expanding into the Android universe there is no convenient 2x scale factor to take advantage of. I guarantee you it's not complicated to break Cocos2D's dependence on "retina" to set the scale factor of loaded textures since I've done it a handful of times for various projects. The annoying part is that the method to do so has changed with nearly every release of Cocos since that code keeps getting refactored.

1) It adds a lot of complexity in both the implementation and to anybody using it, and it's not even close to being the same solution. I can absolutely, 100%, guarantee you that changing the projection and content scale values will be an order of magnitude simpler. I have actually done this more than once because it's not that hard of a change to make. Then we won't need 3 separate coordinate systems just to make things work. 2) The problem is not what position types are used. The problem is that on an iPad, the scaled position type means that the object is actually twice as big, and the coordinates used for everything are twice as big. The position type only applies to reading or writing the position property so it appears to be in a different coordinate system that it really is.

[Birkemose]

I tried the Physics test, and I can not get them to behave identically. Wasn't that the idea?

In the universal physics apps I have made so far, I have always used the exact same coordinate system for physics. It simply doesn't work if you don't. With chipmunk as a separate package, it was fairly easy to scale the physics data to the screen, but I can see the problems if chipmunk is an integrated part of cocos2d.

I guess we haven't thought that properly through.

Anyways. It makes little sense to create a coordinate system which satisfies UI first, and physics second. It should obviously be the other way around. Personally I am not too happy with the added complexity in CCNode. Our design goal was to simplify CCNode, so I would say that goal hasn't been met so far.

I have previously proposed a custom projection, where the user can set whatever projection he feels like. I guess that would fix the physics scaling. Am I right on that?

[vlidholt]

Lars, you are completely right about the physics coordinate system should be exact the same for tablets and phones. This is the point I am trying to address. However, no matter how we twist things the physics coordinates and the Cocos2d coordinates will never be the same. This is because they inherently works differently, the physics is global and needs to be and Cocos2d is using a node graph.

The new positioning allows the user to use logical coordinates that can be (optionally) scaled up on tablets. It currently works super well with both UI and game logic. For many games and game studios I have worked with when constructing the positioning (Zynga/PocketGems/Wooga) just scaling things up is not good enough and the UI is a very large part of their games. I start to feel like an old broken record, but not having good working solutions for making the UI look good on different form factors may be a deal breaker for these companies.

So, how do we make the physics work correctly and the same for the different form factors?

1) Have the physics work in it's own coordinate system that is always the same. 2) Add some limitations on the nodes below a physics node (if they have a physics body attached). The limitations would be that they can only use the position and rotation property. The same goes for any node in-between the physics node and the node with the body. These are the properties affected by the physics, and should be sufficient for almost all physics games. To enforce this we could potentially add asserts. 3) Use custom transformations between the physics node and the physics bodies. For these only the value of the position and the rotation properties are used. Kind of a lightweight transformation matrix that doesn't take any of the fancy positioning into account. (It's possible the anchor point needs to go somewhere in here though.) 4) When changing the nodes' positions, just set their position and rotation properties.

Pros: Very easy to implement (by far the easiest solution presented so far), would work with the fancy positioning for the point and scaled positions. Hence, it would work on both iPhones and iPad with the same simulation. We could use the area property of physics bodies. Cons: Doesn't work with relative position types. Doesn't work with transforms such as scale and skew for nodes below the physics node and which are simulated.

Potentially, this could be altered to also work with the scale property, but if it sacrifices the area property of physics bodies I personally think those are more important.

[slembcke]

Working examples are useful, so I implemented what I've been describing, and it's even gotten a little easier than it was in the past: https://github.com/cocos2d/cocos2d-iphone/compare/projection-scale

Now to enable retina mode you do it the regular UIKit way instead of calling "enableRetina" on the director later. Cocos2D simply uses the glView as it's configured.

glView.contentScaleFactor = [[UIScreen mainScreen] scale];

If you want to double the content scale on iPads so that the coordinates more closely match those of the the iPhone screen, you simply need to double the content scale. Then the iPad will act as though it has a 512x384 point screen with a 2x content scale or 4x content scale on iPad retina.

if(UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPad){
    director_.contentScaleFactor *= 2.0;
}

This is all functionality that existed in public Cocos2D methods before, but changing the values had somewhat disastrous results since there was some hardcoded dependence on only allowing a 2x content scale, and it being tied to having a retina screen. All I did was remove those dependencies. If a game can deal with slightly blurry graphics (if it does a lot of rescaling anyway for instance), you can even set content scale values that are non-integer.

Now there is a single coordinate system for everything. Cocos2D, SpriteBuilder and physics.

The patch is half Mac compatible, but I figured that should wait until merging.

[vlidholt]

The move to a content scale that can have other values that 1x or 2x was long overdue and is needed for sure. This solution is good for games that simply want to scale up/scale down their content (which is one common use case and should be supported). However, it is not at all good enough for all cases supported by SpriteBuilder and these needs to work too. That means that the solution I've proposed above needs to be implemented too for everything to work.

[dissidently]

How important are physics to the future of UI, GUI and games in the touch and tilt framed interactive worlds?

I think Birkemose got it right when he said this hasn't been sufficiently thought through.

Making decisions on how to wrap a physics engine around arbitrary systems and their coordinates is inherently backwards and will lead to ever more complex "solutions". And nightmares.

The physics engine of choice is superior (sometimes vastly so) to the real rival in its features and functionality for UI, GUI and most game types. It's a key point of difference.

Physics as a means of communicating interactivity response and responsiveness is now the dominant paradigm of iOS. And therefore audience expectations and the demands of future producers.

Embracing these realities means truly embracing the physics engine chosen. You've chosen well in terms of engine. But not yet chosen to embrace it.

Make Chipmunk the core system (yes, for everything it could conceivably be used for) and then everything else will, very literally, fall into place.

Sent from my iPad

On Nov 19, 2013, at 7:23 AM, Viktor Lidholt notifications@github.com wrote:

The move to a content scale that can have other values that 1x or 2x was long overdue and is needed for sure. This solution is good for games that simply want to scale up/scale down their content (which is one common use case and should be supported). However, it is not at all good enough for all cases supported by SpriteBuilder and these needs to work too. That means that the solution I've proposed above needs to be implemented too for everything to work.

\ Reply to this email directly or view it on GitHub.

[cocojoe]

You know I sort of agree with dissidently, I like the idea of physics being at the core of everything, very much like the universe ;-)

In general on the above going by my own use case in game design, which is universal and works across all iOS and also Android devices. I used a unified cocos2d/chipmunk design resolution with content being scaled up for 'HD'.

As it stands I see UI as seperate, in my own game I use a virtual grid of 32,32 divided into my design resolution for general purpose (centered) UI placement. For the most part this works nicely, however for some UI elements it's nice for them to expand out to fill the extra available space for example iPhone 5.

For the elements I want to utilise the extra space I can flag them so they will use the device resolution for the virtual grid instead of the device resolution.

Now that we have Slice9 it could also automagically expand UI elements out as well (when flagged).

Not sure if this helps tbh However it is a realistic and common use case.

[slembcke]

So to be clear, this isn't even just about physics. It goes beyond that. I've helped people to do this when their motivation was to share code and art assets between retina iPhone and non-retina iPad. A side effect is that you can use it as a consistent physics coordinate system too. As myself, @Birkemose, dissidently and @cocojoe have agreed, this is already a common thing that people want to do. The scaled position type is solving a similar problem by adding separate coordinate system, but that is used exclusively by the position property and not consistent with the rest of the system.

As I've perhaps explained poorly, the content scale fix is only an optional replacement for some uses of the scaled position type. Instead of using the scaled position type to multiply only the positions of objects, the entire "points" coordinate system used pretty consistently throughout Cocos2D is changed instead. This isn't mutually exclusive with the scaled position type.

The use case for the scaled vs. point position types were level and scene objects should use the scaled position types while UI could use the points position type. Now with the possibility of controlling the content size and the definition of what a "point" is, the level and scene objects can use the points type, and now the UI widgets can use the scaled position type if they want to be given more physically consistent sizes and positions.

The scale can even be changed to match the PPI of the specific device being used if it is the designer's with to create explicit, physically sized UI widgets. A year ago on iOS devices, Cocos2D points implied a scale of about 150 points per inch. With the iPad Mini and a myriad of supported Android devices, there is no longer any consistency in the PPI at all. Reducing the dependence on an assumed PPI of the "points" coordinate system also furthers one of the goals that the position modes were originally created for.

If updating SpriteBuilder to use the CCDirector.contentScale property instead of the CCDirector.positionScalefactor as the way to handle tablet scaling is an issue, I can assist with that.

[Birkemose]

Looks like we are a bit stuck on the positioning. What is the english word for an unsolvable situation? ... ahh yes ... marriage.

I am not trying to lecture anyone here, but it strikes me as odd, that all these bright minds can not settle on the best possible solution. It starts to sound to me, like a good ui, and good physics are not possible at the same time, which I very much doubt. I also think everybody on the team, should acknowledge the skills and dedication of the other team members. We are all here, because we are pretty damn good - and care - about what we do. And yeah. That includes me.

Would it be an idea, maybe to take a step back, and find out what the best possible solution would be, if we didn't have to worry about anything? What would work best, if we could chose freely? If we can agree on that, next step is to find out if this is feasible within the current timeframe, and if not, what corners can we cut, to meet the december release.

I would be very disappointed, if we settle for second best.

[slembcke]

Viktor sent me an update SpriteBuilder file with a number of UI items in it to prove the contentScale fix is still compatible with everything SpriteBuilder needs to do. Screenshot time.

iPhone preview in the SpriteBuilder editor:

iPad preview in the SpriteBuilder editor:

Since the content scale is changing how points is defined, I changed all of the nodes with a "points" position type to the "Scaled by position scale factor" position type. Content sizes were changed in basically the same way. Points is now the regular coordinate type again, and the scaled type is used for UI. Those were the only changes made to the SpriteBuilder files.

In the code, the only change is this from the applicationDidFinishLaunching: method.

if(UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPad){
    director_.contentScaleFactor *= 2.0;
    director_.positionScaleFactor *= 0.5;
}

In other words, treat the iPad as a device that uses 2 pixels per point so the coordinates match up similarly with the retina iPhone. The position scale factor is set to 0.5 on the iPad so that the UI will run in the ideal ~150 PPI range. As I mentioned before, it's now possible to adjust this value on a per-device basis if a designer wishes to have buttons appear the same physical size on an iPad Mini or Android tablet.

I did run into a bug that I'm certain can be fixed. CCButtons in particular are calculating their screen size incorrectly from their label. I spot checked a few of the other node types, including CCLabelTTF, and none of them had the same issues. I added transparent cyan scale-9 sprites over the buttons with the same position and content scale settings.

On the iPhone simulator: (Note that it's a retina screenshot and I didn't halve it's size before uploading)

On the iPad simulator:

Again, the cyan boxes all show up at the right size and place since they are given explicit content sizes instead of preferred sizes. The CCLabelTTF and CCSlider that I left in when debugging also correctly keep their same physical size. I'm still looking into why the buttons layout incorrectly, but that is the only issue I've run into.

edit: Also, since you can't see it without it animation. The physics works correctly and identically between the two versions.

[slembcke]

Fixed:

Turns out I just had to edit a couple extra CCButton settings that were hiding at the bottom of the settings scroller. So no further code changes were required.

edit: Source code available here

[dissidently]

Yeah, let's take a step back, get a deeper perspective, and put a true camera system in place. Half of all future problems revolving around placement and positioning are self resolving and instantly solved by doing this, and a world of new opportunities opens up.

By "true" camera system I mean one in which the camera moves, not the scene to a fixed view. And where multiple cameras are possible, so making mini-maps, split views, etc is easy.

On Tue, Nov 19, 2013 at 11:15 AM, Birkemose notifications@github.comwrote:

Looks like we are a bit stuck on the positioning. What is the english word for an unsolvable situation? ... ahh yes ... marriage.

I am not trying to lecture anyone here, but it strikes me as odd, that all these bright minds can not settle on the best possible solution. It starts to sound to me, like a good ui, and good physics are not possible at the same time, which I very much doubt. I also think everybody on the team, should acknowledge the skills and dedication of the other team members. We are all here, because we are pretty damn good - and care - about what we do. And yeah. That includes me.

Would it be an idea, maybe to take a step back, and find out what the best possible solution would be, if we didn't have to worry about anything? What would work best, if we could chose freely? If we can agree on that, next step is to find out if this is feasible within the current timeframe, and if not, what corners can we cut, to meet the december release.

I would be very disappointed, if we settle for second best.

— Reply to this email directly or view it on GitHubhttps://github.com/cocos2d/cocos2d-iphone/issues/371#issuecomment-28822627 .

[slembcke]

Okay! Progress on getting this all working in SpriteBuilder too.

There were a number of annoying Cocos/Mac issues that I had to discover one by one relating to the contentScale fix. The only thing remaining should be converting input coordinates. The iOS director uses my inverse projection patch (this was a couple years ago), but I guess that never got added to the Mac director. Should be easy to move that to the abstract director class.

As for SpriteBuilder. Now that the Cocos/Mac issues are getting cleared up, it seems to be very close to working without many additional changes. The actual content seems to be working correctly, but some of the SpriteBuilder UI drawn using Cocos need to be fixed (rulers, guides, etc).

[slembcke]

Changes to both Cocos2D and SpriteBuilder continue to be mild, but slow going with a lot of small issues that take a long time to debug. Ran into more iOS/Mac specific code for font handling that was screwing up some content scale properties. Ended up doing a full project search for CC_PLATFORM_IOS/CC_PLATFORM_MAC and found a few other things to fix. AFAIK, that should be the last of the Mac/contentScale fixes. The only remaining known issues are SpriteBuilder specific.

As of right now it's still not quite done, but is very close. The example from before works as expected in the editor itself:

Remaining issues:

• Label values on the rulers are currently disabled. This is actually a separate texture loading issue. Cocos2D loads textures assuming that they have the same contentScale as the director is set to regardless of the file's suffix. (i.e. falling back to loading a non-hd image gives you incorrect contentSizes of textures instead of just low-res ones) We either need to provide a high-res label atlas (will probably do this as a temporary fix), or fix this long standing bug (would really like to do this by release time).
• Creating a CCLayer .ccb file seems to flat out crash with several divide by zero issues. Not sure why yet. The other .ccb file types seem to work correctly.
• The layout containers seem to ignore the content scale. Not sure why yet. All the other UI components appear to work correctly.
• Some cleanup to do still. I changed a few hardcoded screen size values, but they really need to be recalculated based on user settings instead.
• We should consider renaming the positionScale property since it applies to scales and content sizes as much as it does to positions. Maybe CCDirector.uiScale makes more sense?

Code is here: https://github.com/slembcke/SpriteBuilder/tree/contentScale

[slembcke]

@everyone CCDirector.positionScaleFactor needs a new name. The value has been applied to more than just positions, and now it's more UI focused than it was before. The idea is that it's something you can multiply against that lets you convert points to a more physically consistent size (as in inches/cm) across different device types.

Possibilities:

• CCDirector.nativeScaleFactor
• CCDirector.UIScaleFactor
• CCDirector.physicalScaleFactor

I think UIScaleFactor is the most clear. I'll let Viktor make the final call though since it's originally his feature.

@vlidholt I figured out the SpriteBuilder layer bug by adding some asserts. It's creating a ResolutionSetting object with a scale of 0. This is broken in the non-patched SpriteBuilder code too, just not quite as dramatically. If you create a new layer ccb file and set anything to use the scaling factor, you end up with lots of NaNs and infinity. If CCDirector.contentScaleFactor gets set to 0, all sorts of weird and immediate crashes happen though.

I'm not sure how to fix this. It's not exactly clear to me what is different about the node ccb files. They also call [ResolutionSetting initWithSerialization:] with a scale of zero, but it sets the scale to a different value somewhere else.

[vlidholt]

The scale should never be 0, so that is obviously a bug. It should instead default to 1.

I think calling it UIScaleFactor may be the one that causes least confusion is easiest to explain.

[slembcke]

I think it's finally done. All of the know issues have been resolved. Including fixing the rulers a half dozen times. :-\

The changes to SpriteBuilder itself are pretty light: https://github.com/slembcke/SpriteBuilder/compare/contentScale

Most of the changes ended up in Cocos2D and I created a separate pull request for that here: https://github.com/cocos2d/cocos2d-iphone/pull/408

[vlidholt]

As far as I can tell this is now working nicely. Closing this issue.