Coordinate systems between Metal and OpenGL

[lemoon]

Translation: It seems that there are differences in the coordinate systems between Metal and OpenGL. Should I handle the Y-axis coordinates in the vs shader?

[LukasBanana]

Short answer: Yes, you can handle it in the vertex shader.

Long answer: There is a difference in the coordinate systems, but this should be handled (almost entirely) by LLGL. Did you modify the Hello Triangle example? As long as you set the viewport to the current window size (or SwapChain resolution rather) the triangle should be stretched over the entire window: https://github.com/LukasBanana/LLGL/blob/80ffad48a8ae5f6c4d00449a11b6c4f2abcddfca/examples/Cpp/HelloTriangle/Example.cpp#L239 Judging by the screenshot, which shows both triangles in the same oritentation (and not one of them upside down), only the viewport is misaligned, not the vertex projection.

If you are transforming the triangle with a projection matrix (orthogonal or perspective), you have to account for the coordinate system the underlying rendering API supports, which you can query with RenderSystem::GetRenderingCaps().screenOrigin. The tutorials and examples use the same projection matrix for all backends except the Z-clipping is transformed for a unit cube (see ExampleBase::PerspectiveProjection), i.e. Z coordinates are projected into the range [-1, +1] (OpenGL, Vulkan) instead of [0, +1] (Direct3D, Metal).

There is also an option to let LLGL modify the GLSL vertex shader, with some more details in the documentation (see ShaderCompileFlags::PatchClippingOrigin). This is the only kind of shader processing LLGL does, though. Anything else must be handled by external tools such as DirectXShaderCompiler, glslang, SPIRV-Cross etc.

[lemoon]

I haven't made any modifications to the sample code, and it seems that all the current examples have issues with coordinate system differences between GL and Metal. Adding ShaderCompileFlags::PatchClippingOrigin seems not work.

[LukasBanana]

You should only use this flag for vertex shaders and only when you use the shader to render into a texture. The documentation explains this in more detail: https://github.com/LukasBanana/LLGL/blob/fed3c0c5c0871d1fb11bc2a31c2a1ebc2442205a/include/LLGL/ShaderFlags.h#L111

The screenshots show clearly that the viewport is not updated. Perhaps the windowing system does not work correctly with the newer macOS. I am still on macOS Mojave (10.14) with an older Intel MacBook, so I can't test anything with the M1 or M2 chips. Can you check if this condition is triggered when you resize the window? https://github.com/LukasBanana/LLGL/blob/fed3c0c5c0871d1fb11bc2a31c2a1ebc2442205a/sources/Platform/MacOS/MacOSWindow.mm#L481

[lemoon]

vars:

this    LLGL::MacOSWindow * 0x600000233040  0x0000600000233040
LLGL::Window    LLGL::Window    
wnd_    NSWindow *  0x100e06440 0x0000000100e06440
NSResponder NSResponder 
_firstResponder NSWindow *  0x100e06440 0x0000000100e06440
_level  int 0
_backgroundColor    id  0x0 0x0000000000000000
_postingDisabled    unsigned char   '\0'
_flushDisabled  unsigned char   '\0'
_miniIcon   id  0x0 0x0000000000000000
_frameSaveName  id  0x0 0x0000000000000000
_toolbar    id  0x0 0x0000000000000000
_defaultButtonCell  id  0x0 0x0000000000000000
_initialFirstResponderX id  0x0 0x0000000000000000
_windowLayoutsByScreenLayout    __NSDictionaryM *   1 key/value pair    0x000060000025a260
_resizeGenerationForScreenLayout    unsigned long long  0
_saveWindowLayoutSuspensionCount    unsigned long long  0
_autoPositionMask   unsigned int    0
_didSetAutoPositionMask bool    false
_windowNum  long long   10895
_trackingAreaManager    _NSTrackingAreaAKManager *  0x600002c12380  0x0000600002c12380
_lastWindowMovedEvent   id  0x0 0x0000000000000000
_contentView    MTKView *   0x102074a00 0x0000000102074a00
_delegate   MacOSWindowDelegate *   0x600000259ce0  0x0000600000259ce0
_styleMask  unsigned char   '\a'
_auxiliaryStorage   NSWindowAuxiliary * 0x100e32b30 0x0000000100e32b30
_lastLeftHit    id  0x0 0x0000000000000000
_lastRightHit   id  0x0 0x0000000000000000
_fieldEditor    id  0x0 0x0000000000000000
_borderView NSThemeFrame *  0x100e32f00 0x0000000100e32f00
_dragTypes  id  0x0 0x0000000000000000
_winEventMask   int -1071906784
_currentZoomState   unsigned long long  0
_previousZoomState  unsigned long long  0
_previousScreenNumber   long long   0
_screenAtStartMove  id  0x0 0x0000000000000000
_didStartMove   bool    false
_contentLayer   CALayer *   0x6000002370e0  0x00006000002370e0
_appCachedWindowContextID   long long   3986250854
_previewRepresentableActivityItems  id  0x0 0x0000000000000000
prevMotionOffset_   LLGL::Offset2D  
x   int32_t 0
y   int32_t 0

Another bug log report

select renderer:
 1.) Null
 2.) OpenGL
 3.) Metal
3
selected renderer: Metal
2023-08-17 11:13:21.418357+0800 Example_HelloTriangle[91416:1786691] Metal API Validation Enabled
2023-08-17 11:13:21.505000+0800 Example_HelloTriangle[91416:1786691] [Window] Warning: Window NSWindow 0x100637fa0 ordered front from a non-active application and may order beneath the active application's windows.
Renderer:             Metal 2.1
Device:               Apple M2 Max
Vendor:               Apple
Shading Language:     Metal Shading Language
Swap Chain Format:    BGRA8UNorm
Depth/Stencil Format: <undefined>
2023-08-17 11:21:53.007295+0800 Example_HelloTriangle[91416:1786691] IOSurface creation failed: e00002be parentID: 00000000 properties: {
    IOSurfaceAllocSize = 7684096;
    IOSurfaceBytesPerElement = 4;
    IOSurfaceBytesPerRow = 6400;
    IOSurfaceCacheMode = 1792;
    IOSurfaceHeight = 1200;
    IOSurfaceMapCacheAttribute = 0;
    IOSurfaceName = QuartzCore;
    IOSurfacePixelFormat = 1111970369;
    IOSurfacePixelSizeCastingAllowed = 0;
    IOSurfaceWidth = 1600;
    IsDisplayable = 1;
} (likely per client IOSurface limit of 16384 reached)
2023-08-17 11:21:53.007547+0800 Example_HelloTriangle[91416:1786691] [Utilities] CAIOSurfaceCreate: failed to allocate IOSurface (size: 1600 x 1200, format: BGRA)
2023-08-17 11:21:53.007623+0800 Example_HelloTriangle[91416:1786691] [CAMetalLayer nextDrawable] returning nil because allocation failed.
2023-08-17 11:21:53.008172+0800 Example_HelloTriangle[91416:1786691] -[_MTLCommandBuffer presentDrawable:], line 771: error 'drawable must not be nil.'
-[_MTLCommandBuffer presentDrawable:]:771: failed assertion `drawable must not be nil.'
(lldb) 

[LukasBanana]

Can you please outline a little bit what is happening here in your screenshots? Did you modify anything? Have you resized the window and this assertion was triggered or what happened?

[LukasBanana]

I actually experienced this bug myself now that I got to enable high-resolution display support in the CMakeLists.txt script. This should be fixed with 7a9414f.