Rendering issue on iOS devices

[arie-matsliah]

Greetings, I develop the FlyWire Codex at Princeton Neuroscience Institute. My knowledge of neuroglancer is limited, so there's a good chance this is not an issuer rather misuse. Trying to visualize 3D volumes of brain regions. Here's an example link. It renders well on mac/win/android but not on iOS devices (iPhone/iPad). Any idea what could be wrong?

Thanks in advance, Arie

[jbms]

On iOS, Apple only permits the Safari web browser so there is no way around any limitations it may have. Other browsers like "Chrome" and "Firefox" available in the app store are actually just reskinned versions of Safari.

Safari 15 and later have support for WebGL 2, which is required by Neuroglancer, but it is possible there are still limitations. If you can provide a screenshot or any error messages, that may help in determining what the problem is.

[rmorey]

@jbms thanks for your response, I am on the FlyWire Codex team with Arie. Apologies in advance for the awkwardly sized screenshots

For example, take this neuroglancer view

Which on macOS Safari, and other desktop browsers, renders as so:

Note both layers render, the whole-brain mesh and the neuron.

Loading the same view on iOS (after waiting some time for the particularly large neuron to load), we see this:

No whole-brain mesh. This view sometimes simply crashes the entire page on iOS as well:

These were captured on a device running the latest iOS 17 Beta, but we see the same results on other iOS versions

Upon connecting the iOS device up to the debugger, and loading the page, I am seeing some errors:

We're continuing to debug and explore this issue, and will provide anything else we find out. Thank you for your help

[rmorey]

Here is our construction of the view URL, in case it is helpful

def url_for_cells(segment_ids, data_version):
    if data_version not in DATA_SNAPSHOT_VERSION_DESCRIPTIONS:
        log_error(
            f"Invalid version '{data_version}' passed to 'url_for_cells'. Falling back to default."
        )
        data_version = DEFAULT_DATA_SNAPSHOT_VERSION

    config = {
        "dimensions": {"x": [1.6e-8, "m"], "y": [1.6e-8, "m"], "z": [4e-8, "m"]},
        "projectionScale": 30000,
        "layers": [
            {
                "type": "image",
                "source": "precomputed://https://bossdb-open-data.s3.amazonaws.com/flywire/fafbv14",
                "tab": "source",
                "name": "EM",
            },
            {
                "source": "precomputed://gs://flywire_neuropil_meshes/whole_neuropil/brain_mesh_v3",
                "type": "segmentation",
                "objectAlpha": 0.05,
                "hideSegmentZero": False,
                "segments": ["1"],
                "segmentColors": {"1": "#b5b5b5"},
                "skeletonRendering": {"mode2d": "lines_and_points", "mode3d": "lines"},
                "name": "brain_mesh_v3",
            },
            {
                "type": "segmentation",
                "source": f"precomputed://gs://flywire_v141_m{data_version}",
                "tab": "segments",
                "segments": [
                    str(sid) for sid in segment_ids
                ],  # BEWARE: JSON can't handle big ints
                "name": f"flywire_v141_m{data_version}",
            },
        ],
        "showSlices": False,
        "perspectiveViewBackgroundColor": "#ffffff",
        "showDefaultAnnotations": False,
        "selectedLayer": {
            "visible": len(segment_ids) > 1,
            "layer": f"flywire_v141_m{data_version}",
        },
        "layout": "3d",
    }

    return f"{NGL_FLAT_BASE_URL}/#!{urllib.parse.quote(json.dumps(config))}"

[chrisj]

I haven't had much time to debug it but it is reproducible using the Xcode IOS simulator and attaching the mac safari dev tools to the simulator.

[copypasteearth]

Any fix for this? The same is happening to me. But the problem is a 308 http error and the difference in some GET request succeeding and failing is safari is requesting

/data/somefile For ones that succeed

And /data//somefile For ones throwing 308 http error.

Is there some configuration for the webserver I'm using to serve the files? I'm using flask airflow

[jbms]

As far as the initial issue, I suspect it may be that the mesh is too large and it's exceeding some memory limit on iOS. Reducing the size of the mesh (e.g. with simplification) may help.

Not sure about the extra slash issue, can you provide more details about that?

[copypasteearth]

When neuroglancer is requesting info files it is fine. But when it requests some of the data that is in the info files sometimes it requests

/data/file

and sometimes it requests

/data//file

The double slash throws a 308 error on safari which results in the cannot fetch chunk error above

When I run my web server locally it works but when I deploy to ec2 it fails

[rmorey]

Hey @jbms, thanks again for your help with this.

As far as the initial issue, I suspect it may be that the mesh is too large and it's exceeding some memory limit on iOS. Reducing the size of the mesh (e.g. with simplification) may help.

We don't believe it's a size/memory issue. As an example, take this mainline neuroglancer view showing a very small mesh. On all other platforms/browsers it renders correctly, but on iOS does not render at all.

[jbms]

If you set the opacity of the meshes to 1 (in rendering tab) does it then render on iOS?

[rmorey]

Ahh, yes it does apparently, thank you! I think we will be able to use this as a workaround for rendering neuropil meshes, but not for our whole-brain mesh

[copypasteearth]

@jbms @rmorey has this been resolved, i am trying to use opacity and silhouete in iOS safari and it just makes them dissapear?

[rmorey]

@jbms @rmorey has this been resolved, i am trying to use opacity and silhouete in iOS safari and it just makes them dissapear?

I believe it's not, there's some underlying issue with the way WebKit handles transparency in WebGL

[jbms]

There are a few different issues mentioned here. Can you clarify which ones you are referring to?

Let's keep this issue just for the meshes only appearing with opacity 1, and open a separate issue for the slash issue if that still exists.

One thing that isn't clear to me: the out of bounds memory access error --- does that still occur? How is that related to the meshes appearing or not and how does it relate to the opacity?

[rmorey]

@jbms

There are a few different issues mentioned here. Can you clarify which ones you are referring to?

from our perspective, the only issue is the one you mention, that transparent meshes don't render.

One thing that isn't clear to me: the out of bounds memory access error --- does that still occur? How is that related to the meshes appearing or not and how does it relate to the opacity?

I originally posted that error because it came up, I don't know whether or not it's related. I can do some testing to see if it reliably shows up alongside the rendering issue.

I have no idea what the slash issue is, we're not seeing that.

[copypasteearth]

Oh the slash issue can be discarded, I am just wondering about opacity now

[copypasteearth]

@jbms @rmorey any solutions to this problem? I am still trying to get opacity and sihloetting to work with neuroglancer, but the meshes are just invisible on ipads

[jbms]

@jbms @rmorey any solutions to this problem? I am still trying to get opacity and sihloetting to work with neuroglancer, but the meshes are just invisible on ipads

Can you provide the contents of the webgl2 tab on webglreport.com?

Unfortunately I don't have a convenient way to test out iOS safari.

The code for transparent rendering is here: https://github.com/google/neuroglancer/blob/ada384e9b27a64ceb704f565fa0989a1262fc903/src/neuroglancer/perspective_view/panel.ts#L612

The likely candidates for the issue are:

1. A problem with rendering to a R32F texture (https://github.com/google/neuroglancer/blob/ada384e9b27a64ceb704f565fa0989a1262fc903/src/neuroglancer/perspective_view/panel.ts#L180), which is used to accumulate the transparent rendering information.
2. A problem with copying it back to the main buffer, possibly something involving the stencil buffer.

You could try reading back the accumulation textures to see if it contains any non-zero values. You could also try changing the texture format from R32F / RED to RGBA32F / RGBA . The implementation might happen to support RGBA better.