ipypopout solara content not displaying properly

[havok2063]

I am seeing strange behavior when using Solara to embed Jdaviz into a front-end app. When the button to popout Jdaviz into a browser window is clicked, the application fails to display. Instead the following content message is displayed: A widget with mount-id="solara-main" should go here. As far as I can tell, I have the correct data-base-url and data-voila-host attributes set on the html body.

This is only seen in my production environment, which is solara app mounted into a fastapi app, deployed behind a dual-proxy nginx setup at '/valis/solara'. data-base-url is set to the nginx location path of the solara app, and data-voila-host is set to the main domain (the first proxy). I cannot reproduce this locally, when running in dev mode, or when duplicating the production setup with fake domains, so I'm not sure what the issue could be. It's seen using ipypopout == 1.4, jdaviz==3.10.3, and solara>=1.37, but also seen with earlier and later versions of solara and ipypopout.

If desired I can provide some example code, but I'm not sure it will reproduce the error. What triggers this message? Any thoughts on what could be causing this?

[maartenbreddels]

Hi Brian,

Do you see console messages in the dev console that might help? or maybe 404's in the network tab of the dev console? Do you run multiple workers? If so, the popup window might connect to a different Python process.

[havok2063]

Hi Maarten,

When I load the main page, with the solara embed, there are no network errors shown in the dev console. I see one console warning/error "Cookie “solara-session-id” has been rejected because it is in a cross-site context and its “SameSite” is “Lax” or “Strict”.", which may not be related.

When I click the Jdaviz popout button, in the dev console of the new popout browser window, I see the following error in the dev console Uncaught SyntaxError: Kernel message validation error: JSON.parse: unexpected character at line 1 column 1 of the JSON data from a serialize.js via connectKernel and solaraInit. When the browser loads, I see the solara "Loading App" screen, then the content A widget with mount-id="solara-main" should go here displays.

Yes the app is being run with gunicorn with 4 workers.

In the server logs, I also see the following:

CRITICAL:solara.server.app:Session id mismatch when reusing kernel (hack attempt?): session-id-cookie-unavailable:b82aee18-26f1-47f7-bbb2-f5d4e100c02b != 2764244a-25fb-4d9a-abd6-9832ff7b5e3f

as well as

[WARNING]: base url 'http://api.sdss.org' does not end with root path '/valis/solara' (UserWarning)
WARNING:py.warnings:/usr/local/lib/python3.10/site-packages/solara/server/starlette.py:404: UserWarning: base url 'http://api.sdss.org' does not end with root path '/valis/solara'

This could be a configuration mismatch behind a reverse proxy and can cause issues with redirect urls, and auth.

See also https://solara.dev/documentation/getting_started/deploying/self-hosted
It looks like the reverse proxy sets the x-script-name header to '/valis'
            It looks like the root path was configured to '/valis/solara' in the settings
            It looks like the root path set by the asgi framework was configured to '/valis/solara'

  warnings.warn(msg)

but I'm not setting the x-script-name in my nginx config, and my solara root path is set correctly.

[maartenbreddels]

Whow, you are hitting all the edges cases :)

with the solara embed

Do you mean it is embedded in an iframe? Solara probably does not know it is running under https, and does not set samesite=none, I've explain this in https://github.com/widgetti/solara/pull/806 . Let me know if this explain the situation. Probably X-Forwarded-Proto is set to http.

Yes the app is being run with gunicorn with 4 workers.

I think we should improve this in the docs. I've done this at https://github.com/widgetti/solara/pull/807 Does that make it more clear why this setup does not work?

I think the setup should be 4 different server processes running on different ports, and nginx setup with sticky sessions.

Maybe we can eventually put your configuration (or the gist of it) in our documentation.

The warning you get might be a false positive, although this one is strange:

It looks like the reverse proxy sets the x-script-name header to '/valis'

Especially if you did not set it. Maybe nginx does it by default?

[havok2063]

Whow, you are hitting all the edges cases :)

Haha, yes I do enjoy pushing the boundaries! ;)

Do you mean it is embedded in an iframe?

Yes. The code for that is https://github.com/sdss/zora/blob/main/src/components/Solara.vue

Solara probably does not know it is running under https, and does not set samesite=none, I've explain this in #806 . Let me know if this explain the situation. Probably X-Forwarded-Proto is set to http.

I'm pretty sure we're using secure https. But I can double check this. Both proxies have proxy_set_header X-Forwarded-Proto $scheme; set and all domains look correct to me.

I think we should improve this in the docs. I've done this at #807 Does that make it more clear why this setup does not work?

Ah yes I think that makes sense. And now I can recreate the issue locally if I adjust the workers. Although it is more intermittent than in our production deploy, sometimes it works and sometimes I get the widget message. Here is an example of our deploy setup for this particular project, https://github.com/havok2063/dual-proxy.

I think the setup should be 4 different server processes running on different ports, and nginx setup with sticky sessions.

I can try this but it may be a bit tricky for this particular project. I'll have to think about it. We don't have nginx plus so session persistence needs to be hash or ip_hash in the upstream. Not sure if sticky sessions are available in the open source nginx?

Especially if you did not set it. Maybe nginx does it by default?

Yeah it's weird. I see the warning even in my demo repo, so I wonder if there's an issue when both the FastAPI starlette and the Solara starlette set a root path?

[havok2063]

@maartenbreddels I set the number of my gunicorn workers to 1. Now when I try to popout jdaviz, I no longer get the A widget with mount-id="solara-main" should go here. but instead I see a perpetually spinning "Loading App" icon, with a continuous "Server disconnected" messages. I see a Uncaught SyntaxError: Kernel message validation error: JSON.parse: unexpected character at line 1 column 1 of the JSON data error message in the dev console and the web application logs show the traceback:

[2024-10-23 19:04:11 +0000] [4] [INFO] ('69.143.88.252', 0) - "WebSocket /jupyter/api/kernels/3bf310ea-2ee5-4be9-8708-8e657d98fe0f/channels?session_id=ab29883e-ca21-474f-8495-c398ef3713c9" [accepted]
[2024-10-23 19:04:11 +0000] [4] [INFO] connection open
[2024-10-23 19:04:12 +0000] [4] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 254, in run_asgi
    result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 149, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 75, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 443, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 149, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 75, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 443, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 149, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 341, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 82, in app
    await func(session)
  File "/usr/local/lib/python3.10/site-packages/solara/server/starlette.py", line 227, in kernel_connection
    await _kernel_connection(ws)
  File "/usr/local/lib/python3.10/site-packages/solara/server/starlette.py", line 303, in _kernel_connection
    await thread_return
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/solara/server/starlette.py", line 294, in websocket_thread_runner
    anyio.run(run_wrapper)  # type: ignore
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 68, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 204, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 199, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.10/site-packages/solara/server/starlette.py", line 285, in run_wrapper
    await run(ws_wrapper)
  File "/usr/local/lib/python3.10/site-packages/solara/server/starlette.py", line 280, in run
    await server.app_loop(ws_wrapper, ws.cookies, headers_dict, session_id, kernel_id, page_id, user)
  File "/usr/local/lib/python3.10/site-packages/solara/server/server.py", line 125, in app_loop
    context = initialize_virtual_kernel(session_id, kernel_id, ws)
  File "/usr/local/lib/python3.10/site-packages/solara/server/kernel_context.py", line 391, in initialize_virtual_kernel
    raise ValueError("Session id mismatch")
ValueError: Session id mismatch
[2024-10-23 19:04:12 +0000] [4] [INFO] connection closed

This is using jdaviz 3.10.3, starlette 0.27.0, solara 1.39.0 and ipypopout 1.4.0.

[maartenbreddels]

Hi Brian,

This is related to:

"Cookie “solara-session-id” has been rejected because it is in a cross-site context and its “SameSite” is “Lax” or “Strict”."

You should have in your server logs the following message:

Cookies with samesite=none require https, but according to the asgi framework, the scheme is {request.scope['scheme']!r}
and the x-forwarded-proto header is {request.headers.get('x-forwarded-proto', 'http')!r}. We will fallback to samesite=lax.

If you embed solara in an iframe, make sure you forward the x-forwarded-proto header correctly so that the session cookie can be set.

See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite for more information on samesite cookies.

Also check out the following Solara documentation:
 * https://solara.dev/documentation/getting_started/deploying/self-hosted
 * https://solara.dev/documentation/advanced/howto/embed

Can you confirm that?

The code that checks this can be found here: https://github.com/widgetti/solara/blob/98621928e210668f7c7f69161d62476b4cbe2775/solara/server/starlette.py#L451

Let me know if this helps you.

Regards,

Maarten