Blank page when loading cellxgene after cellxgene update

LisaSikkema commented 3 years ago

Describe the bug Hi there,

I've been using cellxgene for a while now (really great, thanks a lot!!), and decided to update cellxgene today to see if it would speed up cellxgene (using pip install --upgrade cellxgene). However, since my update I only get a blank screen when I go to the server page.

I'm using nginx to launch a cellxgene instance from a virtual machine that is accessible from a remote computer, in case that is of any relevance.

Before the update, everything was running perfectly. This is the command I use (with example dataset in this case): cellxgene launch -p 5007 -t test https://cellxgene-example-data.czi.technology/pbmc3k.h5ad

Here's the cellxgene output:

[cellxgene] Starting the CLI...
[cellxgene] Loading data from pbmc3k.h5ad.
[cellxgene] Warning: Moving element from .uns['neighbors']['distances'] to .obsp['distances'].

This is where adjacency matrices should go now. 
[cellxgene] Warning: Moving element from .uns['neighbors']['connectivities'] to .obsp['connectivities'].

This is where adjacency matrices should go now. 
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
[cellxgene] Launching! Please go to http://localhost:5007 in your browser.
[cellxgene] Type CTRL-C at any time to exit.

The issue might be related with an earlier issue of mine with cellxgene VIP (https://github.com/interactivereport/cellxgene_VIP/issues/21) and partly discussed in a subsequent email thread with you guys.

Here's what GoogleChrome developer tool shows when I try to load the page:

Any help would be appreciated!

Version:

hosted
Chrome and safari
cellxgene version: 0.16.6

MaximilianLombardo commented 3 years ago

Hey @LisaSikkema ,

Thank you for using cellxgene and for your question!

Before we can dive into your problem, would you mind giving me a bit more information?

Could you launch with the --verbose flag so that we can see some more detailed output from cellxgene?
Would you be able to forward (mlombardo@chanzuckerberg.com) or summarize the email thread that you mentioned and think is relevant to this issue?
Can you provide details on your Nginx configuration in case there are permissions/port issues that are causing conflict? In your previous issue with CellxgeneVIP, what was the resolution with NGINX Proxy (it seems that you and the VIP team solved in private communication)
Just for completeness, also let me know what version of python you are running on your server

Thanks!

Max

LisaSikkema commented 3 years ago

Hi Max,

Thanks for your reply. Output with verbose flag:

$ cellxgene launch --verbose -p 5005 -t test https://cellxgene-example-data.czi.technology/pbmc3k.h5ad
[cellxgene] Starting the CLI...
[cellxgene] Loading data from pbmc3k.h5ad.
[cellxgene] Warning: Moving element from .uns['neighbors']['distances'] to .obsp['distances'].

This is where adjacency matrices should go now. 
[cellxgene] Warning: Moving element from .uns['neighbors']['connectivities'] to .obsp['connectivities'].

This is where adjacency matrices should go now. 
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
WARNING:root:Type float64 will be converted to 32 bit float and may lose precision.
[cellxgene] Launching! Please go to http://localhost:5005 in your browser.
[cellxgene] Type CTRL-C at any time to exit.
 * Serving Flask app "server.app.app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
INFO:werkzeug: * Running on http://localhost:5005/ (Press CTRL+C to quit)
INFO:werkzeug:127.0.0.1 - - [03/Feb/2021 16:31:11] "GET / HTTP/1.0" 200 -

I forwarded the email thread.

Also emailed you the nginx config file (I'd rather not post it here, but maybe we can summarize the relevant parts here once we've figured out the problem).

I think the solution was some change in the VIP code, not sure what exactly they did, I pulled the updated code on 23 Sept. 2020 and then the problem was solved.

Python version: 3.7.9

Thanks!

Lisa

MaximilianLombardo commented 3 years ago

Hey Lisa,

Thanks for the extra info. I need just a bit more information from you:

are you using cellxgene VIP? if yes, can you reproduce the error with cellxgene?
can you reproduce the error locally with cellxgene?

LisaSikkema commented 3 years ago

Yes, I can reproduce the error in a clean environment with just python and cellxgene installed, no VIP
No, when I run it locally I have no problems, so I guess it's related to the nginx part then?

Thank you!

MaximilianLombardo commented 3 years ago

Thanks for the info Lisa!

Based on that information - I'd think the problem is with the Nginx (but I don't know for certain). The output from running cellxgene launch with the --verbose flag looked ok to me. I think one thing you can do is check the logs for Nginx and see if there are any errors thrown - that should give some more context on the problem, see example below:

sudo tail /var/log/nginx/error.log

Your path to the log file may different - I'm also assuming your VM is a linux machine.

Additionally - I saw that from conversation with the VIP team (shared via email) that your previous Nginx configuration did not allow for larger file sizes - is your current config allowing for the allocation of enough memory? (I imagine this sin't your problem - since you are using the pbmc test dataset, but just wanted to double check, since it seemed to be causing issues for you before)

LisaSikkema commented 3 years ago

Thanks for the pointers!

I just emailed you the log.

Yes I already adapted the nginx configuration so that larger file sizes are okay. But indeed that shouldn't really be the problem, since I ran a 400,000 cells by 25,000 genes object on cellxgene (older version) previously without issues, and the example dataset is much smaller.

MaximilianLombardo commented 3 years ago

Since it seems that this issue has to do with Nginx configuration rather than cellxgene (determined through private communication), this issue will be closed, but may be updated with the final solution (still in progress)

LisaSikkema commented 3 years ago

okay so this was indeed related to nginx, we managed to fix the problem. We had to add/change three lines to my nginx configuration file. Here's the relevant snippet before:

location /cellxgene {
    proxy_pass                            http://localhost:5005/;
        proxy_set_header Host                 $http_host;
        proxy_set_header X-Real-IP            $remote_addr;
        proxy_set_header X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto    $scheme;
        sub_filter 'href="static' 'href="cellxgene/static';
    sub_filter_types *;
    sub_filter_once on;
    # To add basic authentication to v2 use auth_basic setting.
        auth_basic "Registry realm";
        auth_basic_user_file /etc/nginx/conf.d/nginx.htpasswd;
        # for VIP:
    client_max_body_size 50M;
    }

and after:

location /cellxgene {
    proxy_set_header Accept-Encoding "";
        proxy_pass                            http://localhost:5005/;
        proxy_set_header Host                 $http_host;
        proxy_set_header X-Real-IP            $remote_addr;
        proxy_set_header X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto    $scheme;
        sub_filter 'href="static' 'href="cellxgene/static';
    sub_filter 'src="static' 'src="cellxgene/static';
    sub_filter_types *;
    sub_filter_once off;
    # To add basic authentication to v2 use auth_basic setting.
        auth_basic "Registry realm";
        auth_basic_user_file /etc/nginx/conf.d/nginx.htpasswd;
        # for VIP:
    client_max_body_size 50M;
    }

so we added two lines,
proxy_set_header Accept-Encoding ""; and sub_filter 'src="static' 'src="cellxgene/static'; and set sub_filter_once to off Those changes weren't necessary in the older cellxgene version I was using before.

chanzuckerberg / cellxgene

Blank page when loading cellxgene after cellxgene update #2032