Carbohydrate builder timeouts on large structures

[gitoliver]

Tool: glycam.org/txt with DGalpa1-4[4DManpa1-]<240>3DGlcpa1-OME

Bug Description:

On actual I get a "504 Gateway Time-out nginx" trying to get to https://glycam.org/cb/build_sequence/ from the glycam.org/txt page.

In my local dev env I get 502 Bad Gateway when accessing http://172.28.0.2/cb/build_default_sequence/. I think this where we should focus.

This happens after exactly 30 seconds each time.

Note I can build a default structure for this in gmml in 40 seconds, and the structure files do eventually show up in the build directory, but after the 502 error has been shown. You can watch the gems log and see it's not responding in time, but finishing eventually.

Building it twice results in the same error, the "evaluation" happens every time and it's taking too long. A sequence module re-factor is required to fix this, but in the meantime if we can wait longer that would be great.

I tried editing the following files in both: Proxy/nginx and Django/nginx:

oliver@frankus:~/Programs/GLYCAM_Dev_Env/V_2/Proxy/nginx$ tail conf.d/proxy.conf
proxy_set_header Host               $host;
proxy_set_header X-Real-IP          $remote_addr;
proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto  $scheme;
proxy_set_header X-Forwarded-Host   $host;
proxy_set_header X-Forwarded-Port   $server_port;
**proxy_read_timeout 900;
proxy_connect_timeout 900;
proxy_send_timeout 900;
send_timeout 900;**

oliver@frankus:~/Programs/GLYCAM_Dev_Env/V_2/Proxy/nginx$ tail -20 nginx.conf 
    # load configs
    # include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
        # Timeouts
    keepalive_timeout 1d;
    send_timeout 1d;
    client_body_timeout 1d;
    client_header_timeout 1d;
    proxy_connect_timeout 1d;
    proxy_read_timeout 1d;
    proxy_send_timeout 1d;
    fastcgi_connect_timeout 1d;
    fastcgi_read_timeout 1d;
    fastcgi_send_timeout 1d;
    memcached_connect_timeout 1d;
    memcached_read_timeout 1d;
    memcached_send_timeout 1d;
    lingering_timeout 1d;
    lingering_time 1d;
}

This did nothing; it still gives the same 502 error after 30s.

[danwentworthart]

I want to guess that Nginx has some sort of default timeout threshold that this request is exceeding. So one question is, what should that threshold be? Another question: Would this mean that even sequence evaluations need to be run on separate processes so that we guarantee a response before timeout, and then we poll for the evaluation response? Or is there a build holding us up, and maybe that could be what is threaded, and we could return the json within the existing threshold?

This would require input from Lachele/Gray'son, as if it were me, I would address this in a Sequence gemsModule refactoring pass, whenever that bubbles up in priority.