Error when trying to find the memory information on the GPU

[awhitesong]

I am trying to run a program (flask API) through uwsgi on the nginx server. The program accesses GPU memory on first API call. I am able to run the program in normal mode when i do e.g :

uwsgi --ini /var/www/test/test_uwsgi.ini (Above command runs the program normally)

But when i try to run the same program in emperor mode through the command : uwsgi --emperor "/var/www/test/test_uwsgi.ini"

It throws this error.

 Error when trying to find the memory information on the GPU: initialization error Error allocating  139392 bytes of device memory (initialization error). 
 Driver report 0 bytes free and 0 bytes total [pid: 6032|app: 0|req: 1/1] xx.xx.xx.xx () {40 vars in 703 bytes} [Wed Apr  6 07:50:53 2016] 
 GET / => generated 291 bytes in 16269 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0) 
 announcing my loyalty to the Emperor...

How could this be resolved? How the emperor mode is conflicting with GPU memory ?

My system information : lsb_release -a

Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:    14.04 
Codename:   trusty

[xrmx]

If you can't share a reproducer or at least some debug hints that uwsgi is doing something (very) wrong i think this is quite difficult for us to help you. Said that looking at your command line example i think you are misusing the emperor, --emperor option value should be the dir that contains the .ini and not a path to a file no?

[awhitesong]

I got to know about this command from here I've tested this command and it works with the files not involving any GPU memory accesses (e.g simple hello world programs). But as soon as my program includes GPU memory access, the command fails with above shown error.

Here is the emperor.log which seems clear

Wed Apr  6 07:33:43 2016 - [emperor] vassal /var/www/test/test_uwsgi.ini has been spawned
Wed Apr  6 07:33:43 2016 - [emperor] vassal /var/www/test/test_uwsgi.ini is ready to accept requests
Wed Apr  6 07:34:04 2016 - [emperor] vassal /var/www/test/test_uwsgi.ini is now loyal

To give a sample example of the code:

This code works fine with emperor (the above shown command), it just returns the simple sys.path:

from flask import Flask, jsonify, request
import sys
import time

app = Flask(__name__)

@app.route('/')
def hello_world():
  prefix = sys.prefix
  path = str(sys.path)
  return jsonify({'prefix':prefix,'path':path })

if __name__ == '__main__':
  app.run(debug=True)

But when i run this ( I am importing a simple pretrained model in the GPU using keras, before the first API request @app.before_first_request ) with uwsgi and nginx server, i get the previously shown error in /var/log/uwsgi/test_uwsgi.log

from flask import Flask, jsonify ,request
import theano
from keras.models import model_from_json

#this code is executed before first request
@app.before_first_request
def model_load():
    #normal variable declarations..
    .....
    #here i'm loading weights into the GPU.. now the browser shows `internal server error 500` and `test_uwsgi.log` shows the above shown GPU memory error log. 
    model = model_from_json(open( parentdir + "/my_model_architecture.json").read())
    model.load_weights( parentdir + "/my_model_weights.h5")
    print "model loaded.."

 # this should execute after the above function but never executes with browser showing `internal server error`.
@app.route('/')
def hello_world():
    prefix = sys.prefix
    path = str(sys.path)
    return jsonify({'prefix':prefix,'path':path })

if __name__ == '__main__':
  app.run(debug=True)

However this all works fine if i use --ini instead of --emperor. But why is --emperor mode conflicting the loading of weights in GPU giving GPU initialization error?

[xrmx]

@awhitesong well the question should more be: what is theano doing that is broken by uwsgi emperor behaviour? :) Please paste your uwsgi config. If you don't already have them try separately enable-threads and / or lazy-apps and see if they make a difference.

[awhitesong]

I haven't created uwsgi.config. I am using supervisord for managing the uWSGI Emperor process. This is my supervisord.conf

[program:uwsgi-emperor]
command=/home/ubuntu/anaconda2/bin/uwsgi --emperor "/var/www/test/test_uwsgi.ini" --die-on-term --master --uid     www-data --gid www-data --logto /var/log/uwsgi/emperor.log
autostart=true
autorestart=true
redirect_stderr=true

The complete command to run emperor is this:

/home/ubuntu/anaconda2/bin/uwsgi --emperor "/var/www/test/test_uwsgi.ini" --die-on-term --master --uid www-data --gid www-data --logto /var/log/uwsgi/emperor.log

How do i include enable-threads / lazy-apps ?

[xrmx]

@awhitesong the uwsgi config i am referring to is _testuwsgi.ini

[awhitesong]

Here it is:

[uwsgi]
#application's base folder
base = /var/www/test

#python module to import
app = app
module = %(app)

home = /home/ubuntu/anaconda2
pythonpath = %(base)

#socket file's location
socket = /var/www/test/%n.sock

#permissions for the socket file
chmod-socket    = 666

#the variable that holds a flask application inside the module imported at line #6
callable = app

#location of log files
logto = /var/log/uwsgi/%n.log