'getpwuid(): uid not found: xxxx'

[pnovotnak]

I'm about 50% sure this is a Django / Django Haystack bug...

The error happens when running on:

• NGINX 1.4.4
• uWSGI 2.0
• PostgreSQL 9.3
• elasticsearch 0.90.9
• Django 1.4.4

When running several very expensive searches against elasticsearch I get this error:

and the whole worker is toast.

Here's my python requirements file:

Django==1.4.10
PIL==1.1.7
Whoosh==2.4.0
django-grappelli==2.4.8
django-haystack==2.0.0
django-mailchimp==0.1.30
django-registration==0.7
django-singletons==0.1.6
django-tagging==0.3.1
django-tinymce==1.5.2
psycopg2==2.4.5
pyelasticsearch==0.6.1
python-cjson==1.0.5
requests==2.1.0
simplejson==3.3.2
six==1.5.2
sorl-thumbnail==11.12
wsgiref==0.1.2

Here's my global defaults config:

[uwsgi]
autoload = true
master = true
cheaper-algo = spare
cheaper = 2
cheaper-initial = 5
workers = 30
cheaper-step = 1
idle = 300
no-orphans = true
pidfile = /run/uwsgi/%(deb-confnamespace)/%(deb-confname)/pid
socket = /run/uwsgi/%(deb-confnamespace)/%(deb-confname)/socket
stats = /tmp/%(deb-confname).uwsgi.stats
chmod-socket = 660
log-date = true
uid = www-data
gid = www-data
vacuum = true
max-requests = 5000
harakiri = 20

Here's my app's config:

[uwsgi]
max-requests = 500
uid = xxxxxxxxxx
workers = 5
chdir =  /var/www/xxxxxx
module = xxxxx.wsgi:application
home = /var/www/xxxxxxx
daemonize=/var/log/uwsgi/%n.log
touch-reload=../.touch-reload

[unbit]

Strange error, do you have the file/line (from the backtrace) raising it ?

[pnovotnak]

I suppose that might indeed help, eh :)

ERROR:django.request:Internal Server Error: /search/******
Traceback (most recent call last):
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
  File "/var/www/************/dev/****************/views.py", line 225, in **********
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/haystack/query.py", line 157, in _manual_iter
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/haystack/query.py", line 164, in _fill_cache
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/haystack/backends/__init__.py", line 485, in get_results
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 942, in run
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/haystack/backends/__init__.py", line 26, in wrapper
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 510, in search
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 96, in decorate
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 591, in search
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 568, in _search_or_count
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 239, in send_request
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/requests/sessions.py", line 394, in get
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/requests/sessions.py", line 348, in request
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/requests/sessions.py", line 274, in prepare_request
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/requests/utils.py", line 76, in get_netrc_auth
  File "/var/www/************/dev/env/local/lib/python2.7/site-packages/requests/utils.py", line 73, in <genexpr>
  File "/var/www/************/dev/env/lib/python2.7/posixpath.py", line 260, in expanduser
KeyError: 'getpwuid(): uid not found: 1001'

[unbit]

wow first time i see using .netrc after years :) those uid 1001 suggest you something ? (unless www-data maps to 1001 i do not think it is a uWSGI issue)

[pnovotnak]

Too young, never in my life seen a .netrc

Haha, meant to sensor that UID too. oooops :)

Yeah, that UID is a user created to serve this app- it's the correct UID, there is a www-data group (NGINX). The app is set to run as UID 1001, and GID www-data.

So you think it's a haystack / pyelasticsearch / requests issue?

[unbit]

for some reason it is not able to access /etc/passwd to resolve the mappings between 1001 and its name

[pnovotnak]

Defining the "$HOME" variable via the 'env' setting, actually does fix the problem: https://github.com/toastdriven/django-haystack/issues/924

But, then I run into another issue.. Django throws a "TemplateDoesNotExist" but when I thumb through the places it's looking- the file exists at one of them, and it's rw for user specified in the app's config...

TemplateDoesNotExist at /search/
search/results.html
Request Method: GET
Request URL:  http://**********/search
Django Version: 1.4.10
Exception Type: TemplateDoesNotExist
Exception Value:  
search/results.html
Python Executable:  /usr/local/bin/uwsgi
Python Version: 2.7.3
Python Path:  
['.',
 '',
 '/var/www/************/dev/env/lib/python2.7',
 '/var/www/************/dev/env/lib/python2.7/plat-linux2',
 '/var/www/************/dev/env/lib/python2.7/lib-tk',
 '/var/www/************/dev/env/lib/python2.7/lib-old',
 '/var/www/************/dev/env/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2',
 '/usr/lib/python2.7/lib-tk',
 '/var/www/************/dev/env/local/lib/python2.7/site-packages',
 '/var/www/************/dev/env/local/lib/python2.7/site-packages/PIL',
 '/var/www/************/dev/env/lib/python2.7/site-packages',
 '/var/www/************/dev/env/lib/python2.7/site-packages/PIL',
 '/var/www/************/dev/project/django_apps']
Server time:  Tue, 7 Jan 2014 10:13:13 -0600
Template-loader postmortem

Using loader django.template.loaders.app_directories.Loader:
/var/www/************/dev/env/local/lib/python2.7/site-packages/grappelli/templates/search/results.html (File does not exist)
...
/var/www/************/dev/project/django_apps/search/templates/search/results.html (File exists)

ls -l /var/www/****/dev/project/django_apps/search/templates/search/results.html # (File exists)

-rw-rw-r-- 1 \ \ /var/www/****/dev/project/django_apps/search/templates/search/results.html

(owned by user/ user's group)

Here's the traceback:

Traceback (most recent call last):
  File "/var/www/********/dev/env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
  File "/var/www/sargento-cpd/dev/project/django_apps/home/views.py", line 31, in home
  File "/var/www/sargento-cpd/dev/env/local/lib/python2.7/site-packages/django/shortcuts/__init__.py", line 20, in render_to_response
  File "/var/www/sargento-cpd/dev/env/local/lib/python2.7/site-packages/django/template/loader.py", line 169, in render_to_string
  File "/var/www/sargento-cpd/dev/env/local/lib/python2.7/site-packages/django/template/loader.py", line 145, in get_template
  File "/var/www/sargento-cpd/dev/env/local/lib/python2.7/site-packages/django/template/loader.py", line 138, in find_template
TemplateDoesNotExist: home.html

Here are my uWSGI params:

 uwsgi_param QUERY_STRING        $query_string;
 uwsgi_param REQUEST_METHOD      $request_method;
 uwsgi_param CONTENT_TYPE        $content_type;
 uwsgi_param CONTENT_LENGTH      $content_length;

 uwsgi_param REQUEST_URI     $request_uri;
 uwsgi_param PATH_INFO       $document_uri;
 uwsgi_param DOCUMENT_ROOT       $document_root;
 uwsgi_param SERVER_PROTOCOL     $server_protocol;
 uwsgi_param UWSGI_SCHEME        $scheme;

 uwsgi_param REMOTE_ADDR     $remote_addr;
 uwsgi_param REMOTE_PORT     $remote_port;
 uwsgi_param SERVER_PORT     $server_port;
 uwsgi_param SERVER_NAME     $server_name;

The site's nginx defines 'root' as /var/www/*****/

The initial request runs OK, other requests run just fine, even other searches, and other repeated searches. It's just when I run that specific query that returns 1000+ results on the same page (and is refreshed before the request completes) that things run amuck. I tried running Apache Bench against another search page, with the concurrency set to the max number of uWSGI workers and everything went just fine.

Again, once it happens, the worker is toast. Every subsequent page returns a template error.

Maybe it's a Django bug...

[pnovotnak]

If I limit the results with [:N] syntax it runs OK.

[pnovotnak]

Sorry for the bother, this is NOT a uwsgi issue. It happens when running with the Django dev server.