Unicode Input crashes the system

[GoogleCodeExporter]

I edited the course "Structure and Interpretation of Computer Programs" by 
pasting in the following paragraph between <PASTED> and </PASTED>

<TEXT>
we are interested in teaching you about programming, not about any par-
ticular programming language. We consider a series of techniques for 
controlling program
complexity, such as functional programming, data abstraction, object-oriented 
program-
ming, and query systems. To get past generalities you must have programming 
practice
in some particular language, and in this course we use Scheme, a dialect of 
Lisp. This
language is particularly well-suited to the organizing ideas we want to teach. 
Our hope,
however, is that once you have learned the essence of programming, you will 
find that
picking up a new programming language is but a few days’ work.
</PASTED>

As a result, I got this stack trace, perhaps related to the non-standard ' in 
"a few days' work".

Loading __main__, app version = 2d.344695753666345470
django.__file__ = 
'/base/python_runtime/python_lib/versions/third_party/django-1.1/django/__init__
.py', django.VERSION = (1, 1, 0, 'final', 0)
Fixing auth domain ('google.com')
Exception in request: UnicodeEncodeError: 'ascii' codec can't encode character 
u'\u2019' in position 887: ordinal not in range(128)
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/handlers/base.py", line 92, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/k16-8888/2d.344695753666345470/demo/views.py", line 738, in submit_edits
    doc = create_doc(data_dict)
  File "/base/data/home/apps/k16-8888/2d.344695753666345470/demo/views.py", line 377, in create_doc
    text = str(element.get('val'))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 
887: ordinal not in range(128)
<type 'exceptions.AttributeError'>: 'module' object has no attribute 
'handler500'
Traceback (most recent call last):
  File "/base/data/home/apps/k16-8888/2d.344695753666345470/main.py", line 140, in <module>
    main()
  File "/base/data/home/apps/k16-8888/2d.344695753666345470/main.py", line 110, in real_main
    util.run_wsgi_app(application)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 97, in run_wsgi_app
    run_bare_wsgi_app(add_wsgi_middleware(application))
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 115, in run_bare_wsgi_app
    result = application(env, _start_response)
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/handlers/wsgi.py", line 241, in __call__
    response = self.get_response(request)
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/handlers/base.py", line 134, in get_response
    return self.handle_uncaught_exception(request, resolver, exc_info)
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/handlers/base.py", line 165, in handle_uncaught_exception
    callback, param_dict = resolver.resolve500()
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/urlresolvers.py", line 266, in resolve500
    return self._resolve_special('500')
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/urlresolvers.py", line 255, in _resolve_special
    callback = getattr(self.urlconf_module, 'handler%s' % view_type)
<type 'exceptions.AttributeError'>: 'module' object has no attribute 
'handler500'

Now, the system is broken, giving the following trace:

Fixing auth domain ('google.com')
Exception in request: SystemError: Parent module 'django.template' not loaded
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/handlers/base.py", line 92, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/k16-8888/2d.344695753666345470/demo/views.py", line 224, in login_wrapper
    return func(request, *args, **kwds)
  File "/base/data/home/apps/k16-8888/2d.344695753666345470/demo/views.py", line 446, in index
    'in_progress_courses': in_progress_courses})
  File "/base/data/home/apps/k16-8888/2d.344695753666345470/demo/views.py", line 157, in respond
    context_instance=RequestContext(request))
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/template/context.py", line 106, in __init__
    for processor in get_standard_processors() + processors:
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/template/context.py", line 74, in get_standard_processors
    from django.conf import settings
SystemError: Parent module 'django.template' not loaded
<type 'exceptions.AttributeError'>: 'module' object has no attribute 
'handler500'
Traceback (most recent call last):
  File "/base/data/home/apps/k16-8888/2d.344695753666345470/main.py", line 110, in real_main
    util.run_wsgi_app(application)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 97, in run_wsgi_app
    run_bare_wsgi_app(add_wsgi_middleware(application))
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 115, in run_bare_wsgi_app
    result = application(env, _start_response)
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/handlers/wsgi.py", line 241, in __call__
    response = self.get_response(request)
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/handlers/base.py", line 134, in get_response
    return self.handle_uncaught_exception(request, resolver, exc_info)
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/handlers/base.py", line 165, in handle_uncaught_exception
    callback, param_dict = resolver.resolve500()
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/urlresolvers.py", line 266, in resolve500
    return self._resolve_special('500')
  File "/base/python_runtime/python_lib/versions/third_party/django-1.1/django/core/urlresolvers.py", line 255, in _resolve_special
    callback = getattr(self.urlconf_module, 'handler%s' % view_type)
<type 'exceptions.AttributeError'>: 'module' object has no attribute 
'handler500'

Running on yqhe12:4151 statusz INFO

Original issue reported on code.google.com by den...@google.com on 14 Sep 2010 at 5:55

[GoogleCodeExporter]

Apparently this problem with pasting in unicode is a known bug.

Original comment by den...@google.com on 14 Sep 2010 at 8:46

• Changed title: Unicode Input crashes the system
• Changed state: Accepted

[GoogleCodeExporter]

Here is a start.  This two-liner patch seems to fix the issue for ri-text 
thing.  It also fixes the same bug that manifests when you make a link to a 
document whose title has non-ascii in it.

  http://linus.mtv.corp.google.com:8080/

is running with the patch

Original comment by j...@google.com on 14 Oct 2010 at 7:33

[GoogleCodeExporter]

Filed a code review request for this as issue 72

Original comment by j...@google.com on 14 Oct 2010 at 8:16

[GoogleCodeExporter]

Fixed by Issue 72. r284

Original comment by vc...@google.com on 20 Oct 2010 at 7:27

• Changed state: Fixed