search

cavb1205 / django-solr-search

Automatically exported from code.google.com/p/django-solr-search
0 stars 0 forks source link

support non ascii charactère like "é" #14

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. http://localhost:8000/search/?q=é

What is the expected output? 
Result if there is or no result 
What do you see instead?
"""
Environment:

Request Method: GET
Request URL: http://localhost:8000/search/
Django Version: 1.1 alpha 1 SVN-9939
Python Version: 2.5.2
Installed Applications:
[... long list of app ...]
Installed Middleware:
('... some middleware ...')

Traceback:
File
"/home/souris/workdir/lincolnloop/projects/populous/src/populous_env_0228/ve/src
/django/django/core/handlers/base.py"
in get_response
  86.                 response = callback(request, *callback_args,
**callback_kwargs)
File
"/home/souris/workdir/lincolnloop/projects/populous/src/populous_env_0228/ve/src
/solango-trunk/solango/views.py"
in select
  36.             paginator = SearchPaginator(params, request)
File
"/home/souris/workdir/lincolnloop/projects/populous/src/populous_env_0228/ve/src
/solango-trunk/solango/paginator.py"
in __init__
  44.         self.results = connection.select(params)
File
"/home/souris/workdir/lincolnloop/projects/populous/src/populous_env_0228/ve/src
/solango-trunk/solango/solr/connection.py"
in select
  177.         response = self.issue_request(self.select_url + "?" + query.url)
File
"/home/souris/workdir/lincolnloop/projects/populous/src/populous_env_0228/ve/src
/solango-trunk/solango/solr/query.py"
in url
  204.         query = urllib.urlencode(params)
File "/usr/lib/python2.5/urllib.py" in urlencode
  1250.             v = quote_plus(str(v))

Exception Type: UnicodeEncodeError at /search/
Exception Value: 'ascii' codec can't encode character u'\xe9' in position
0: ordinal not in range(128)

"""

The root cause is that urllib.urlencode does not support unicode well.

Please provide any additional information below.

Original issue reported on code.google.com by yann.ma...@gmail.com on 1 Mar 2009 at 7:13

GoogleCodeExporter commented 9 years ago 
This branch aims at improving the unicode support.
  * https://code.launchpad.net/~yml/solango/unicode
It is not yet there but it is close enough to be tested I have some issue now 
with
the "Facets".

Feedbacks are more than welcome
--yml

Original comment by yann.ma...@gmail.com on 2 Mar 2009 at 1:10

GoogleCodeExporter commented 9 years ago 
in fact in some circustances query.url does not return an url that solr 
understand so
in the case below :
"""
Out[0]: 
{'facet.field': u'author',
 'facet.field': u'model',
 'fl': [],
 'fq': [],
 'hl.fl': u'text',
 'q': ['beyonc\xc3\xa9', "model:[u'populous_news__story']"],
 'rows': u'5',
 'sort': [],
 'start': u'0'}
ipdb> n
>
/home/souris/workdir/lincolnloop/projects/populous/src/populous_env_0228/ve/src/
solango-trunk/solango/solr/connection.py(177)select()
    176         # Submits the response to solr
--> 177         response = self.issue_request(self.select_url + "?" + query.url)

"""
solr does not return a valid XML. As of now query.url is equal to :
"""
ipdb> query.url
Out[0]:
'rows=5&facet.field=model&facet.field=author&q=beyonc%C3%A9+AND+model%3A%5Bu%27p
opulous_news__story%27%5D&start=0&hl.fl=text&facet=true&hl=true'
"""
Any idea on what this url should look like ?

tanks,
--yml

Original comment by yann.ma...@gmail.com on 2 Mar 2009 at 1:55

GoogleCodeExporter commented 9 years ago 
Yes, Solango does need to be a better citizen. I will take a look at this branch
today. It however seems like a major overhaul to me.

The url doesn't look that bad. Your problem is within the q:

q=beyonc%C3%A9+AND+model%3A%5Bu%27populous_news__story%27%5D

The model part doesn't look right. But that's just a gut feeling.

Original comment by sean.cre...@gmail.com on 2 Mar 2009 at 3:39

GoogleCodeExporter commented 9 years ago 
sean you are right the model is not ok it should be something like this :
q=beyonc%C3%A9+AND+model:(populous_news__story+OR+populous_blog_post)

As of now I have no idea on how to fix it.
Your feedback suggestion fix is more than welcome. In addition of this I am not 
sure
that I have taken the best approach.

Thanks
--yml

Original comment by yann.ma...@gmail.com on 2 Mar 2009 at 3:55

GoogleCodeExporter commented 9 years ago 
revision 18 of the branch seems to be working in all the test I have done so 
far.

Original comment by yann.ma...@gmail.com on 2 Mar 2009 at 7:17

GoogleCodeExporter commented 9 years ago 
Hello Sean,
Did you had a chance to look at the proposal ? I would be glad to hear form you 
on this.
Thank you
Regards,
yann

Original comment by yann.ma...@gmail.com on 25 Mar 2009 at 11:33

GoogleCodeExporter commented 9 years ago 
Is there a way that you can just send me a diff? It's hard for me to go through 
this
code and pick out where you changed things.

Original comment by sean.cre...@gmail.com on 26 Mar 2009 at 12:32

GoogleCodeExporter commented 9 years ago 
Sean,
As of now the best I have is the the for diffs that can be found there :
http://bazaar.launchpad.net/~populous/solango/unicode/changes/18?start_revid=18

You are particularly interested by the revisions from 14 to 18. Let me know if 
this help.

Thanks for your help.
Regards,
Yann

Original comment by yann.ma...@gmail.com on 26 Mar 2009 at 12:56

GoogleCodeExporter commented 9 years ago 
Here is a patch for this particular error. Works-for-me(tm), please see if it 
can be
included.

Note it would be better to handle all values as unicode (generally speaking: any
string inside the program), but this is a subject for a more ambitious 
refactoring.

Original comment by bert.mat...@gmail.com on 22 Apr 2009 at 1:42

Attachments:

GoogleCodeExporter commented 9 years ago 
A fix was put in r36. Please check it out.

Thanks

Original comment by sean.cre...@gmail.com on 23 Jun 2009 at 4:31

GoogleCodeExporter commented 9 years ago 
The fix in r36 does not appear to fully fix the problem for me. Adding the 
second
half of the patch attached by lvscar to Issue 14
(http://code.google.com/p/django-solr-search/issues/detail?id=41) appears to 
fix it.

Without the second half of that patch, I still get the following exception:

Traceback:
File "/usr/lib/pymodules/python2.5/django/core/handlers/base.py" in get_response
  86.                 response = callback(request, *callback_args, **callback_kwargs)
File "/home/abeld/WORK/project/solango/views.py" in __call__
  36.         return self.main(request, form_class, template_name, extra_context={})
File "/home/abeld/WORK/project/solango/views.py" in main
  60.                 paginator = SearchPaginator(params, request)
File "/home/abeld/WORK/project/solango/paginator.py" in __init__
  48.         self._get_pagination_links(request)
File "/home/abeld/WORK/project/solango/paginator.py" in _get_pagination_links
  51.         base = utils.get_base_url(request,["page",])
File "/home/abeld/WORK/project/solango/utils.py" in get_base_url
  31.             ret += urllib.urlencode(get) + "&"
File "/usr/lib/python2.5/urllib.py" in urlencode
  1250.             v = quote_plus(str(v))

Exception Type: UnicodeEncodeError at /search/
Exception Value: 'ascii' codec can't encode character u'\xe1' in position 1: 
ordinal
not in range(128)

Original comment by Daniel.A...@gmail.com on 11 Jul 2009 at 12:19

GoogleCodeExporter commented 9 years ago 
Added the second of the patch in r42. Let me know how it looks.

Original comment by sean.cre...@gmail.com on 15 Jul 2009 at 3:00

GoogleCodeExporter commented 9 years ago 
Any objections to closing this now?

Original comment by craig.ds@gmail.com on 20 Sep 2009 at 11:43

GoogleCodeExporter commented 9 years ago

Original comment by craig.ds@gmail.com on 22 Sep 2009 at 9:17