Closed GoogleCodeExporter closed 9 years ago
597 “Comment out facet counts”, and this one, 599 “Refine facet values
shown, but no opp returned in SERP”, are essentially related to the same
underlying problem of the facet counts and what facet values we are showing in
them not being directly related to what is being pulled up in the SERP results
or in the tab level counts. I’d like to have Kelvin and his new guy, Tien,
help us get this issue figured out before you invest much more time in it. (I
think you said this issue has been troublesome to resolve so far.) Please send
me an example of the Solr queries that are run to derive the tab level opp
counts, the two Refine by facet opp counts, and the corresponding SERP results
on the geo page for the example I ran in
http://echoditto.latest.footprint2009dev.appspot.com/search#num=10&start=1&vol_l
oc=Greer%2C%20SC&distance=25&type=all&sort=Relevance&timeperiodstart=start%20dat
e&timeperiodend=end%20date&cache=1 and include an explanation of how they all
work briefly so I can pass that along to Kelvin/Tien and also make sure that MT
and I understand it fully as well. Perhaps MT or I might have an insight that
could be useful.
Original comment by danstryk...@gmail.com
on 21 Mar 2011 at 4:20
[deleted comment]
[deleted comment]
[deleted comment]
I think I have the facet count problem figured out so here is what I believe
you need to do to get the counts working properly:
1. Add the standard date fq to the end of the facet count queries is the fix
that does the trick. See examples of the current and the corrected facet
queries in the queries below.
&fq=((eventrangeend:[2011-03-24T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+20
11-03-24T23:59:59.999Z])+OR+(eventrangeend:+"1971-01-01T00:00:000Z"+AND+eventran
gestart:"1971-01-01T00:00:000Z"))
2. For performance reasons as per my other email, please put any current facet
filtering terms like the following into the fq instead of q for best
performance while you are making the above fix, Stuff like this should be in
the fq, not in the q:
+AND+virtual:true+AND+micro:false+AND+self_directed:false. Facet queries
should just have q=*:* unless I’m missing something
3. And just for maximum accuracy in our facet counts vs. the main query, change
the solrconfig.xml to use <str name="fields">aggregatefield</str>, not the
fields Kelvin originally set up in the config as the signature field values for
collapsing. Collapse should be occurring on the same field as we are using for
the main search. aggregatefield in the pipeline is these three fields:
description, org_name, title. Kelvin only set up the signature as <str
name="fields">title,abstract</str> based on an passing comment by Adam, but I
know it is not right. Make the change, restart Solr, and rebuild the index
from scratch and hopefully we are good to go. The collapsing is going to
return different counts if we are searching on the abstract for the main query
and different fields for the collapse,
http://code.google.com/p/allforgood/source/browse/branches/release_2_0.transitio
nal/SOLR/conf/solrconfig.xml#1078
SERP main results query
http://li169-139.members.linode.com:8983/solr/select/?&wt=xml&sort=Relevance%20d
esc&fq=%28%28eventrangeend:[2011-03-24T00:00:00.000Z+TO+*]+AND+eventrangestart:[
*+TO+2011-03-24T23:59:59.999Z]%29+OR+%28eventrangeend:+%221971-01-01T00:00:000Z%
22+AND+eventrangestart:%221971-01-01T00:00:000Z%22%29%29&rows=100&start=0&q={!sp
atial+lat%3D34.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28g
eo_distance%29%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:f
alse+AND+micro:false&fl=abstract,categories,org_name,detailurl,event_date_range,
feed_providername,ical_recurrence,id,latitude,location_string,longitude,openende
d,title,description&bq=categories:vetted^15+eventrangestart:[*+TO+NOW%2B6MONTHS]
^15+eventrangestart:[NOW+TO+NOW%2B1MONTHS]^10+eventrangestart:[NOW+TO+*]^5+event
rangestart:[NOW-6MONTHS+TO+*]^7+eventrangeend:[*+TO+NOW%2B6MONTHS]^7+eventrangee
nd:[NOW+TO+NOW%2B1MONTHS]^10+-feed_providername:meetup^2+eventduration:[1+TO+10]
^10
<result name="response" numFound="80" start="0">
Current inaccurate facet count query which shows inaccurate numbers <result
name="response" numFound="1544" start="0"/>
http://li169-139.members.linode.com:8983/solr/select/?wt=xml&q={!spatial+lat%3D3
4.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_distance%2
9%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND+micr
o:false&facet.mincount=2&facet.field=signature&facet.field=provider_proper_name_
str&facet=on&rows=0&facet.query=Military+Families&facet.query=Veterans&facet.que
ry=Education&facet.query=Hunger&facet.query=Animals&facet.query=Health&facet.que
ry=Seniors&facet.query=Technology&facet.query=Poverty
Same query as above, but with fq on dates <result name="response" numFound="80"
start="0"/>
http://li169-139.members.linode.com:8983/solr/select/?wt=xml&q={!spatial+lat%3D3
4.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_distance%2
9%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND+micr
o:false&facet.mincount=2&facet.field=signature&facet.field=provider_proper_name_
str&facet=on&rows=0&facet.query=Military+Families&facet.query=Veterans&facet.que
ry=Education&facet.query=Hunger&facet.query=Animals&facet.query=Health&facet.que
ry=Seniors&facet.query=Technology&facet.query=Poverty&fq=%28%28eventrangeend:[20
11-03-24T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+2011-03-24T23:59:59.999Z]
%29+OR+%28eventrangeend:+%221971-01-01T00:00:000Z%22+AND+eventrangestart:%221971
-01-01T00:00:000Z%22%29%29
Thanks,
Dan Stryker
All for Good Director of Product Management
Original comment by danstryk...@gmail.com
on 24 Mar 2011 at 4:21
[deleted comment]
This is a summary of the changes that need to be made which all should be quick
and easy for you to do:
1. Change the Refine by Provider facet query to use feed_providername, not
provider_proper_name, in the fq field. (Example below showing that this change
will work.) Maybe putting provider_proper_name:Senior+Corps in the q like you
do for the Category facet queries would work, but better to do it in the fq
using feed_providername:seniorcorps. I’ve asked Tien to clarify why our
current query does not work properly. It may be that fq with text strings does
not work properly. If that is the case, then leave the Refine by Category
drill down as part of q, and don’t move it to fq as was requested.
2. Add the standard date fq to the end of the facet count queries is the fix
that does the trick. See examples of the current and the corrected facet
queries in the queries below.
&fq=((eventrangeend:[2011-03-24T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+20
11-03-24T23:59:59.999Z])+OR+(eventrangeend:+"1971-01-01T00:00:000Z"+AND+eventran
gestart:"1971-01-01T00:00:000Z"))
3. And just for maximum accuracy in our facet counts vs. the main query, change
the solrconfig.xml to use <str name="fields">aggregatefield</str>, not the
fields Kelvin originally set up in the config as the signature field values for
collapsing. Collapse should be occurring on the same field as we are using for
the main search. aggregatefield in the pipeline is these three fields:
description, org_name, title. Kelvin only set up the signature as <str
name="fields">title,abstract</str> based on an passing comment by Adam, but I
know it is not right. Make the change, restart Solr, and rebuild the index
from scratch and hopefully we are good to go. The collapsing is going to
return different counts if we are searching on the abstract for the main query
and different fields for the collapse,
http://code.google.com/p/allforgood/source/browse/branches/release_2_0.transitio
nal/SOLR/conf/solrconfig.xml#1078
Thanks,
CURRENT REFINE BY provider_proper_name:Senior+Corps THAT RETURNS NO RESULTS IN
THE SEARCH
http://li67-22.members.linode.com:8983/solr/select/?&&sort=Relevance%20desc&fq=%
28%28eventrangeend:[2011-03-27T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+201
1-03-27T23:59:59.999Z]%29+OR+%28eventrangeend:+%221971-01-01T00:00:000Z%22+AND+e
ventrangestart:%221971-01-01T00:00:000Z%22%29%29&rows=100&start=0&q={!spatial+la
t%3D34.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_dista
nce%29%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND
+micro:false+AND+provider_proper_name:Senior+Corps&fl=abstract,categories,org_na
me,detailurl,event_date_range,feed_providername,ical_recurrence,id,latitude,loca
tion_string,longitude,openended,title,description&bq=categories:vetted^15+eventr
angestart:[*+TO+NOW%2B6MONTHS]^15+eventrangestart:[NOW+TO+NOW%2B1MONTHS]^10+even
trangestart:[NOW+TO+*]^5+eventrangestart:[NOW-6MONTHS+TO+*]^7+eventrangeend:[*+T
O+NOW%2B6MONTHS]^7+eventrangeend:[NOW+TO+NOW%2B1MONTHS]^10+-feed_providername:me
etup^2+eventduration:[1+TO+10]^10
USING feed_providername:seniorcorps IN THE QUERY WORKS
http://li67-22.members.linode.com:8983/solr/select/?&&sort=Relevance%20desc&fq=%
28%28eventrangeend:[2011-03-27T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+201
1-03-27T23:59:59.999Z]%29+OR+%28eventrangeend:+%221971-01-01T00:00:000Z%22+AND+e
ventrangestart:%221971-01-01T00:00:000Z%22%29%29&rows=100&start=0&q={!spatial+la
t%3D34.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_dista
nce%29%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND
+micro:false+AND+feed_providername:seniorcorps&fl=abstract,categories,org_name,d
etailurl,event_date_range,feed_providername,ical_recurrence,id,latitude,location
_string,longitude,openended,title,description&bq=categories:vetted^15+eventrange
start:[*+TO+NOW%2B6MONTHS]^15+eventrangestart:[NOW+TO+NOW%2B1MONTHS]^10+eventran
gestart:[NOW+TO+*]^5+eventrangestart:[NOW-6MONTHS+TO+*]^7+eventrangeend:[*+TO+NO
W%2B6MONTHS]^7+eventrangeend:[NOW+TO+NOW%2B1MONTHS]^10+-feed_providername:meetup
^2+eventduration:[1+TO+10]^10
Original comment by danstryk...@gmail.com
on 27 Mar 2011 at 9:25
Also, I believe you can remove the facet.field=signatuture from the facet count
queries unless you are using those for something. Please confirm.
Current
http://li67-22.members.linode.com:8983/solr/select/?&q={!spatial+lat%3D34.938727
9+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_distance%29%2C1%2C
1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND+micro:false&
facet.mincount=2&facet.field=signature&facet.field=provider_proper_name_str&face
t=on&rows=0&facet.query=Military+Families&facet.query=Veterans&facet.query=Educa
tion&facet.query=Hunger&facet.query=Animals&facet.query=Health&facet.query=Senio
rs&facet.query=Technology&facet.query=Poverty
Not including it gives the same results
http://li67-22.members.linode.com:8983/solr/select/?&q={!spatial+lat%3D34.938727
9+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_distance%29%2C1%2C
1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND+micro:false&
facet.mincount=2&facet.field=provider_proper_name_str&facet=on&rows=0&facet.quer
y=Military+Families&facet.query=Veterans&facet.query=Education&facet.query=Hunge
r&facet.query=Animals&facet.query=Health&facet.query=Seniors&facet.query=Technol
ogy&facet.query=Poverty
Original comment by danstryk...@gmail.com
on 27 Mar 2011 at 9:32
This is how to make the current query work.
Query provider_proper_name:Senior+Corps
will be parsed as provider_proper_name:Senior defaultsearchfield:Corps.
to search on one field you should use. provider_proper_name:(Senior+Corps)
Regards,
Tien
Original comment by danstryk...@gmail.com
on 27 Mar 2011 at 11:26
Need to re-enable showing the opp counts too.
Original comment by danstryk...@gmail.com
on 28 Mar 2011 at 5:57
I enabled the tab counts, removed the signature facet from the geo counts and
added date range to the fq parameter.
Original comment by jwdemp...@gmail.com
on 30 Mar 2011 at 7:47
I can still replicate the original issue below. Is this fix up?
http://echoditto.latest.footprint2009dev.appspot.com/search#num=10&start=1&vol_l
oc=Greer%2C%20SC&distance=25&type=all&sort=Relevance&timeperiodstart=start%20dat
e&timeperiodend=end%20date&cache=1. Note the Senior Corps (2) opps in the facet
count. Click Senior Corps. Note that no opps shown in SERP.
Original comment by danstryk...@gmail.com
on 30 Mar 2011 at 10:38
Missed adding the parentheses as shown in comment 9. The current query is on
drill down using this value
provider_proper_name_str:HandsOn Network
It should be like the below. no _str and parentheses around the term.
provider_proper_name:(Senior+Corps)
Original comment by danstryk...@gmail.com
on 31 Mar 2011 at 1:13
query has been changed to provider_proper_name:(Senior+Corps)
Original comment by jwdemp...@gmail.com
on 31 Mar 2011 at 5:42
Verified fixed.
Original comment by danstryk...@gmail.com
on 31 Mar 2011 at 6:15
Original issue reported on code.google.com by
danstryk...@gmail.com
on 18 Mar 2011 at 8:21