search

thegooglecodearchive / allforgood

Automatically exported from code.google.com/p/allforgood
0 stars 0 forks source link

Refine facet values shown, but no opp returned in SERP and opp counts off #599

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Go to 
http://echoditto.latest.footprint2009dev.appspot.com/search#num=10&start=1&vol_l
oc=Greer%2C%20SC&distance=25&type=all&sort=Relevance&timeperiodstart=start%20dat
e&timeperiodend=end%20date&cache=1. Note that Senior Corps (2) opps in facet 
count.  Click Senior Corps.  Note that no opps shown in SERP.

Original issue reported on code.google.com by danstryk...@gmail.com on 18 Mar 2011 at 8:21

GoogleCodeExporter commented 9 years ago 
597 “Comment out facet counts”, and this one, 599 “Refine facet values 
shown, but no opp returned in SERP”, are essentially related to the same 
underlying problem of the facet counts and what facet values we are showing in 
them not being directly related to what is being pulled up in the SERP results 
or in the tab level counts.  I’d like to have Kelvin and his new guy, Tien, 
help us get this issue figured out before you invest much more time in it.  (I 
think you said this issue has been troublesome to resolve so far.)  Please send 
me an example of the Solr queries that are run to derive the tab level opp 
counts, the two Refine by facet opp counts, and the corresponding SERP results 
on the geo page for the example I ran in 
http://echoditto.latest.footprint2009dev.appspot.com/search#num=10&start=1&vol_l
oc=Greer%2C%20SC&distance=25&type=all&sort=Relevance&timeperiodstart=start%20dat
e&timeperiodend=end%20date&cache=1 and include an explanation of how they all 
work briefly so I can pass that along to Kelvin/Tien and also make sure that MT 
and I understand it fully as well.  Perhaps MT or I might have an insight that 
could be useful.

Original comment by danstryk...@gmail.com on 21 Mar 2011 at 4:20

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
I think I have the facet count problem figured out so here is what I believe 
you need to do to get the counts working properly:

1.  Add the standard date fq to the end of the facet count queries is the fix 
that does the trick.  See examples of the current and the corrected facet 
queries in the queries below. 
&fq=((eventrangeend:[2011-03-24T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+20
11-03-24T23:59:59.999Z])+OR+(eventrangeend:+"1971-01-01T00:00:000Z"+AND+eventran
gestart:"1971-01-01T00:00:000Z"))

2.  For performance reasons as per my other email, please put any current facet 
filtering terms like the following into the fq instead of q for best 
performance while you are making the above fix,  Stuff like this should be in 
the fq, not in the q: 
+AND+virtual:true+AND+micro:false+AND+self_directed:false.  Facet queries 
should just have q=*:* unless I’m missing something 

3.  And just for maximum accuracy in our facet counts vs. the main query, change 
the solrconfig.xml to use  <str name="fields">aggregatefield</str>, not the 
fields Kelvin originally set up in the config as the signature field values for 
collapsing.  Collapse should be occurring on the same field as we are using for 
the main search.  aggregatefield in the pipeline is these three fields: 
description, org_name, title.  Kelvin only set up the signature as <str 
name="fields">title,abstract</str> based on an passing comment by Adam, but I 
know it is not right.  Make the change, restart Solr, and rebuild the index 
from scratch and hopefully we are good to go.  The collapsing is going to 
return different counts if we are searching on the abstract for the main query 
and different fields for the collapse,  
http://code.google.com/p/allforgood/source/browse/branches/release_2_0.transitio
nal/SOLR/conf/solrconfig.xml#1078 

SERP main results query
http://li169-139.members.linode.com:8983/solr/select/?&wt=xml&sort=Relevance%20d
esc&fq=%28%28eventrangeend:[2011-03-24T00:00:00.000Z+TO+*]+AND+eventrangestart:[
*+TO+2011-03-24T23:59:59.999Z]%29+OR+%28eventrangeend:+%221971-01-01T00:00:000Z%
22+AND+eventrangestart:%221971-01-01T00:00:000Z%22%29%29&rows=100&start=0&q={!sp
atial+lat%3D34.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28g
eo_distance%29%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:f
alse+AND+micro:false&fl=abstract,categories,org_name,detailurl,event_date_range,
feed_providername,ical_recurrence,id,latitude,location_string,longitude,openende
d,title,description&bq=categories:vetted^15+eventrangestart:[*+TO+NOW%2B6MONTHS]
^15+eventrangestart:[NOW+TO+NOW%2B1MONTHS]^10+eventrangestart:[NOW+TO+*]^5+event
rangestart:[NOW-6MONTHS+TO+*]^7+eventrangeend:[*+TO+NOW%2B6MONTHS]^7+eventrangee
nd:[NOW+TO+NOW%2B1MONTHS]^10+-feed_providername:meetup^2+eventduration:[1+TO+10]
^10 
<result name="response" numFound="80" start="0">

Current inaccurate facet count query which shows inaccurate numbers <result 
name="response" numFound="1544" start="0"/>
http://li169-139.members.linode.com:8983/solr/select/?wt=xml&q={!spatial+lat%3D3
4.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_distance%2
9%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND+micr
o:false&facet.mincount=2&facet.field=signature&facet.field=provider_proper_name_
str&facet=on&rows=0&facet.query=Military+Families&facet.query=Veterans&facet.que
ry=Education&facet.query=Hunger&facet.query=Animals&facet.query=Health&facet.que
ry=Seniors&facet.query=Technology&facet.query=Poverty

Same query as above, but with fq on dates <result name="response" numFound="80" 
start="0"/>
http://li169-139.members.linode.com:8983/solr/select/?wt=xml&q={!spatial+lat%3D3
4.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_distance%2
9%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND+micr
o:false&facet.mincount=2&facet.field=signature&facet.field=provider_proper_name_
str&facet=on&rows=0&facet.query=Military+Families&facet.query=Veterans&facet.que
ry=Education&facet.query=Hunger&facet.query=Animals&facet.query=Health&facet.que
ry=Seniors&facet.query=Technology&facet.query=Poverty&fq=%28%28eventrangeend:[20
11-03-24T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+2011-03-24T23:59:59.999Z]
%29+OR+%28eventrangeend:+%221971-01-01T00:00:000Z%22+AND+eventrangestart:%221971
-01-01T00:00:000Z%22%29%29 

Thanks,
Dan Stryker
All for Good Director of Product Management

Original comment by danstryk...@gmail.com on 24 Mar 2011 at 4:21

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
This is a summary of the changes that need to be made which all should be quick 
and easy for you to do:
1.  Change the Refine by Provider facet query to use feed_providername, not 
provider_proper_name, in the fq field.  (Example below showing that this change 
will work.)  Maybe putting provider_proper_name:Senior+Corps in the q like you 
do for the Category facet queries would work, but better to do it in the fq 
using feed_providername:seniorcorps.  I’ve asked Tien to clarify why our 
current query does not work properly.  It may be that fq with text strings does 
not work properly.  If that is the case, then leave the Refine by Category 
drill down as part of q, and don’t move it to fq as was requested.
2.  Add the standard date fq to the end of the facet count queries is the fix 
that does the trick.  See examples of the current and the corrected facet 
queries in the queries below. 
&fq=((eventrangeend:[2011-03-24T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+20
11-03-24T23:59:59.999Z])+OR+(eventrangeend:+"1971-01-01T00:00:000Z"+AND+eventran
gestart:"1971-01-01T00:00:000Z"))
3.  And just for maximum accuracy in our facet counts vs. the main query, change 
the solrconfig.xml to use  <str name="fields">aggregatefield</str>, not the 
fields Kelvin originally set up in the config as the signature field values for 
collapsing.  Collapse should be occurring on the same field as we are using for 
the main search.  aggregatefield in the pipeline is these three fields: 
description, org_name, title.  Kelvin only set up the signature as <str 
name="fields">title,abstract</str> based on an passing comment by Adam, but I 
know it is not right.  Make the change, restart Solr, and rebuild the index 
from scratch and hopefully we are good to go.  The collapsing is going to 
return different counts if we are searching on the abstract for the main query 
and different fields for the collapse,  
http://code.google.com/p/allforgood/source/browse/branches/release_2_0.transitio
nal/SOLR/conf/solrconfig.xml#1078 

Thanks,

CURRENT REFINE BY provider_proper_name:Senior+Corps THAT RETURNS NO RESULTS IN 
THE SEARCH
http://li67-22.members.linode.com:8983/solr/select/?&&sort=Relevance%20desc&fq=%
28%28eventrangeend:[2011-03-27T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+201
1-03-27T23:59:59.999Z]%29+OR+%28eventrangeend:+%221971-01-01T00:00:000Z%22+AND+e
ventrangestart:%221971-01-01T00:00:000Z%22%29%29&rows=100&start=0&q={!spatial+la
t%3D34.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_dista
nce%29%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND
+micro:false+AND+provider_proper_name:Senior+Corps&fl=abstract,categories,org_na
me,detailurl,event_date_range,feed_providername,ical_recurrence,id,latitude,loca
tion_string,longitude,openended,title,description&bq=categories:vetted^15+eventr
angestart:[*+TO+NOW%2B6MONTHS]^15+eventrangestart:[NOW+TO+NOW%2B1MONTHS]^10+even
trangestart:[NOW+TO+*]^5+eventrangestart:[NOW-6MONTHS+TO+*]^7+eventrangeend:[*+T
O+NOW%2B6MONTHS]^7+eventrangeend:[NOW+TO+NOW%2B1MONTHS]^10+-feed_providername:me
etup^2+eventduration:[1+TO+10]^10 

USING feed_providername:seniorcorps IN THE QUERY WORKS
http://li67-22.members.linode.com:8983/solr/select/?&&sort=Relevance%20desc&fq=%
28%28eventrangeend:[2011-03-27T00:00:00.000Z+TO+*]+AND+eventrangestart:[*+TO+201
1-03-27T23:59:59.999Z]%29+OR+%28eventrangeend:+%221971-01-01T00:00:000Z%22+AND+e
ventrangestart:%221971-01-01T00:00:000Z%22%29%29&rows=100&start=0&q={!spatial+la
t%3D34.9387279+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_dista
nce%29%2C1%2C1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND
+micro:false+AND+feed_providername:seniorcorps&fl=abstract,categories,org_name,d
etailurl,event_date_range,feed_providername,ical_recurrence,id,latitude,location
_string,longitude,openended,title,description&bq=categories:vetted^15+eventrange
start:[*+TO+NOW%2B6MONTHS]^15+eventrangestart:[NOW+TO+NOW%2B1MONTHS]^10+eventran
gestart:[NOW+TO+*]^5+eventrangestart:[NOW-6MONTHS+TO+*]^7+eventrangeend:[*+TO+NO
W%2B6MONTHS]^7+eventrangeend:[NOW+TO+NOW%2B1MONTHS]^10+-feed_providername:meetup
^2+eventduration:[1+TO+10]^10

Original comment by danstryk...@gmail.com on 27 Mar 2011 at 9:25

GoogleCodeExporter commented 9 years ago 
Also, I believe you can remove the facet.field=signatuture from the facet count 
queries unless you are using those for something.  Please confirm.

Current
http://li67-22.members.linode.com:8983/solr/select/?&q={!spatial+lat%3D34.938727
9+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_distance%29%2C1%2C
1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND+micro:false&
facet.mincount=2&facet.field=signature&facet.field=provider_proper_name_str&face
t=on&rows=0&facet.query=Military+Families&facet.query=Veterans&facet.query=Educa
tion&facet.query=Hunger&facet.query=Animals&facet.query=Health&facet.query=Senio
rs&facet.query=Technology&facet.query=Poverty

Not including it gives the same results
http://li67-22.members.linode.com:8983/solr/select/?&q={!spatial+lat%3D34.938727
9+long%3D-82.2270568+radius%3D25.0+boost%3Drecip%28dist%28geo_distance%29%2C1%2C
1000%2C1000%29^1}*%3A*&fq=self_directed:false+AND+virtual:false+AND+micro:false&
facet.mincount=2&facet.field=provider_proper_name_str&facet=on&rows=0&facet.quer
y=Military+Families&facet.query=Veterans&facet.query=Education&facet.query=Hunge
r&facet.query=Animals&facet.query=Health&facet.query=Seniors&facet.query=Technol
ogy&facet.query=Poverty

Original comment by danstryk...@gmail.com on 27 Mar 2011 at 9:32

GoogleCodeExporter commented 9 years ago 
This is how to make the current query work.

Query provider_proper_name:Senior+Corps
will be parsed as provider_proper_name:Senior defaultsearchfield:Corps.
to search on one field you should use. provider_proper_name:(Senior+Corps)

Regards,
Tien

Original comment by danstryk...@gmail.com on 27 Mar 2011 at 11:26

GoogleCodeExporter commented 9 years ago 
Need to re-enable showing the opp counts too.

Original comment by danstryk...@gmail.com on 28 Mar 2011 at 5:57

GoogleCodeExporter commented 9 years ago 
I enabled the tab counts, removed the signature facet from the geo counts and 
added date range to the fq parameter.

Original comment by jwdemp...@gmail.com on 30 Mar 2011 at 7:47

GoogleCodeExporter commented 9 years ago 
I can still replicate the original issue below.  Is this fix up? 
http://echoditto.latest.footprint2009dev.appspot.com/search#num=10&start=1&vol_l
oc=Greer%2C%20SC&distance=25&type=all&sort=Relevance&timeperiodstart=start%20dat
e&timeperiodend=end%20date&cache=1. Note the Senior Corps (2) opps in the facet 
count.  Click Senior Corps.  Note that no opps shown in SERP.

Original comment by danstryk...@gmail.com on 30 Mar 2011 at 10:38

GoogleCodeExporter commented 9 years ago 
Missed adding the parentheses as shown in comment 9.  The current query is on 
drill down using this value
provider_proper_name_str:HandsOn Network

It should be like the below.  no _str and parentheses around the term.
provider_proper_name:(Senior+Corps)

Original comment by danstryk...@gmail.com on 31 Mar 2011 at 1:13

GoogleCodeExporter commented 9 years ago 
query has been changed to provider_proper_name:(Senior+Corps)

Original comment by jwdemp...@gmail.com on 31 Mar 2011 at 5:42

GoogleCodeExporter commented 9 years ago 
Verified fixed.

Original comment by danstryk...@gmail.com on 31 Mar 2011 at 6:15