List Variables instead of datasets in search table

[aashish24]

@benbu Any progress on this end?

[benbu]

WIP 161-list-variables-on-search

[benbu]

Here's what I have, experimenting with floating items

[aashish24]

Looking good but need tags (top 3 may be?). Aashish will help if needed.

[benbu]

It appears that none of the variables in the database have tags entered

> db.files.find({},{'variables.tags': 1})
{ "_id" : ObjectId("523b0f58236f442367ae2479"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae247a"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae247b"), "variables" : [ ] }
{ "_id" : ObjectId("523b0f58236f442367ae247c"), "variables" : [ ] }
{ "_id" : ObjectId("523b0f58236f442367ae247d"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae247e"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae247f"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae2480"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae2481"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae2482"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae2483"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae2484"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae2485"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae2486"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f58236f442367ae2487"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f59236f442367ae2488"), "variables" : [ ] }
{ "_id" : ObjectId("523b0f59236f442367ae2489"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b0f59236f442367ae248a"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b26eab1f9a42ee88189c1"), "variables" : [ { "tags" : [ ] } ] }
{ "_id" : ObjectId("523b2a9bb1f9a42ee88189c3"), "variables" : [ { "tags" : [ ] } ] }

Do we need to fix this in the import script?

Also, the esgf query doesn't appear to return any tags, see example returned xml https://gist.github.com/benbu/6998806 The most user friendly text I see for a variable is the variable_long_name

[aashish24]

I see. I thought we got tags for ClimatePipes Phase I. Let me have a look. In the mean time, can use long variable names?

[benbu]

Doesn't look like the variable_long_name is being stored in database or being returned by the esgf query.

For esgf, here is the line that gets the variable name

variables = map(lambda x : {'name': x[0].get_content()}, context.xpathEval('./ns:variables/ns:variable/@name'))

and here's the (shortened) xml I think

<result name="response" numFound="11216" start="0" maxScore="0.23420182">
  <doc>
    <str name="id">GeoMIP.output1.ULAQ.ULAQ.G3.mon.atmos.Amon.r1i1p1.v20120904|cmip-dn1.badc.rl.ac.uk</str>
    ...
    </arr>
    <arr name="variable">
      <str>BrOx</str>
      <str>Bry</str>
      ...
    </arr>
    <arr name="variable_long_name">
      <str>Mole fraction of BrOx</str>
      <str>Mole fraction of Bry</str>
      ...
    </arr>

I'm not very familiar with xpath, how can I modify this to also return the long name along with the variable name?

I tried using this xpath expression

'./ns:variables/ns:variable_long_name/@name'

to no avail.

For mongoDB, I tried stepping through the import code to see where if the long name available, and wasn't able to find it. Does vtkNetCDFReader parse the variable_long_name meta data?

[cjh1]

On Tue, Oct 22, 2013 at 12:31 PM, benbu notifications@github.com wrote:

Doesn't look like the variable_long_name is being stored in database or being returned by the esgf query.

For esgf, here is the line that gets the variable name

variables = map(lambda x : {'name': x[0].get_content()}, context.xpathEval('./ns:variables/ns:variable/@name'))

and here's the (shortened) xml I think

GeoMIP.output1.ULAQ.ULAQ.G3.mon.atmos.Amon.r1i1p1.v20120904|cmip-dn1.badc.rl.ac.uk ... BrOx Bry ... Mole fraction of BrOx Mole fraction of Bry ... I'm not sure that xml is right? The sample I found on my disk looks like this:

<variables vocabulary="CF-1.0">^M
  <variable name="albisccp" vocabulary_name="cloud_albedo"

units="1">ISCCP Mean Cloud Albedo^M <variable name="clc" vocabulary_name="convective_cloud_area_fraction_in_atmosphere_layer" units="%">Convective Cloud Area Fraction^M <variable name="clcalipso" vocabulary_name="cloud_area_fraction_in_atmosphere_layer" units="%">CALIPSO Cloud Fraction^M <variable name="clhcalipso" vocabulary_name="cloud_area_fraction_in_atmosphere_layer" units="%">CALIPSO High Level Cloud Fraction^M <variable name="clic" vocabulary_name="mass_fraction_of_convective_cloud_ice_in_air" units="1">Mass Fraction of Convective Cloud Ice^M <variable name="clis" vocabulary_name="mass_fraction_of_stratiform_cloud_i

What do you want to extract? The content of the tag?

[benbu]

Yeah looks like the content is the long name.

[cjh1]

In which case your want something like this:

map(lambda x : {'name': x[0].get_content()}, context.xpathEval('./ns:variables/ns:variable/[@name]'))

Rather than selecting the attribute value, select the tag that has the attribute.

On Tue, Oct 22, 2013 at 1:46 PM, benbu notifications@github.com wrote:

Yeah looks like the content is the long name.

— Reply to this email directly or view it on GitHubhttps://github.com/OpenGeoscience/geoweb/issues/161#issuecomment-26825151 .

[benbu]

Thanks Chris, that worked.

[aashish24]

Has merge conflict:

++<<<<<<< HEAD +.table-heading-left {

• text-align: left; +} + +input[type="checkbox"][class="group-selection"] {
• margin-top: 0;
• margin-bottom: 2px;
• vertical-align: middle; +} ++=======
  • .variable-item {
  • width: 290px;
  • padding: 4px;
  • margin: 4px;
  • -webkit-border-radius: 4px;
  • -moz-border-radius: 4px;
  • border-radius: 4px;
  • background: #fff;
  • float: left;
  • }
  • .variable-name {
  • width: 180px;
  • white-space: nowrap;
  • overflow: hidden;
  • text-overflow: ellipsis;
  • padding-left: 8px;
  • padding-right: 8px; -} ++} ++>>>>>>> origin/161-list-variables-on-search

@benbu could you please fix?

[benbu]

Done. Ready to go into master.

[aashish24]

@benbu I might have missed the Window I guess. It has merge conflict with master and next now.

[benbu]

I merged master in again, fixed the conflicts, and merged it into next again.

[aashish24]

I am not getting any results when search for "temperature" do you see that too?

[benbu]

It's really slow, but I eventually get results

[cjh1]

I don't seem to get any.

Chris

On Tue, Nov 12, 2013 at 1:46 PM, benbu notifications@github.com wrote:

It's really slow, but I eventually get results

— Reply to this email directly or view it on GitHubhttps://github.com/OpenGeoscience/geoweb/issues/161#issuecomment-28321026 .

[aashish24]

What is weird is that when I am typing temperature to the point of tempe..I see the list but when I complete the word it disappears.

[aashish24]

I liked the interface. The next step would be to display bit more meta data on mouse hover. Also, we need to change the legend variable name to long name. All this could go into a new branch.

[benbu]

I created a new issue addressing the remaining concerns. This branch can be merged.

[aashish24]

@cjh1 Can you update the branch to current master and merge it after testing?

[cjh1]

The layout seems to be a little meshed up? I think the search results need go in another div? See attachment

[cjh1]

@benbu I can't seem to get any of the datasets to plot of the map with this branch?

[benbu]

I just fixed the layout. I'll try to see why plotting stopped working.

[cjh1]

Just to clarify when I mean plot I mean no colors on the map :-)

On Wed, Jan 22, 2014 at 3:10 PM, benbu notifications@github.com wrote:

I just fixed the layout. I'll try to see why plotting stopped working.

— Reply to this email directly or view it on GitHubhttps://github.com/OpenGeoscience/geoweb/issues/161#issuecomment-33062262 .

[cjh1]

Scratch that, looks like I didn't have Vistrails configured. The layout looks better. The only problem I see is gaps in the layout?

[benbu]

Yeah I saw that too, and was trying to figure out why it was happening but no luck. It's always that same spot. I'm going to try using inline-block instead of floats to see if that fixes it.

[cjh1]

Thanks,

I wonder if we could add an icon or text to the boxes to indicate if the dataset if from the database or esgf?

On Wed, Jan 22, 2014 at 3:20 PM, benbu notifications@github.com wrote:

Yeah I saw that too, and was trying to figure out why it was happening but no luck. It's always that same spot. I'm going to try using inline-block instead of floats to see if that fixes it.

— Reply to this email directly or view it on GitHubhttps://github.com/OpenGeoscience/geoweb/issues/161#issuecomment-33063269 .

[cjh1]

Why does it take so long for the database search results to come back? This used to be pretty instantaneous, is it the cost of the temporal and spacial restrictions?

[benbu]

I was able to fix the layout, turns out an empty string for the timerange was causing it. I also added a source badge, that displays either 'local' or 'ESGF'.

As for the speed, sometimes my database results come back instantly, other times it takes a while. When I investigated slowness in the past, it didn't seem like the bottleneck was on the javascript side, it displays things as soon as the ajax request returns.

I'm not sure what could be causing it, you might be right about the spatial and temporal restrictions. Is there some kind of indexing that can be done on these? I'd be surprised if this is the problem though considering the local database doesn't have a large number of items. But then again, I'm new to constructing mongo queries so maybe I created the most inefficient query possible.

[cjh1]

Layout looks good, as does the source badge, however, I don't seem to be able to drag them onto the map any more?

On Thu, Jan 23, 2014 at 3:05 PM, benbu notifications@github.com wrote:

I was able to fix the layout, turns out an empty string for the timerange was causing it. I also added a source badge, that displays either 'local' or 'ESGF'.

As for the speed, sometimes my database results come back instantly, other times it takes a while. When I investigated slowness in the past, it didn't seem like the bottleneck was on the javascript side, it displays things as soon as the ajax request returns.

I'm not sure what could be causing it, you might be right about the spatial and temporal restrictions. Is there some kind of indexing that can be done on these? I'd be surprised if this is the problem though considering the local database doesn't have a large number of items. But then again, I'm new to constructing mongo queries so maybe I created the most inefficient query possible.

— Reply to this email directly or view it on GitHubhttps://github.com/OpenGeoscience/geoweb/issues/161#issuecomment-33162823 .

[benbu]

Oops, I guess <li> items aren't draggable in the same way divs are, atleast not without other changes. I switched back to floating divs, since the issue was due to an empystring anyway. Should work now.

[cjh1]

Much better :-) Thanks

Regarding the performance. Its definitely a regression of some sort. If you try searching on what we have deployed at climatepipes.kitware.com. The local result come up pretty instantaneously and the ESGF results are not far behind.

On Thu, Jan 23, 2014 at 3:46 PM, benbu notifications@github.com wrote:

Oops, I guess

items aren't draggable in the same way divs are, atleast not without other changes. I switched back to floating divs, since the issue was due to an empystring anyway. Should work now.

— Reply to this email directly or view it on GitHubhttps://github.com/OpenGeoscience/geoweb/issues/161#issuecomment-33166627 .