Create a pop up in GeoParser that displays the metadata record. FIXED - Unable to see the clusters using existing Solr Index

[antrromet]

Hey @smadha

So I had a couple of questions.

1. I have already run the GeoTopicParser on all my files that are indexed into my Solr. I have indexed the 'Geographic_LONGITUDE', 'Geographic_NAME', 'Geographic_LATITUDE' keys returned from the GeoTopicParser into my index, so do I still need to click on the Geoparse button?
2. I did follow the steps given in the wiki. But it failed with the following errors. Any idea what might be going wrong?

{"responseHeader":{"status":400,"QTime":42},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","org.apache.solr.common.SolrException"],"msg":"undefined field: \"core_names\"","code":400}}

Internal Server Error: /query_crawled_index/http://localhost:8983/solr/scientific_content_enrichment/polar/user/pass
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/Antrromet/Documents/USC/Spring2016/CDA_CSCI599/GeoParser/geoparser_app/views.py", line 261, in query_crawled_index
    core_name = get_index_core(domain_name, indexed_path)
  File "/Users/Antrromet/Documents/USC/Spring2016/CDA_CSCI599/GeoParser/geoparser_app/solr_admin.py", line 66, in get_index_core
    raise "Can't create core with core name {0}".format(core_name)
TypeError: exceptions must be old-style classes or derived from BaseException, not str
[25/Mar/2016 03:42:18] "GET /query_crawled_index/http://localhost:8983/solr/scientific_content_enrichment/polar/user/pass HTTP/1.1" 500 14718

[smadha]

Please see the documentation for installation on main page. Geoparser require a standalone solr server. By default it runs at 8983 but you can change it in config.txt L7 , L8

Your solr index would be a input to geoparser. It will then parse all the documents from your index and store results in its own solr index.

You need to start the solr server present in geoparser app. https://github.com/MBoustani/GeoParser/tree/master/Solr/solr-5.3.1

See this wiki page for architecture - https://github.com/MBoustani/GeoParser/wiki#high-level-architecture-of-geoparser

[antrromet]

@smadha Ah ok. Does that mean I run my local Solr server on say port 8984 and then run the Geoparser solr at say port 8983 (default as given in the config.txt). And then as an input to GeoParser I need to give the Solr 8984. Then it will store all the parsed data into Solr 8983. Is my understanding correct?

[smadha]

Also though you already ran GeoTopicParser this wont affect working of GeoParser and it will redo the same thing again.

Khooshe is a standalone application by itself. If you want you can integrate your results separately in Khooshe. You just need to follow structure of https://github.com/MBoustani/Khooshe/blob/master/sample_points/sample_points.csv

[smadha]

Yup - your second comment is correct.

[antrromet]

Awesome...Understood! Thanks for the prompt reply. I have already worked with Khooshe and yes I know its a standalone application. I saw that you recently integrated the same into GeoParser.

I will try to run Geoparser asap and return here with the status.

[smadha]

Thanks Antriksh :)

[chrismattmann]

good job guys

[antrromet]

Hey @smadha I was successfully able to Geoparse my solr index. Thanks for your help! :)

[smadha]

Thanks @antrromet . Did you face any new hiccups during the process?

Also I see UI broken (tabs not in same line). Had this been always the case for you?

[antrromet]

No there weren't any hiccups. Yes the tabs thing have always been like that.

[antrromet]

@chrismattmann Professor, in the assignment,

13a. Create a pop up in GeoParser that displays the metadata record.

Is this what you are looking for?

[antrromet]

@smadha I don't think the search functionality is working. I can see the clusters on the map as shown in above screenshots. But when I search for some location name, or for Nasa (you can see there is a doc in above screenshot) then it does nothing.

[smadha]

Thanks for the feedback. I fixed UI issue as well, you can pull.

Yup, search is not there yet. It's coming soon (Let me know if you are interested working on it, I will love to work along with you)

Regarding metadata - it looks very cool. Can you please raise a PR? This is what we thought about displaying meta data -

1. Add one more input field under "Show more options". This will take a CSV of field name for now. Later we will want to change it to a drop down of all available fields
2. When user hover over a bubble it will instantly see name of location and link to solr doc.
3. In parallel app will make a async call to original solr index fetching configured meta data.
4. Display meta data in similar fashion as your snapshot.

Does that makes sense?

@antrromet @chrismattmann

[smadha]

Also this is a prior issue regarding metadata

https://github.com/MBoustani/GeoParser/issues/52

[antrromet]

@smadha I understand the requirement for metadata. Lemme work on that and hopefully I should have some results for you soon!

[smadha]

Thanks :)

We can collaborate if need be.

[MBoustani]

@antrromet and @smadha I like the idea of letting user to select what Solr fields to show on pop-ups but i think that would happen after everything geoparsed. Before user click "View" to see the points, we can have a button to shows a modal page with all the Solr fields as checkbox, user can select/unselect fields to see on pop-ups. So my default we can have URL to Solr or some small number of fields to show.

[smadha]

@MBoustani It makes sense to me. Sometime back @chrismattmann suggested to use luke handler to get list of all solr fields. Also I found we can use this query- 'select?q=:&wt=csv&rows=0&facet' to get a list of solr fields.

I think we need to store a list of solr fields for every solr index we geotag in admin core. We can use stored values to default on modal page. I will create an API to get/set list of fields and update here.

@antrromet what do you think about it? I merged your PR but flow will be complete only after we build input part.

[chrismattmann]

We shouldn't ask the user each time what fields to show it should be in an easily changeable configure file

[antrromet]

@MBoustani @smadha Agreed. I think the best would be displaying a modal with checkboxes of all the available solr fields. If we follow the configuration file method, then the user would have to change the configuration file, and then load it in the website. The checkboxes would be easier for the user to change.

[smadha]

@chrismattmann - well the idea was to make field selection a mandatory step in the flow. So that user can always control what it sees. But I agree that a alert after pressing "View" can become irritating for some users.

What if we introduce a new button on action panel for updating popup content? This button will open a modal to select what Solr fields to show on pop-ups. Thoughts? @MBoustani This way though it would not be a mandatory step but it will be make the functionality more evident.

[MBoustani]

@smadha that is exactly what I had in mind, a new button to show modals with all solr fields and user can check/unckeck that to see on pop-ups and we save the config to not ask the user everytime.

[chrismattmann]

Guys this should be a config step not a runtime step

[chrismattmann]

In other words it should be hidden from the casual user and only be an admin step or infrequent step seeded with sensible defaults

[MBoustani]

@antrromet, @smadha and @chrismattmann another way of showing metadata is when user click on each point on map instead of pop-ups we show a model middle of screen that shows all the Solr fields using D3 http://bl.ocks.org/mbostock/1093025

[antrromet]

@MBoustani That would be cool, but here's my thought. Now when the user sees so many points plotted on the map and he just wants to browse through the points, suppose he wants to try and see what 1 point is. He clicks on it and then sees the above D3. Then he sees and realizes this point is not interesting, so he closes the model and repeats the process. The only drawback with this is that the user will have to click twice just to see any information for the point. And this clicking twice he will have to do for each point on the map, if he is just browsing casually.

A popup on hover is much faster because he can quickly see some relevant info and if wants to know further he can click on detail. I think instead of redirecting the user to the local solr link, maybe we could have the above D3 when he wants to see all the details.

[smadha]

One update from me - I added list of popup fields in return_points_khooshe API. I also connected it with popups. Fields can be set through below API call - http://localhost:8000/set_idx_fields_for_popup/<index_url>/<domain>/id,title

[antrromet]

@chrismattmann @MBoustani @smadha The consensus reached was that we would be displaying a popup on hover of points. That popup by default would contain some basic information like say the Title, id whatever (This would be the first 4-5 fields of the index by default). And then when the user clicks for more info on the popup we can have the D3 in a model that displays all the information with all the fields associated with that point.

We wont be using the API that @smadha created above, but we will keep it just in case for future.

I will start working on this. @smadha I will also add the lazy loading. I wasn't aware that there would be a remote Solr server.

[antrromet]

As the discussion above, I've modified the popup display. By default it shows first 4 fields of the index. These can be anything. And when the user clicks on more, a D3 is displayed with all the fields. Screenshots of the same are attached. Lemme know what you guys think!

[smadha]

I think it looks awesome. 💃

One heads up that content of a single field could be too huge to show in popups. So we should truncate it after may be 100-200 chars. What say?

[antrromet]

@smadha Yes, truncating it to around 140 chars.

[MBoustani]

@antrromet seems like the modal for D3 chart does not fully scroll down. Can you fix the modal to scroll completely to the end of chart?

[antrromet]

@MBoustani Can you please take an update and check? I did not face any issues while scrolling till the end. I tried in all 3 browsers: Chrome, Safari and Firefox. Attached are the screenshots for the same.

There was an other issue in Firefox in which the modal was cut off in the end if it was large. I fixed that issue for Firefox.

If the problem persists could you tell me which browser you are using and which OS?

[smadha]

@antrromet How many fields are there in your solr index? Can you try with a larger number of fields like 30 may be?