KWIC (keyword in context) view in search results

[miglamigla]

I implemented some version of printing results of search query with a keyword in context. At the moment, it uses the string slicing technique. I did write code that would split the sentences where searched words occur into words and then slice on the words, but changed it (for simplicity's sake) to slicing on characters (without splitting the results). This isn't yet perfect:

• for some queries it does not print anything (although it does seem to find the matches)
• it also prints duplicate lyric slices, but this might be due in part to duplicate song lyrics in the text file, and in part because of the code.
• it works only for single word/part of word searches
• and of course it searches only through lyrics and doesn't find artist names

[miglamigla]

I will try to work out what is wrong, but it would be useful if you could look over that piece of code (if you have time) - perhaps you can see already what's wrong? Or else, which parts raise questions? Any insight would be useful.

[Alina-enni]

I cannot get the KWIC feature to work when I run the search engine. Search results are still just presented as full documents for some reason. The code is identical to the code in the repository, so I have no idea why this is

[Alina-enni]

It also seems that instead of showing the keywords in context, the search is just producing duplicates for me. For example, when searching "bones" which should only bring up "Fix you" by Coldplay (= 1 match), the results are 10 documents that are all the same:

[miglamigla]

Updated the code following advice from Yves. It searches for each word in the searchlist through each of the documents. If it finds something, it appends a string (artist, song name, part of lyrics) to matches, which is a list of strings for now (I tried to make tuple, but had no idea how to get it to print out the results; so this option is temporary). Some other comments:

• the .find() function finds any occurrence within the string, so if we search for "day" it finds "day, days, Wednesday, Wednesdays", but only "day, days" appears in the searchlist. Because of this I changed the regex, which is the basis for the searchlist. In general, our search now finds all occurrences of the query anywhere in the word.
• for some reason I don't understand, the code bites off letters of certain bandnames. I've noticed only "Beatle", "Blu", "Blue Octobe", "nnie Lennox". Probably something in the code?
• there still was some repetition in the results. I think, the line "if tuple_doc_and_index not in count" is sometimes ignored, but I don't understand why...
• this code is still very specific to the lyrics2.txt file, so any changes to that might require changes to the code as well
• and of course, all changes were done only for the single word search :D I'll get to the multiple word search later

[miglamigla]

Found the mistake why I couldn't print the tuple before. Now the results look better on the HTML page.

[Alina-enni]

This sounds great! I will have a look at what the code produces on my laptop later on. I guess all of the issues aren't solved yet but hopefully this is progress

[Alina-enni]

Yeah I just ran the current code and it definitely looks better now, and the KWIC seems to work for me now too! I searched 'cold' and I'm not sure why it finds 'cold' and 'coldplay' but only displays 'Michael Jackson's Thriller' though

[miglamigla]

oh no...... but I think I know where the problem is. the matching docs show occurrences not only in lyrics, but in artist and song name; essentially, in all the keys in tfv5.vocabulary_ . And the "found words" is based on that too. But we get the results searching for the queries only in the lyrics (excluding the artist and song name). I hope that makes sense. It feels like the whole piece of code for the results should then be rewritten, but I'm out of ideas :(((

[Alina-enni]

I see what you mean! So even though 'coldplay' matches a number of documents, the printed out results will only include hits in the actual lyrics. I'll have a look at the code to see if I can do anything about this

[Alina-enni]

Looking at the code and this issue, I'm wondering if we can add some kind of conditional that checks if the query matches an artist name

[Alina-enni]

I think we may need to make a decision on whether we allow the user to search artist or song names at all because this may be too hard to execute

[miglamigla]

Hmm, I would vote for searching just the lyrics - seems to be more variability there. But then another question - should we put in some query restrictions (e.g. printing sorry, no results, if the query is only an artist/song name and found nowhere in the lyrics)?

[miglamigla]

kwic works, so closing the issue