Rust program that allows us to swiftly extract the proper nouns from a paragraph, while retaining the performance target.

[IoIxD]

Pull Request

Describe it here:

This adds a sub-directory named "wordsearch" which contains a Rust program that does what the title describes.

Some things to note:

• It works using the official Scrabble dictionary. When the program starts, it creates 26 vectors for each letter of the alphabet, each containing a group of characters, and loads the dictionary words into their respective letters. When it receives a paragraph, it checks the first letter of each of the words before checking it against the corresponding vector of words to see if it's in there, and if it's not, it's considered a proper noun. This is not a 100% correct solution but it has been given large paragraphs and works with about 95% accuracy. However some words may have to be chopped or added to that list since there are brands that have one word that could be in the dictionary ("Discord" and "Meta" come to mind. "Twitter" is also there.)

  • It removes any non-alphabetical character from a word before comparing, which is useful for successfully comparing a few words in the alphabet and also words that might have special symbols, such as trademarks and restricted symbols. It also means we save computational time not considering words that aren't alphabetical, and on that note:
  • It does not compare words with an apostrophe specifically. This is because Scrabble actually doesn't have that letter so it doesn't consider "you've" a word anyways, and "youve" isn't in there either. This seems like one solution to many problems, but actually the dictionary we're using is English only and non-alphabetical words will not make it through the program anyways, and when we consider those two languages, this is probably the only case where words with specific glyphs need to not be considered.
  • On that note, we should likely consider the Spanish dictionary. But at the moment this is an English only site so it's fine.

• It is a TCP server. This seems weird and wasteful but in my opinion it's not much weirder (in fact, in the golang code this will be less weird) then having it be a run-once program, and having it be a server allows us to retain data that will otherwise be reread and recalculated on each program startup.

• The reason this is in Rust and not Golang is because I wanted this to add minimal overhead to the program, and optimized Golang code will do this much slower then Rust I'm afraid.

There is not yet code to utilize this from the golang side, so none of the checkboxes apply.

[IoIxD]

I want to actually talk about why this was closed lmao

Google does not actually use the keywords meta tag. Some sources also say Bing and DuckDuckGo uses it to flag sites for scam; apparently it has historically been used for scam sites. So it's not worth it to do this.