jeheydorn
commented
5 years ago Those txt files are the output of a command line based Java program I wrote which uses Stanford's CoreNLP libraries. The parser finds noun-verb pairs, noun-adjective pairs, and names of people and places. The source code for the book parser is not included with the generator for a few reasons:
- Stanford's CoreNLP library has an open source license which requires anything which has a binary link to it to have the same license. So if the book parser were part of the generator, then the generator and anything software that used it and is distributed would need to have the same license.
- Stanford CoreNLP would add about 250MB to the runable jar file, which is 10 times its current size. And the github repository isn't big enough to keep such large binaries around.
- The book parser consumes multiple GBs of RAM and multiple cores for 20 minutes to 2 hours to parse just one book, depending on the length of the book.
If you want the Java source code, I'd be happy to add it to a new public repository, but you would need to download the source code, compile it, and link it to Stanford CoreNLP. Are you still interested?
rjbprime
I have a question regarding the addition of new book txt files. Where would one find new premade sets to import? Or is it fairly easy to make new ones? Googling does not seem to bring up anything, but I must admit, I didn't look for long.
Is there a script of some sort to automate this, or is it done by hand?