Resources

[dariusk]

This is an open issue where you can comment and add resources that might come in handy for NaNoGenMo.

There are already a ton of resources on the old resources threads for the 2013 edition and the 2014 edition.

[javierarce]

Maybe someone finds my collection of libraries and APIs useful: https://github.com/javierarce/toolbox

[cpressey]

After last year's NaNoGenMo I said I would try to extract useful things from the NaNoGenLab and package them in a more useful way, and, so, yes, this is what I extracted:

• T-Rext cleans up spacing and punctuation in a text file.
• Guten-gutter attempts to strip Project Gutenberg boilerplate from a text file (and succeeds more often than gutenizer does)
• seedbank makes any Python script record and be able to replay the random seeds it chooses, with a one-line change. Useful for improving reproducibility.

They all require Python 2.x (tested on Python 2.7.6 but will probably still work on some earlier versions) but only seedbank requires that you write your script in Python - the other two can be used as stand-alone tools.

They're all in the public domain, as is all the stuff in the NaNoGenLab. So, please steal, fold, spindle, and mutilate as you see fit.

[MichaelPaulukonis]

Conceptually, you might want to stumble around The Living Handbook of Narratology - for some overviews of narrative theory, and some alternate takes on what might constitute narrative.

[ikarth]

ProcJam is more visually oriented, but it does have some useful resources that come out of it, like Tracery, a JavaScript library for generating stories from expanding grammars.

[MichaelPaulukonis]

Tracey was recently mentioned in Generative Text - the mailing list/google-group that sprouted from the end of NNGM2014. Lately, there's been a lot of partisan ranting about templates. I may resemble some of that remark.

[enkiv2]

I'd like to point out that the BotAlly slack group has a NaNoGenMo room, and that the GenArt slack group that sprouted from the Generative Text mailing list is mostly full of NaNoGenMo2014 people.

On Mon, Oct 26, 2015 at 11:40 AM Michael Paulukonis < notifications@github.com> wrote:

Tracey was recently mentioned in Generative Text https://groups.google.com/forum/#!forum/generativetext - the mailing list/google-group that sprouted from the end of NNGM2014. Lately, there's been a lot of partisan ranting about templates. I may resemble some of that remark.

— Reply to this email directly or view it on GitHub https://github.com/dariusk/NaNoGenMo-2015/issues/1#issuecomment-151178023 .

[ikarth]

Since we're speaking of visual novels, I should probably mention the Yarn dialog editor. I don't know that'll be all that useful directly for generative stuff, but in case someone is trying to procedurally generate a dialog tree, it might be handy. Or you could always generate a Twine file.

[ikarth]

@enkiv2 Where's the signup for the GenArt group? I need to check that out.

[enkiv2]

The genart group is at https://generativeart.slack.com ; I think it might be invite-only. Mike Paulukonis controls the group & can probably send invites.

On Mon, Oct 26, 2015 at 12:15 PM Isaac Karth notifications@github.com wrote:

@enkiv2 https://github.com/enkiv2 Where's the signup for the GenArt group? I need to check that out.

— Reply to this email directly or view it on GitHub https://github.com/dariusk/NaNoGenMo-2015/issues/1#issuecomment-151193274 .

[enkiv2]

As for botally, the public signup is here: http://t.co/OiOB7197BH ; people periodically check for registrations and send invites out.

On Mon, Oct 26, 2015 at 12:18 PM John Ohno john.ohno@gmail.com wrote:

The genart group is at https://generativeart.slack.com ; I think it might be invite-only. Mike Paulukonis controls the group & can probably send invites.

On Mon, Oct 26, 2015 at 12:15 PM Isaac Karth notifications@github.com wrote:

@enkiv2 https://github.com/enkiv2 Where's the signup for the GenArt group? I need to check that out.

— Reply to this email directly or view it on GitHub https://github.com/dariusk/NaNoGenMo-2015/issues/1#issuecomment-151193274 .

[dariusk]

NB: I'm one of three admins for the #BotALLY slack--we welcome new additions but we take our code of conduct VERY seriously so please only join if you're okay with being in a place where you almost inevitability will be nudged by an admin concerning your conduct.

[MichaelPaulukonis]

Yeah, there's not a lot going on in https://generativeart.slack.com - and if there's a way to allow people to request an invite, I'm all ears. @dariusk - what's the source for the #botALLY registration doc+script?

---

UPDATE: he's the GenerativeArt invite page: http://tinyurl.com/genartslackrequest

[ikarth]

@MichaelPaulukonis I've seen people use a Google Form to manage signups for Slack groups.

[enkiv2]

So far as I can tell, the signup just is a web form that sends an email to the slack maintainer. Maybe google forms has this functionality by itself?

On Mon, Oct 26, 2015 at 12:29 PM Isaac Karth notifications@github.com wrote:

@MichaelPaulukonis https://github.com/MichaelPaulukonis I've seen people use a Google Form to manage signups for Slack groups.

— Reply to this email directly or view it on GitHub https://github.com/dariusk/NaNoGenMo-2015/issues/1#issuecomment-151197445 .

[dariusk]

It just emails us.

Let's stop this conversation, I don't want to add too much noise to the resources thread.

[ikarth]

There's been a lot of work done with Neural Networks this year, including text-based output (such as the generated Magic: the Gathering cards). That might be an interesting avenue to pursue.

Abulafia uses MediaWiki for its generators. Not directly useful, but I keep forgetting to mention it here.

Here's a method of procedurally generating a wilderness in a text adventure format. People who are doing a Swallows type simulation approach may find its ideas useful.

[vijithassar]

Multiverse JSON is a light JSON syntax/spec and Python compiler script with which to store editorial projects in small logical units so they can be quickly reconfigured according to build parameters you define.

[aparrish]

Since last year's NaNoGenMo I've made a few libraries that might be of interest:

• pronouncingpy, a simple and easy-to-use interface to the CMU pronouncing dictionary
• pronouncingjs, a Javascript port of the above
• pycorpora, a Python interface for Corpora Project

I also made Context-Free GenGen, which is sort of like a mini-Tracery except driven by Google Sheets (a la the original GenGen). Probably not great for making a whole novel, but maybe cool for prototyping ideas with context-free grammar generation.

[mgiraldo]

• some NYPL-related resources
• if you want to add some savory meals to your novel, you could try the NYPL menus dataset
• a sanitized version of the menus dataset by @trevormunoz and Katie Rawson

[enkiv2]

I have a few tools I've made in the past year too:

• GGC https://github.com/enkiv2/ggc, a language for generative grammars that compiles to python (very similar to tracery)

Some novelty filters:

• dirtyreader https://github.com/enkiv2/dirtyreader, a tool to increase the profanity of text in a configurable way
• aphasia https://github.com/enkiv2/aphasia and synonym-warp https://github.com/enkiv2/synonym-warp, tools for replacing words with related words
• tone-poem https://github.com/enkiv2/tonepoem, blackout-gen https://github.com/enkiv2/blackout-gen, sestina-gen https://github.com/enkiv2/sestina-gen, and haiku-generator https://github.com/enkiv2/haiku-generator: tools for producing different varieties of poems from source text

And, some novelty generators:

• anxiety-generator https://github.com/enkiv2/anxiety-generator, a python port of Paul Ford's tool for procedurally generated passive-aggressive verbal abuse
• Tinder For Guinea Pigs https://github.com/enkiv2/tfgp, which generates nonsense startup pitches
• tftf-gen https://github.com/enkiv2/tftf-gen, which describes randomly generated science fiction set pieces (based on a description of Thing From The Future)

On Mon, Oct 26, 2015 at 2:21 PM Allison Parrish notifications@github.com wrote:

Since last year's NaNoGenMo I've made a few libraries that might be of interest:

• pronouncingpy https://github.com/aparrish/pronouncingpy, a simple and easy-to-use interface to the CMU pronouncing dictionary
• pronouncingjs https://github.com/aparrish/pronouncingjs, a Javascript port of the above
• pycorpora https://github.com/aparrish/pycorpora, a Python interface for Corpora Project https://github.com/dariusk/corpora

I also made Context-Free GenGen http://cfgg.decontextualize.com/, which is sort of like a mini-Tracery except driven by Google Sheets (a la the original GenGen http://tinysubversions.com/gengen/). Probably not great for making a whole novel, but maybe cool for prototyping ideas with context-free grammar generation.

— Reply to this email directly or view it on GitHub https://github.com/dariusk/NaNoGenMo-2015/issues/1#issuecomment-151236207 .

[ikarth]

If you're looking for a large corpus of internet comments, the mostly complete Reddit corpus is available.

[coleww]

i have been pretty heavily tooling up for nanogenmo this year by way of publishing a ton of poetic node modules to npm.

• word-vomit replaces words with other words of the same part of speech
• queneau buckets for letters
• queneau buckets for words
• replace words with words that rhyme
• n-plus, replace words with the word N words ahead of it in a dicitonary
• replace character sequences with other character sequences, good for l-system poetry potentially
• detect lipogrammatic text (omits certain characters)
• detect text that snowballs (i.e, every word is 1 char longer than the previous)
• add some random diacritic marks to yr text

Another resource i have created is the weirdly-specific-corpora project, which is a fork of dariusk/corpora for any lists that are too weirdly specific to be included in that project. I expect I will end up making a lot of weird lists while generating novels.

[dariusk]

Someone (I believe @jkirchartz) posted this link to an actual spam blog comment generator template that was accidentally posted as a comment on a guy's blog.

[enkiv2]

There is a generator for hilariously poorly written sex scenes http://www.fiftyshadesgenerator.com/. The source is embedded in the page there.

On Tue, Oct 27, 2015 at 4:14 PM Darius Kazemi notifications@github.com wrote:

Someone (I believe @jkirchartz https://github.com/jkirchartz) posted this link to an actual spam blog comment generator template http://alexking.org/blog/2013/12/22/spam-comment-generator-script that was accidentally posted as a comment on a guy's blog.

— Reply to this email directly or view it on GitHub https://github.com/dariusk/NaNoGenMo-2015/issues/1#issuecomment-151646047 .

[MichaelPaulukonis]

See sample code in various languages via the Language Survey, issue #17

[ikarth]

@enkiv2 suggested that a short introduction to some historical techniques might be useful in the resources thread to point out possible approaches for people who might not be familiar with some of the precedents of text generation. So I thought I'd write about a few of them.

Dada incorporated several techniques that involved randomly assembling prior texts into new poetry. The cut-up technique takes an existing text, cuts it to pieces, and then reassembles the bits in a new order. This list of Surrealist techniques might also be inspirational.

Oulipo is a group of writers who use constraints to define their writing. Relevant examples include Queneau's Cent Mille Milliards de Poèmes (A Hundred Thousand Billion Poems), which combines ten-line sonnets line-by-line; Perec's Life a User's Manual, which is structured like a knight's tour of a chessboard; Queneau's Excercises in Style, 99 retellings of the same story; Calvino's The Castle of Crossed Destinies, stories interperted via Tarot cards; and Queneau's "A Story as You Like It".

Even where they aren't directly generative, the approaches used by Oulipo writers often point to alternative forms a novel can take. For example, Calvino's Invisible Cities and If on a winter's night a traveler.

Oulipo is also significant because many of its members wrote about systems for generating stories and about incorporating computers into writing. Italio Calvino's essay "Prose and Anticombinatorics," about using the computer to to find the constraints for a murder mystery; Paul Fournel's "Computer and Writer: The Centre Pompidou Experiment"; and Claude Berge's "For a Potential Analysis of Combinatory Literature" are particularly relevant.

Lastly, Jorges Luis Borges tends to crop up a lot, particularly for his short stories: "The Library of Babel", "The Book of Sand", "An Examination of the Work of Herbert Quain", "The Garden of Forking Paths", "Pierre Menard, Author of the Quixote", and "Tlön, Uqbar, Orbis Tertius".

[ikarth]

After @rbechtel mentioned the source code for Micro-Talespin, I went poking around to see what other early AI storytelling systems had source code availible.

Micro-Talespin in Common Lisp, by Warren Sack another source [About Talespin](http://lispm.de/mts https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/)

For Tale-Spin itself: In 2008 Meehan found a copy of the original MLisp source code for Tale-Spin (sans the data files). There's an extensive discussion of the source code in Wardrip-Fruin's Expressive Processing, though I'm not aware of any copies online.

Eliza/Doctor: In Java Also in Java BASIC JavaScript Python The original Lisp source code- github repo

Skald: a Scala reimplementation of Minstrel Remixed, which is itself a Scala reimplementation of Scott Turner's Minstrel (originally in Lisp). Scott Turner on MINSTREL.

[enkiv2]

As an example of exercises in minimalism and apophenia, Nick Monfort has a collection of one kilobyte story generators in python. https://grandtextauto.soe.ucsc.edu/2008/11/30/three-1k-story-generators/

On Sat, Oct 31, 2015 at 11:20 AM Isaac Karth notifications@github.com wrote:

After @rbechtel mentioned the source code for Micro-Talespin https://github.com/dariusk/NaNoGenMo-2015/issues/59, I went poking around to see what other early AI storytelling systems had source code availible.

Micro-Talespin in Common Lisp, by Warren Sack http://lispm.de/source/misc/micro-talespin.lisp another source http://eliterature.org/images/microtalespin.txt About Talespin http://lispm.de/mts%20https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/

For Tale-Spin itself https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/: In 2008 Meehan found a copy of the original MLisp source code for Tale-Spin (sans the data files). There's an extensive discussion of the source code in Wardrip-Fruin's Expressive Processing, though I'm not aware of any copies online.

Eliza/Doctor: https://en.wikipedia.org/wiki/ELIZA In Java https://code.google.com/p/simple-semantic-desktop/source/browse/trunk/Progs2/Eliza/eliza.java?r=4 Also in Java http://www.chayden.net/eliza/Eliza.html BASIC https://www.jesperjuul.net/eliza/ELIZA.BAS JavaScript https://www.jesperjuul.net/eliza/ Python http://www.jezuk.co.uk/cgi-bin/view/software/eliza The original Lisp source code http://elizagen.org/- github repo https://github.com/jeffshrager/elizagen

Skald https://sites.google.com/a/soe.ucsc.edu/eis-skald/: a Scala reimplementation of Minstrel Remixed, which is itself a Scala reimplementation of Scott Turner's Minstrel (originally in Lisp). Scott Turner on MINSTREL https://grandtextauto.soe.ucsc.edu/2007/10/30/scott-turner-on-minstrel/.

— Reply to this email directly or view it on GitHub https://github.com/dariusk/NaNoGenMo-2015/issues/1#issuecomment-152746654 .

[MichaelPaulukonis]

@ikarth et alia - this year I have been looking at Tale-Spin a bit, even though I don't think I will be using it as a model for what I will be doing. However, I do have some more resources in my NMGM repo and will be adding more there as the month progresses.

[ikarth]

Public service announcement: if you try to download too many files from Project Gutenberg too quickly, they'll ban you for 24 hours.

[dariusk]

If you're hitting Gutenberg rate limits there are also tons of mirror sites: http://www.gutenberg.org/MIRRORS.ALL

[enkiv2]

Gutenberg has a bot policy & instructions on how to avoid rate limits: https://www.gutenberg.org/wiki/Gutenberg:Information_About_Robot_Access_to_our_Pages

(Not sure if this applies to official mirrors. Unofficial mirrors are also floating around.)

On Sun, Nov 1, 2015 at 9:44 AM Darius Kazemi notifications@github.com wrote:

If you're hitting Gutenberg rate limits there are also tons of mirror sites: http://www.gutenberg.org/MIRRORS.ALL

— Reply to this email directly or view it on GitHub https://github.com/dariusk/NaNoGenMo-2015/issues/1#issuecomment-152831132 .

[hugovk]

Project Gutenberg also has CDs and DVDs of many books for offline use, I've used one of these a lot. https://www.gutenberg.org/wiki/Gutenberg:The_CD_and_DVD_Project

[ikarth]

ORBIS: The Stanford Geospatial Network Model of the Roman World has an API: http://orbis.stanford.edu/api/

[PPKFS]

https://github.com/TheCommieDuck/nell-extractor

I've just written a quick thing that extracts useful data from CMU's NELL ontology.

Output is a list such as:

lemon_balm_tea beverage agriculturalproduct Lemon Balm Tea

which is an entity called lemon_balm tea, which is a type of beverage and a type of agriculturalproduct, and is called Lemon Balm Tea in English.

Plenty of false-positive entries - beverage general gives about 8,700 entries, mostly different kinds of tea and wine - but it's nice enough if someone wants to try the ontological approach?

I'll probably be refining it throughout the month.

[MichaelPaulukonis]

For those interested in Project Gutenberg texts, there is also GITenberg, mentioned in #89 by NNGM participant @sethwoodworth

[hugovk]

If you need food, NYPL has an API around (and CSV dumps of) their historical collection of restaurant menus at What's On The Menu?

http://menus.nypl.org/data

Lots of the dishes and prices have been transcribed and are also available via the API.

I made a Python interface: https://github.com/hugovk/whatsonthemenu

(for https://twitter.com/menubot and http://menubot.tumblr.com)

[ikarth]

For those of us using NLTK: Text Analysis with NLTK Cheatsheet: https://blogs.princeton.edu/etc/files/2014/03/Text-Analysis-with-NLTK-Cheatsheet.pdf

[jemisa]

neural-storyteller is a recurrent neural network for generating little stories about images. Code is at https://github.com/ryankiros/neural-storyteller and samples https://medium.com/@samim/generating-stories-about-images-d163ba41e4ed

[ikarth]

Linked Data has links to a large number of datasets, many of which are also Open Data. Datahub lists a large number of datasetes, and can be filtered by license, if you're only interested in public domain or Creative Commons datasets. Pleiades is a gazetter of ancient geographic places.

[ikarth]

Baba - a Javascript-based designer for text generators. A generator that outputs text generators should be a useful resource for non-coders, but it looks like the live web demo is down so it'll take a little bit of work to get it to run if you download it.

[ikarth]

I've created a thread for beginner's questions and tutorials, as a supplement to the main Resources thread: #152

[ikarth]

Working in Python? Having unicode issues? Even if you properly take care of the proper encoding and decoding on your end, your source data may be incorrectly encoded. Or, worse, Windows might have mucked it up. If you've got “ or — all over your text, you might want to consider using ftfy. Fixes bad unicode and mojibake; doesn't convert non-unicode to unicode.

[hugovk]

If you want to avoid certain words in your output text, here's a number of things. These are mainly aimed at bots, but should still be generally useful.

Here's a JavaScript, Python, Ruby and PHP word filter: https://github.com/dariusk/wordfilter

Here's a headline filter: https://github.com/molly/CyberPrefixer/blob/master/offensive.py

Tips on transphobic joke detection: http://tinysubversions.com/notes/transphobic-joke-detection/

Some lists of bad words: https://github.com/shutterstock/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words https://gist.github.com/ryanlewis/a37739d710ccdb4b406d http://www.bannedwordlist.com/lists/swearWords.txt

[Inactive] muted Twitter topics: https://github.com/sjml/bot-innocence

Some general etiquette things: http://tinysubversions.com/2013/03/basic-twitter-bot-etiquette/ http://www.crummy.com/2013/11/27/0

[greg-kennedy]

If you need to plug in some Markov Chains in Perl, you could do a lot worse than the module I built for it this year. https://github.com/greg-kennedy/MarkovChain

[MichaelPaulukonis]

• List of Longest Novels (not necessarily available as free etexts)
• Top 100 Gutenberg texts - changes weekly
• Project Gutenberg tips-n-tricks
• Vocabulary analysis of Project Gutenberg

[MichaelPaulukonis]

For anyone working with a twitter-sourced corpus, myna is a "JS lib that compiles a tweet to machine speakable text". Or non-twitter-literate text.

For example, the tweet:

"OH: Myna is #SuperAwsome for #TTS and #TwitterBeginners! It is written by @kn bit.ly/ykbz6b"

compiles into plain English:

"Katsuya Noguchi overheard: Myna is Super Awesome for TTS and Twitter Beginners! It is written by Katsuya Noguchi"

[dariusk]

Hey all. Since this is our most popular (and most-subscribed to) issue, I just wanted to let people know I have a new issue open to discuss whether we do NaNoGenMo 2016 here on Github again, or somewhere else. Feel free to participate! https://github.com/dariusk/NaNoGenMo-2015/issues/199