Writing E-Team - Githubissues

francois-here commented 3 years ago

Here the code. Here the 50000 word output.

Initial intent

Hi there, Using gpt-2, I'm interested in exploring tricks to bring narrative coherence across longer texts, possibly recycling some ideas from https://www.scritturacollettiva.org/metodo.html. Looking forward to emulation from other participants. KR Francois

Execution

I went out with the naive idea that gpt could understand the structure of a text if trained accordingly. So the training material includes text snippets (pompously called chapters) with a prompt containing: • a summary of the snippet created with Bert Summarizer and tagged with <|summary|>, • the third paragraph of the snippet tagged <|line-03|>, • the final paragraph of the snippet tagged <|line-last|>, • the first paragraph of the snippet tagged <|chapter-begin|>. And, in deed, even though gpt-2 writes sequence by sequence, it did recognize a pattern and often picked up the first sentence and the third one. The final one mostly got lost. But that picking of the third sentence seemed to be mostly statistics-based. It does not stick to the rest of the text.

pjfpotter commented 3 years ago

narrative coherence ay? A bold move. I must find out more about this gpt-2, also loving the Scrittura Industriale Collettiva

francois-here commented 3 years ago

An extract of the training file

<|chapter|> <|summary|>I was not quite sure whether they had locked the door; and when I dared move, I got up and went to see. <|MISS009|>, who had a spoiled temper, a very acrid spite, a captious and insolent carriage, was universally indulged. Her beauty, her pink cheeks and golden curls, seemed to give delight to all who looked at her, and to purchase indemnity for every fault. <|line-03|>All <|FAMILY015|> <|FAMILY003|>’s violent tyrannies, all his sisters’ proud indifference, all his mother’s aversion, all the servants’ partiality, turned up ... <|line-last|> I could not answer the ceaseless inward question—why I thus suffered; now, at the distance of—I will not say how many years, I see it clearly. <|chapter-begin|> My seat, to which <|SURNAME001|> and the bitter Miss <|MISS008|> had left me riveted, was a low ottoman near the marble chimney-piece; the bed rose before me; to my right hand there was the high, dark wardrobe, with subdued, broken reflections varying the gloss of its panels; to my left were the muffled windows; a great looking-glass between them repeated the vacant majesty of the bed and room. I was not quite sure whether they had locked the door; and when I dared move, I got up and went to see. Alas! yes: no jail was ever more secure. Returning, I had to cross before the looking-glass; my fascinated glance involuntarily explored the depth it revealed. All looked colder and darker in that visionary hollow than in reality: and the strange little figure there gazing at me, with a white face and arms specking the gloom, and glittering eyes of fear moving where all else was still, had the effect of a real spirit: I thought it like one of the tiny phantoms, half fairy, half imp, <|SURNAME001|>’s evening stories represented as coming out of lone, ferny dells in moors, and appearing before the eyes of belated travellers. I returned to my stool. Superstition was with me at that moment; but it was not yet her hour for complete victory: my blood was still warm; the mood of the revolted slave was still bracing me with its bitter vigour; I had to stem a rapid rush of retrospective thought before I quailed to the dismal present. All <|FAMILY015|> <|FAMILY003|>’s violent tyrannies, all his sisters’ proud indifference, all his mother’s aversion, all the servants’ partiality, turned up in my disturbed mind like a dark deposit in a turbid well. Why was I always suffering, always browbeaten, always accused, for ever condemned? Why could I never please? Why was it useless to try to win any one’s favour? <|SURNAME007|>, who was headstrong and selfish, was respected. <|MISS009|>, who had a spoiled temper, a very acrid spite, a captious and insolent carriage, was universally indulged. Her beauty, her pink cheeks and golden curls, seemed to give delight to all who looked at her, and to purchase indemnity for every fault. <|FAMILY015|> no one thwarted, much less punished; though he twisted the necks of the pigeons, killed the little pea-chicks, set the dogs at the sheep, stripped the hothouse vines of their fruit, and broke the buds off the choicest plants in the conservatory: he called his mother “old girl,” too; sometimes reviled her for her dark skin, similar to his own; bluntly disregarded her wishes; not unfrequently tore and spoiled her silk attire; and he was still “her own darling.” I dared commit no fault: I strove to fulfil every duty; and I was termed naughty and tiresome, sullen and sneaking, from morning to noon, and from noon to night. My head still ached and bled with the blow and fall I had received: no one had reproved <|FAMILY015|> for wantonly striking me; and because I had turned against him to avert farther irrational violence, I was loaded with general opprobrium. “Unjust!—unjust!” said my reason, forced by the agonising stimulus into precocious though transitory power: and Resolve, equally wrought up, instigated some strange expedient to achieve escape from insupportable oppression—as running away, or, if that could not be effected, never eating or drinking more, and letting myself die. What a consternation of soul was mine that dreary afternoon! How all my brain was in tumult, and all my heart in insurrection! Yet in what darkness, what dense ignorance, was the mental battle fought! I could not answer the ceaseless inward question—why I thus suffered; now, at the distance of—I will not say how many years, I see it clearly. <|chapter-end|>

francois-here commented 3 years ago

Code for text generation

#Watchdog
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import shutil
import linecache
import gc
import re
import fileinput
import datetime
import sys #enable import of mymodules
sys.path.insert(1, '/home/francois/gpt-2/') # insert at 1, 0 is the script path (or '' in REPL)
import mymodules #from /src folder
from playsound import playsound

DIR_TO_WATCH = "/home/francois/gpt-2/writer/input_auto1"

class Watcher:
    DIR_TO_WATCH = DIR_TO_WATCH
    print("Ready to receive input "+DIR_TO_WATCH)

    def __init__(self):
        self.observer = Observer()

    def run(self):
        event_handler = Handler()
        self.observer.schedule(event_handler, self.DIR_TO_WATCH, recursive=True)
        self.observer.start()
        try:
            while True:
                time.sleep(5)
        except:
            self.observer.stop()
            print("Error")
        self.observer.join()

class Handler(FileSystemEventHandler):
    DIR_TO_WATCH = DIR_TO_WATCH

    @staticmethod
    def on_any_event(event):
        if event.is_directory:
            return None

        elif event.event_type == 'created':
            # Take any action here when a file is first created.
            print("Received created event - %s." % event.src_path)
            filepath="/home/francois/gpt-2/writer/input_auto1/"
            filename = event.src_path.strip(filepath)

            def shorten_line_to_lastpoint(inputtext):
                questionmark_position = inputtext.rfind("?")
                finaldot_position = inputtext.rfind(".")
                if questionmark_position != -1:
                    outputtext = inputtext[0:questionmark_position+1]
                elif finaldot_position != -1:
                    outputtext = inputtext[0:finaldot_position+1]
                return outputtext

            def shorten_line_to_firstpoint(inputtext):
                questionmark_position = inputtext.rfind("?")
                dot_position = inputtext.rfind(".")
                exclam_position = inputtext.rfind("!")
                comma_position = inputtext.rfind(",")
                if questionmark_position != -1:
                    outputtext = inputtext.split("?")
                    outputtext = outputtext[0]+"?"
                elif dot_position != -1:
                    outputtext = inputtext.split(".")
                    outputtext = outputtext[0]+"."
                elif exclam_position != -1:
                    outputtext = inputtext.split("!")
                    outputtext = outputtext[0]+"!"
                elif comma_position != -1:
                    outputtext = inputtext.split(",")
                    outputtext = outputtext[0]+","
                else:
                    outputtext = inputtext[:150]+"..."
                if len(outputtext) > 160:
                    outputtext = inputtext[:157]+"..."
                return outputtext

            def shorten_lastline_to_lastsentence(inputtext):
                dot_position = inputtext.rfind(".")
                dot_position = inputtext.rfind(".",0,dot_position)
                #print(dot_position)
                questionmark_position = inputtext.rfind("?")
                questionmark_position = inputtext.rfind("?",0,questionmark_position)
                #print(questionmark_position)
                exclam_position = inputtext.rfind("!")
                exclam_position = inputtext.rfind("!",0,exclam_position)
                #print(exclam_position)
                if dot_position != -1:
                    outputtext = inputtext.split(".",dot_position)
                    outputtext = outputtext[-1]
                elif questionmark_position != -1:
                    outputtext = inputtext.split("?",questionmark_position)
                    outputtext = outputtext[-1]
                elif exclam_position != -1:
                    outputtext = inputtext.split("!",exclam_position)
                    outputtext = outputtext[-1]
                else:
                    outputtext = inputtext
                return outputtext

            def file_number_of_lines(fname):
                with open(fname) as f:
                    for i, l in enumerate(f):
                        pass
                return i + 1

            def number_of_gpt_run(number_of_lines):
                number_of_run = (number_of_lines-1)/(3)
                print("Number of runs = "+str(number_of_run))
                if str(number_of_run).isdigit() == False:
                    number_of_run = str(number_of_run).split(".",1)
                    number_of_run = number_of_run[0]
                return number_of_run

            def create_bookfile(bookcontent,bookfilename):
                #Create output book file
                BOOKFILE = "/home/francois/gpt-2/writer/output_auto/"+bookfilename
                f = open(BOOKFILE, "x") 
                f.write(bookcontent)
                f.close()
                return print("Created book file "+BOOKFILE)

            def create_tmpgpt_input(tmp_linesummary,tmp_line1,tmp_line3,tmp_linefinal):
                #Create gpt-2 input (without file)
                gptinput = "<|chapter|>\n" + "<|summary|>" + tmp_linesummary + "<|line-03|>" + tmp_line3
                gptinput = gptinput + "<|line-last|>" + tmp_linefinal + "<|chapter-begin|>\n" + tmp_line1
                print("Created gpt input ")
                return gptinput

            def select_tmplines(startline):
                #select temporary lines from input file
                tmp_line1 = linecache.getline(event.src_path,startline-2)
                tmp_line2 = linecache.getline(event.src_path,startline-1)
                tmp_linefinal = linecache.getline(event.src_path,startline)
                return tmp_line1, tmp_line2, tmp_linefinal

            def gpt_processing01(seedinput):
                #gpt processing
                model_name='writerrun06'
                models_dir="/home/francois/gpt-2/models/"
                outputlength=500                      #max 1024 tbd, Number of tokens to generate (default 1023, the maximum)
                temperature=1
                top_k=2000                              #max 50257, limits the generated guesses to the top k guesses
                top_p=1.15                               #Nucleus sampling: limits the generated guesses to a cumulative probability. (gets good results on a dataset with top_p=0.9)
                gptoutput = mymodules.single_interact_model_4(models_dir,model_name,seedinput,outputlength,temperature,top_k,top_p)
                return gptoutput

            def postedit_gptoutput(input):
                #post edit gpt-2 output
                output = input.split("<|chapter-end|>",1)
                output = output[0]
                return output

            linetotal = file_number_of_lines(event.src_path)
            print("The file has " + str(linetotal) + " lines.")
            number_of_gpt_run = number_of_gpt_run(linetotal)
            print("Number of runs is "+number_of_gpt_run)
            number_of_run = (linetotal-1)/(3)
            summary = linecache.getline(event.src_path,1) #summary has to be provided as the single first line of the input file
            bookcontent = summary
            i = 1
            while i < number_of_run+1:
                print("Run #"+str(i))
                n=i*3+1
                tmplines = select_tmplines(n)
                seedinput = create_tmpgpt_input(summary,tmplines[0],tmplines[1],tmplines[2])
                print(seedinput)
                gptoutput = gpt_processing01(seedinput)
                print(gptoutput)
                gptoutput = postedit_gptoutput(gptoutput)
                bookcontent = bookcontent+tmplines[0]+gptoutput+tmplines[2]
                #Release memory
                unreachable_objects = gc.collect()
                #Number of collected and deallocated objects
                print("Unreachable objects: "+str(unreachable_objects))
                i += 1

            #tmplines = select_tmplines(1)
            timestampnow = datetime.datetime.now()
            tmpfileID = filename+"_"+timestampnow.strftime("%Y-%m-%d_%H%M_%S")

            #Book creation
            bookfilename = tmpfileID+".txt"
            create_bookfile(bookcontent,bookfilename)

            #Back up processed file
            shutil.move(event.src_path, "/home/francois/gpt-2/writer/"+filename)
            #Release memory
            unreachable_objects = gc.collect()
            #Number of collected and deallocated objects
            #print("Unreachable objects: "+str(unreachable_objects))
            print("Processed successfully "+filename)
            playsound("/home/francois/Music/python_bowlb.wav")

        elif event.event_type == 'modified':
            # Taken any action here when a file is modified.
            print("Received modified event - %s." % event.src_path)

        elif event.event_type == 'deleted':
            # Taken any action here when a file is modified.
            print("Received deleted event - %s." % event.src_path)

        elif event.event_type == 'moved':
            # Taken any action here when a file is modified.
            print("Received moved event - %s." % event.src_path)

if __name__ == '__main__':
    w = Watcher()
    w.run()

francois-here commented 3 years ago

Chapter 1

Chapter 2

Chapter 3

Chapter 4

<|MISS028|> <|MISS004|>’s intimacy at Hartfield was soon a settled thing. Quick and decided in her ways, <|MISS027|> lost no time in inviting, encouraging, and telling her <|MISS022|> come very often; and as their acquaintance increased, so did their satisfaction in each other. Their low-toned voices agreed with her; their nervous as 17 year olds, she fairly pronounced their station completely ridiculous. “I shall not doubt <|MISS019|>,” said <|SURNAME004|>, when Norn Miss <|MISS004|> came up. “When I meet with real poetry, I cannot rest till I have learned <|MISS015|> by heart, <|MISS026|> so made <|MISS015|> partly mine. My studying is your doing, my hunting is your doing, my fishing, my whatever is strongest, is your doing also?” Her voice, sometimes harsh, sometimes mild, always expressive, had <|MISS014|> sweetest cadence of appeal. “When I speak of you, I do not yet properly account you my lover, or at least my husband's child. I have a regard for you, though I know you cannot love me, yet I cannot deny <|MISS019|> you are mine. You are young, beautiful, meritorious, outstanding gentleman—at least, <|MISS014|> most exceptional of the purest sex. I may be hideous in my own eyes, but at <|MISS014|> behind which you view <|MISS015|> you see a righteous giant. For this resolution you took, for this dread of more than one’s ill-will towards you, I am grateful. I firmly believe you were resolved <|MISS026|> disposed to be firm to me; to which, indeed, I was a little sensitive at first. But I proved impervious to prejudiced motives at first, <|MISS026|> then, if I must make a hard bargain, I shall make <|MISS015|> anyway. Where there is profit—trial, <|MISS026|> sometimes suffering—I shall cross over. Would not <|MISS015|> be a bad you? This, indeed, is a precious proposition. I will hear you thrice before I forsake it. Not a second of time passes. I hold out my hand <|MISS026|> say to you—ye dare say it—enter.” Norn how she boarded. One of her classmates, a girl of eighteen, moved quickly. Another step stamped onAnd he was really a very pleasing young man, a young man whom any woman not fastidious might like. He was reckoned very handsome; his person much admired in general, though not by her, there being a want of elegance of feature which she could not dispense with:—but the girl who could be gratified by a <|FAMILY021|> <|FAMILY010|>’s riding <|MISS014|> the country <|MISS022|> get walnuts for her might very well be conquered by Mr. <|FAMILY002|>’s admiration.

Chapter 5

<|SURNAME100|> is the best girl to marry in town. “I do not know what your opinion may be, Mrs. <|FAMILY001|>,” said Mr. <|FAMILY003|>, “of this great intimacy between <|MISS027|> and <|MISS028|> <|MISS004|>, but I think it a bad thing.” “I think they will neither of <|MISS020|> do the other any good.” “Then they are not good; they are rather <|MISS014|> chiefs or gilders of scoundrels. I guess my little friend <|FAMILY 000|> has <|MISS028|> excellent chance of avoiding disgrace. If you cannot do this, I view it as a tribute to your discretion.” “My short cut!” inserted into <|MISS014|> midst of their Turban tour “Then they are not good; they are rather <|MISS014|> chiefs or gilders of scoundrels. I guess my little friend <|FAMILY 000|> has <|MISS028|> excellent chance of avoiding disgrace. What I want is, that you should write to your sisters <|MISS026|> tell them of <|MISS014|> desperate problems their state presents. I must acknowlege that some of them are extremely appalling. If you cannot do this, I view it as a tribute to your discretion. Otherwise I am blank <|MISS026|> mute.” “I suppose you are reluctant to speak out plainlyly, <|MISS007|>?” “I am not frank.” “Why? What is the matter?” “Because you are afraid of being plainly speaking, <|MISS007|>?” “Oh, that is sh! t! This city doesn’t deserve <|MISS019|> the best-looking <|MISS026|> <|MISS014|> personification of taste shall give it.” “Oh, that is sh! t! This city doesn’t deserve <|MISS019|> the best-looking <|MISS026|> <|MISS014|> personification of taste shall give it.” There were wishes at Randalls respecting <|MISS027|>’s destiny, but it was not desirable <|MISS022|> have <|MISS020|> suspected; and the quiet transition which Mr. <|FAMILY003|> soon afterwards made <|MISS022|> “What does <|FAMILY001|> think of the weather; shall we have rain?” convinced her that he had nothing more <|MISS022|> say or surmise <|MISS014|> Hartfield. Mr. <|FAMILY003|> therefore took <|MISS028|> early departure, evidently very busy indeed. SURNAME100|> is <|MISS014|> best girl, <|MISS026|> when she is good she is excellent. You cannot imagine <|MISS015|>'s fascination. There is music, dancing, <|MISS026|> sentiment, under her cotillion. […] My little friend! she is all set to bomulous. Soon after 10 p.m. we had another meal, <|MISS026|> then we separately vacated <|MISS014|> dining-room. <|SURNAME001|> would have guarded <|MISS014|> vault in <|MISS014|> room for which she was responsible, but I preferred speaking to her self-respecting neighbours. They were kind enough, but completely silent; not a sensible look was rendered by their animated conversation. I suppose they believed <|MISS019|> they could not inconvenience themselves with <|MISS014|> presentation all <|MISS014|> gifts of speech; <|MISS026|> so in response to my wish to eat something, they grudgingly obliged me to show <|MISS014|> tolerable appetite of gratitude; <|MISS026|> after I had given them something to swallow, <|MISS015|> was imagined <|MISS019|> I had washed down their confection with my own digestive juices. This I did. I lingered till Grace came. She followed me into <|MISS014|> dining-room, where she sat with her mother <|MISS026|> grandmother. She looked terrible, though she had just regained <|MISS019|> of oil <|MISS026|> perfume she had lost, <|MISS026|> was reduced to her motherly energy. After all, though her brown eyes {|MISS026|> slightly up-turned estate might be termed fertile, they were but desert sites—consecrated by fate to verdant peak <|MISS026|> beautiful valley. There was something in <|MISS014|> texture of her nature-- <|MISS019|> may be termed ideal. <|chapter-end

francois-here commented 3 years ago

That's it for this year.

francois-here commented 3 years ago

I see, I should use labels and not close the issue :). But how to do it?

francois-here commented 3 years ago

Note on chapter input

The first input file has 4 lines The first line acts as a summary. The second, third and fourth lines are the text that will be expanded by GPT by adding text between existing lines. Chapter 1 was processed 3 times from the sample below to the output above.

Sample chapter input (chapter 1)

hugovk commented 3 years ago

I've added a preview label! Do you have some 50k output somewhere, to get a completed label?

francois-here commented 3 years ago

I've added a preview label! Do you have some 50k output somewhere, to get a completed label?

Hi Hugo, Thanks for asking. The code and the 50000 word output are added.

NaNoGenMo / 2020

Writing E-Team #17

Initial intent

Execution

An extract of the training file

Code for text generation

Chapter 1

Chapter 2

Chapter 3

Chapter 4

Chapter 5

Note on chapter input

Sample chapter input (chapter 1)