donglixp
commented
4 years ago Hi @GenTxt ,
I ran a sampling using the same seed prompt. The results are as follows:
Winston sat back . A sense of complete helplessness had descended upon him . A deep breath of fear filled him . He was too late . In fact the last thing he had thought was the thought that he was doomed to this world , and no matter how soon , he would get to this . To escape this life , he was the only way to survive in this life . He wanted to be home , to find the others and return safely to earth , so that he could find the family of his great . Winston felt nothing wrong with his situation . He would be in no position to tell the other passengers at the boarding platforms that they would be rescued , and he could make certain that he never got to the ship . He would never take it to the safety of the vessel and not return . Once he had started his flight , he would be allowed to board the passenger compartment . Winston knew that if the ship got stuck in a mud lake , he would likely die there , which his life would be spared . He knew that he would not be able to escape the wreckage of ship without even getting through it or getting in the air currents of the mud . All of his thoughts had gone straight back to the day he had first dreamed up a life in this earth and so he would not ever regret this decision again . It was all just an accident and he didn ’ t need anything . So he was in a nominative position for the entire voyage on this ship but no one dared try anything in such a dire situation . He was just the one being held in his own , no matter how hard he tried . He had to be the one of the few people left that ever would see him go through this journey and get along with everyone . There would never be more sorrow on his shoulders to be brought back to life . Winston was still very angry that he had been sent over into the ocean for just this little time . He now felt that he should do nothing less to the fate of his children and siblings . Winston himself was not the kind of person to let anyone see how he felt . He was a young man , and he was a young orphan . Winston knew that he wasn ’ t the kind of person to put it off . It was just an accident and he was the type of person to take chances and make a mistake . He was too young for any of this to be his fault . There was no reason to blame his mistakes and he wanted to be on the ship , but he just couldn ’ t wait to start to be
We didn't implement the sampling function in the current huggingface unilm fork. Note that the sampled text would be different with different checkpoints. We use top-k (k=40) sampling algorithm to generate outputs. The flag --new_segment_ids is necessary to initialize the correct segment embedding (index=2) for L2R LM. We also need to sample the distribution using the masked-LM way.
GenTxt
I would appreciate if another user could share the sample commands that can reproduce the interesting results included with the arvix paper. No success so far with original MS repo and only limited, repetitious output using the current huggingface unilm fork and same prompt.
Cheers
Seed line from 1984 used in original paper:
Winston sat back. A sense of complete helplessness had descended upon him.
Published output:
He was now helpless and unmoving. What would that mean? To become a monk under her mother’s rule. What would it mean for him to be put at the center of the most dangerous of all of demons, the demon known as Azrael? He would never know exactly what that was, or even how it would end, unless he could sense the strength of Azrael’s mind and soul. It wasn’t only the existence of his mother, it was all he knew, and every moment. He was trapped within her embrace, a trapped, torn, battered, crippling presence he only knew it would mean was eternity. And in the instant that he was lost his life changed forever, when Azraelis’ spirit touched his mind, he was engulfed in this dark and mystical, so enchanting and beautiful force.