Machine Learning to Generate a new SCP

ChaelCodes / chaelcodes.github.io

Portfolio site for ChaelCodes. Blogs, podcasts, and talks hosted here!

https://www.chael.codes

MIT License

2 stars 0 forks source link

Machine Learning to Generate a new SCP #53

Open ChaelCodes opened 3 years ago

ChaelCodes commented 3 years ago

Pre-Talk

[ ] Create the talk on Google Slides
[ ] Add the talk to the talks folder in ChaelCodes Google Drive
[ ] Create a Talk promo image (1280x720) or (1920x1080)
[ ] Create a short talk promo bio

During the Talk

[ ] Record the Talk at 1280x720 or 1920x1080 for YouTube

After the Talk

[ ] Upload the Talk to YouTube
[ ] Add it to the Talks playlist
[ ] Release a Talk page with RSS update

Talk Page

[ ] Does it generate a good looking Twitter Image?
[ ] Have you embedded the YouTube video?
[ ] Have you added it to the /Talks page?
[ ] Is there a description of the major points?

YouTube Video

[ ] Thumbnail good?
[ ] Description?
[ ] 10-15 tags?
[ ] Do you have chapters?

Assets

Talk Thumbnail Promo image

Talk Description Short

30s clip for Twitter/Promotion of the intro of the talk

ChaelCodes commented 3 years ago

Tensorflow points directly to https://github.com/somaticio/tensorflow.rb, but it hasn't been updated in 3 years.

A lot of folks seem to use Google Collab. I know it uses Python.

Tensorflow offers tutorials.

Transformers are better than just RNN for sentence, paragraphs, and pages.

Ruby Tensorflow Docker image

GPT-2 in Tensorflow 2.0 in Python

ChaelCodes commented 3 years ago

SCP Wiki

ChaelCodes commented 3 years ago

OpenAI uses GPT3 https://beta.openai.com/docs/api-reference/introduction OMG!! GPT-3 KNOWS WHAT AN SCP IS!

ChaelCodes commented 3 years ago

Look into T3 and Bert from Google?

Bert is Bi-directional and sucks at text generation

ChaelCodes commented 3 years ago

Research Topic: Recipes made using ML that create a dataset. Most of these use RNN and LSTM, which can't compete with a Transformer that understands attention.

Retrain GPT-2 https://medium.com/ai-innovation/beginners-guide-to-retrain-gpt-2-117m-to-generate-custom-text-content-8bb5363d8b7f (too Python Heavy)

Recipes with GPT-3 and Ruby No training

ChaelCodes commented 3 years ago

Classification with Keter, Safe, Cognitohazard, Humanoid?

ChaelCodes commented 3 years ago

GPT-2 Repo with Python Hugging Face model card GPT-2 document editing example

ChaelCodes commented 3 years ago

Fine-tuning a model is what it's called. Transformers has a Python tutorial

ChaelCodes commented 3 years ago

Friday's Task: Can I move gpt2-slackbot to Ruby using the tensorflow gem? BLOG. Better idea, use rubypython to call gpt methods, and only rewrite train/etc.

clone gpt2-slackbot, and setup/test it.
create an scp dataset folder
train and generate_sample.py
replace generate_sample.py with a ruby script.

ChaelCodes commented 3 years ago

Attempt 1 failed for the following reasons:

GPT-2's libraries use Tensorflow 1.14. Ruby's libraries use 2.5. Conversion
Ruby's libraries are incomplete. 2.5 has an entire compat module allowing you to still run the v1 way. NONE OF THEM HAVE IT. Ankane's looks like a vanity project. Somatico hasn't bothered to gemify or touch it in a year, and Tensor_stream doesn't install.
Tensorflow seems to think I'll be putting @tf.function everywhere and that's not a thing in Ruby.
The biggest issue though is that I neither know ML nor Tensorflow, so I often didn't know the purpose of the code I was converting.

ChaelCodes commented 3 years ago

Attempt 2 will involve learning from the ground up and potentially taking over a Tensorflow library. Tensorflow learning guide

Rumale could be interesting. Not really used by many, but actively maintained.