While there is already some Humor Generation work out there using given word context, in this project we explore Humor Generation given images. A joint pipeline of image captioning, question generation from the caption and finally a joke in the form of answer to the question is the brief architecture. We were able to implement it successfully and get comparative result to the works done before.
.
├── data #pickle files for templates and object generations
│ ├── aux_verbs.pkl
│ ├── dobj_dict_classes.p
│ ├── one_slot.pkl
│ ├── pobj_dict_classes.p
│ ├── three_slots.pkl
│ ├── two_slots.pkl
│ ├── verb_transitivity.tsv
│ └── verb_trans.p
├── documents #general documents
│ └── code_slingers_presentation.pdf
├── images #images
│ └── architecture.png
├── initialization_script.sh #for initialization
├── initialization_script_windows.sh
├── README.md
├── requirements.txt
└── src #main python files
├── grammar_parsing.py
├── joke_generator.py
├── main.py
├── parsing.py
├── question_generation.py
├── static #static file for the app
│ ├── css
│ │ └── main.css
│ └── img
│ └── saarland.png
├── templates # html
│ └── upload_image.html
└── views.py #main flask app file
This project is an attempt to generate WHY-jokes following the formula Why did X Y? -Z!"
(e.g. Why did the chicken cross the road? To get to the other side!) based on an image. Hence, it encompasses three main tasks:
1. Image captioning
2. Question generation / Getting relevant entities for joke generation
3. Joke generation
The short architecture of our project:
Note: Since, the project has three models as the pipeline running serially, it requires some memory space and RAM. Make sure you have around 3 GB physical disk space, 4 GB RAM and enough space to install the requirements.
In order to get the model to run, follow these installation instructions.
Pre-requisites:
python==3.7.5
On Windows you will also need Git for Windows.
Optional: to install a specific version of Python:
pyenv install 3.7.5
(To install pyenv
, follow this tutorial, and then this one)
brew install python@3.7
Download Python 3.7.5 for Windows here, run it and follow the instructions.
git clone [link]
Optional: use the package manager pip to install a vitual environment.
bash
pip install virtualenv
virtualenv <name of env> --python /usr/bin/python[version] or <path to your python if its not the mentioned one>
Conda:
conda create --name <name of your env> python=3.7
source name_of_env/bin/activate
On Windows:
name_of_env\Scripts\activate
Conda:
conda activate <name of your env>
(To leave the virtual environment, simply run: deactivate
(with virtualenv) or conda deactivate
(with Conda))
pip install -r requirements.txt
Conda:
conda install pip
pip install -r requirements.txt
Optional: If you're on Mac.
Note: if you are on mac, then first install wget using brew
brew install wget
python -m spacy download en_core_web_lg
This is the step that clones the submodules required for captioning, download pretrained models and set them up. This might take a while, depending on your internet speed. :)
chmod +x initialization_script.sh
./initialization_script.sh
On Windows:
If you did not choose to import Linux tools when installing Git for Windows, you have to use the Git Bash to run the initialization script. Either way, both Git Bash and Windows Command Prompt should be run as administrator.
chmod +x initialization_script.sh
sh initialization_script_windows.sh
NOTE: If you encounter the following error (mostly due to internet issue models are not downloaded properly and you can't unzip)
Archive: models.zip
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of models.zip or
models.zip.zip, and cannot find models.zip.ZIP, period.
Then Kindly rerun the following code
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1-bnWPP6-c42wVQMztHpcVquUGd1XGc-z' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1-bnWPP6-c42wVQMztHpcVquUGd1XGc-z" -O models.zip && rm -rf /tmp/cookies.txt
unzip models.zip
If not, then skip this section!
YAY!! Installation is done! Now you can jump to the execution part and run the web app.
!!! Before running the application, make sure to change the ROOT_PATH
variable in the .env
file to the path of your project.
To run the webapp, run the following code, being in the root directory.
python3 src/views.py
Download
./src/coco-download.sh
sh src/coco-download.sh
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.