search

dongjun-Lee / text-summarization-tensorflow

Tensorflow seq2seq Implementation of Text Summarization.
MIT License
636 stars 196 forks source link

Loading prebuilt pickle file #9

Closed wassimseif closed 6 years ago

wassimseif commented 6 years ago

def build_dict takes around 30 mins to finish. This patch will just check if the pickle file is already there so it doesn't build it again

dongjun-Lee commented 6 years ago

Hello. Thank you for your PR. However, this commit will generate an error when there is no pre-trained word_dict.pickle. I think this way would be better.

wassimseif commented 6 years ago

Hello I added the check is the file exists. Also, I noticed that the build_dataset function takes time. any ideas how can we do the same for it?

dongjun-Lee commented 6 years ago

Saving train_x, train_y, valid_x, valid_y with pickle might help.

wassimseif commented 6 years ago

Will do. btw I'm curious on which GPU did you train this. on a 1080 it's struggling

dongjun-Lee commented 6 years ago

I used two TITAN Xp and it takes a long time for me too.

wassimseif commented 6 years ago

I added the functions to save X & Y to load them later

wassimseif commented 6 years ago

tick tock

dongjun-Lee commented 6 years ago

Hi, sorry for the late reply. I'm sorry but I would skip this pull request, because this is not necessary part and I want to keep the code as simple as possible. I really appreciate your contribution.