Loss is not decreasing

[BenAhn]

Dear. Mr. Nateraw

   I am from South Korea, graduate student, who is interested in NLP. 

First of all thank you for your tensorflow LDA2vec code.

I have tested your code with suggested data, and followed your instruction of installation

• data -. glove.6B.zip -. 20_newsgroups.txt
• parameters -. no. of topics = 20 -. no. of epochs = 200 -. batch_size = 500 -. switch_loss_epoch = 0

After that, i got the result as below. It started loss with 3381 and finished with 3381.

Question

• does your trial also have same loss?
• if there are something that i had wrong, please suggest.

Thank you in advance.

PS - The code is not working with Python 3.7.1, tensorflow-gpu 1.13.1 but python 3.6.8 with tensorflow-gpu 1.12.0 works fine

[nateraw]

Hey! Are you using the default processing in load_20newsgroups.py? I have been messing with the code a lot recently...so some changes may cause end results to fluctuate.

For example, you shouldn't have tokens like "/./lib" showing up. If you aren't using my parameters/preprocessing, can you tell me what you are using instead?

As for the loss - it does that. It's not meant to decrease after a certain point.

As for the version issues - As noted in the main description of this repo, this code is written for Tensorflow 1.5. I should have also specified I used Python 3.5 too. Either way...thank you for letting me know that this works on 3.6.8 w/ tf 1.12.0. Very helpful!!

[nateraw]

Additional note on the loss...it's going to stay the same after only a few epochs, but the topics will continue to learn. So, the loss staying the same does not mean that it isn't continuing to learn. It is just constantly taxing the model at the same level.

[BenAhn]

Thank you for prompt reply!

1. can you tell me what you are using instead?

   • I haven't changed anything from your code.
   • I loaded the data through 'load_20newsgroups.py'
   • However, i have used 'run_20newsgroups.py' at tests folder, not you have on the read me page Do you recommend what you have on the read me page?

2. the loss staying the same does not mean that it isn't continuing to learn

   • I am new to deep learning world ^^. i can see that text is more readable as more epoch.
   • In vision deep learning loss no. is one of evaluation index, i think. (and it feels good to see it goes down ^^)
   • so in this case, the loss is not important?

again thank you~

[dbl001]

How does the algorithm learn:

1. P(T,W) - probability of a topic given word, starting from the prior dirichlet distribution, without a decreasing loss function?

[dbl001]

Sana and I had the same issue with strange and sometimes unprintable characters (unicode?)

[nateraw]

Closing, however I might make a new issue related to the preprocessing issue. The "loss decreasing" part of this issue is not an issue. +

[nateraw]

How does the algorithm learn:

1. P(T,W) - probability of a topic given word, starting from the prior dirichlet distribution, without a decreasing loss function?

@dbl001 it learns by making the Dirichlet distribution (the initializer of the topic matrix) more and more sparse over time. Sort of related...here is a question on stackoverflow showing how dirichlet distributions can be manipulated to become more and more sparse.

[BenAhn]

Thank you for reviewing.

I will wait for your preprocessing updates. and if i have a progress i will share.

From who really want to see LDA2vec is working. ^^