Can the model really learn??

[lzamparo]

I'm seeing a disturbing pattern of not seeing better performance by epoch.

Different macrobatches are showing better or worse performance, but they are very consistent with which files make up the macro-batches

Below is a trace of the output from a sample embedding, trained on the positive probes that pass Han's QC stages. It's clear that across epochs, the macrobatches which cover the same files in the data yield an average loss which does not decrease in an epoch dependent manner:

zamparol$ less fasta_seqs.txt | grep -e average -e macrobatch -e epoch

starting epoch 0 receiving macrobatch from child process running macrobatch 0 average loss: 11.090376 receiving macrobatch from child process running macrobatch 1 average loss: 11.090356 receiving macrobatch from child process running macrobatch 2 average loss: 11.090346 receiving macrobatch from child process running macrobatch 3 average loss: 11.090362 receiving macrobatch from child process running macrobatch 4 average loss: 11.090349 receiving macrobatch from child process running macrobatch 5 average loss: 11.090341 receiving macrobatch from child process running macrobatch 6 average loss: 5.876007 receiving macrobatch from child process running macrobatch 7 average loss: 5.559565 receiving macrobatch from child process running macrobatch 8 average loss: 5.803356

starting epoch 1 receiving macrobatch from child process running macrobatch 0 average loss: 11.090372 receiving macrobatch from child process running macrobatch 1 average loss: 11.090356 receiving macrobatch from child process running macrobatch 2 average loss: 11.090366 receiving macrobatch from child process running macrobatch 3 average loss: 11.090344 receiving macrobatch from child process running macrobatch 4 average loss: 11.090327 receiving macrobatch from child process running macrobatch 5 average loss: 11.090322 receiving macrobatch from child process running macrobatch 6 average loss: 5.513479 receiving macrobatch from child process running macrobatch 7 average loss: 5.577124 receiving macrobatch from child process running macrobatch 8 average loss: 5.952247

starting epoch 2 receiving macrobatch from child process running macrobatch 0 average loss: 11.090325 receiving macrobatch from child process running macrobatch 1 average loss: 11.090312 receiving macrobatch from child process running macrobatch 2 average loss: 11.090318 receiving macrobatch from child process running macrobatch 3 average loss: 11.090284 receiving macrobatch from child process running macrobatch 4 average loss: 11.090295 receiving macrobatch from child process running macrobatch 5 average loss: 11.090289 receiving macrobatch from child process running macrobatch 6 average loss: 5.597625 receiving macrobatch from child process running macrobatch 7 average loss: 5.679941 receiving macrobatch from child process running macrobatch 8 average loss: 5.735664

starting epoch 3 receiving macrobatch from child process running macrobatch 0 average loss: 11.090294 receiving macrobatch from child process running macrobatch 1 average loss: 11.090319 receiving macrobatch from child process running macrobatch 2 average loss: 11.090273 receiving macrobatch from child process running macrobatch 3 average loss: 11.090270 receiving macrobatch from child process running macrobatch 4 average loss: 11.090262 receiving macrobatch from child process running macrobatch 5 average loss: 11.090235 receiving macrobatch from child process running macrobatch 6 average loss: 5.614810 receiving macrobatch from child process running macrobatch 7 average loss: 5.614810 receiving macrobatch from child process running macrobatch 8 average loss: 5.829066

starting epoch 4 receiving macrobatch from child process running macrobatch 0 average loss: 11.090243 receiving macrobatch from child process running macrobatch 1 average loss: 11.090234 receiving macrobatch from child process running macrobatch 2 average loss: 11.090187 receiving macrobatch from child process running macrobatch 3 average loss: 11.090198 receiving macrobatch from child process running macrobatch 4 average loss: 11.090191 receiving macrobatch from child process running macrobatch 5 average loss: 11.090154 receiving macrobatch from child process running macrobatch 6 average loss: 5.606859 receiving macrobatch from child process running macrobatch 7 average loss: 5.774870 receiving macrobatch from child process running macrobatch 8 average loss: 5.686827

starting epoch 5 receiving macrobatch from child process running macrobatch 0 average loss: 11.090167 receiving macrobatch from child process running macrobatch 1 average loss: 11.090161 receiving macrobatch from child process running macrobatch 2 average loss: 11.090095 receiving macrobatch from child process running macrobatch 3 average loss: 11.090075 receiving macrobatch from child process running macrobatch 4 average loss: 11.090047 receiving macrobatch from child process running macrobatch 5 average loss: 11.090009 receiving macrobatch from child process running macrobatch 6 average loss: 5.545681 receiving macrobatch from child process running macrobatch 7 average loss: 5.530536 receiving macrobatch from child process running macrobatch 8 average loss: 5.955928

starting epoch 6 receiving macrobatch from child process running macrobatch 0 average loss: 11.090011 receiving macrobatch from child process running macrobatch 1 average loss: 11.089976 receiving macrobatch from child process running macrobatch 2 average loss: 11.089961 receiving macrobatch from child process running macrobatch 3 average loss: 11.089908 receiving macrobatch from child process running macrobatch 4 average loss: 11.089851 receiving macrobatch from child process running macrobatch 5 average loss: 11.089799 receiving macrobatch from child process running macrobatch 6 average loss: 5.628832 receiving macrobatch from child process running macrobatch 7 average loss: 5.542895 receiving macrobatch from child process running macrobatch 8 average loss: 5.874512

starting epoch 7 receiving macrobatch from child process running macrobatch 0 average loss: 11.089784 receiving macrobatch from child process running macrobatch 1 average loss: 11.089684 receiving macrobatch from child process running macrobatch 2 average loss: 11.089632 receiving macrobatch from child process running macrobatch 3 average loss: 11.089536 receiving macrobatch from child process running macrobatch 4 average loss: 11.089473 receiving macrobatch from child process running macrobatch 5 average loss: 11.089358 receiving macrobatch from child process running macrobatch 6 average loss: 5.663543 receiving macrobatch from child process running macrobatch 7 average loss: 5.709703 receiving macrobatch from child process running macrobatch 8 average loss: 5.704039

starting epoch 8 receiving macrobatch from child process running macrobatch 0 average loss: 11.089377 receiving macrobatch from child process running macrobatch 1 average loss: 11.089262 receiving macrobatch from child process running macrobatch 2 average loss: 11.089121 receiving macrobatch from child process running macrobatch 3 average loss: 11.088958 receiving macrobatch from child process running macrobatch 4 average loss: 11.088927 receiving macrobatch from child process running macrobatch 5 average loss: 11.088632 receiving macrobatch from child process running macrobatch 6 average loss: 5.636472 receiving macrobatch from child process running macrobatch 7 average loss: 5.469601 receiving macrobatch from child process running macrobatch 8 average loss: 5.911286

starting epoch 9 receiving macrobatch from child process running macrobatch 0 average loss: 11.088615 receiving macrobatch from child process running macrobatch 1 average loss: 11.088458 receiving macrobatch from child process running macrobatch 2 average loss: 11.088245 receiving macrobatch from child process running macrobatch 3 average loss: 11.088001 receiving macrobatch from child process running macrobatch 4 average loss: 11.087771 receiving macrobatch from child process running macrobatch 5 average loss: 11.087394 receiving macrobatch from child process running macrobatch 6 average loss: 5.789453 receiving macrobatch from child process running macrobatch 7 average loss: 5.532505 receiving macrobatch from child process running macrobatch 8 average loss: 5.703700

This could be due to lots of things, but mostly it's probably a failure of the DatasetReader to parse the data in a format which is true to what the model (either CBOW or skip-gram) actually expects. This is a problem. I'll try testing with a gensim implementation.

[lzamparo]

Also, I'll update DatasetReader to vary the order in which files are processed

[lzamparo]

After varying both values of K and the stride, it seems the inability to learn is a function both of K being large, and stride being too large. Will produce a fig with the results of my experiments.

[lzamparo]

Also, a replication in the TF implementation of w2v would help convince me that the present implementation is working.

[lzamparo]

Looks like we can learn if informative kmers are close enough to each other; so keep stride low, and maybe keep K low so as not to drop too many useful tokens.