Movielens experiments

[miguelgfierro]

Hey @okuchaiev I've been doing some experiments with the movielens dataset 20M, everything is in 1 GPU P100, wd=0, selu, sdg with momentum and the other default options, here the results:

300 epochs:
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep300, Process time 34708s, RMSE: 0.8270190993497112
* h256.256.256.512_lr0.002_dp0.8_bs64_aug1_ep300, Process time 35469s, RMSE: 0.8298143399008006
* h128.128.256_lr0.005_dp0.8_bs64_aug1_ep300, Process time 30648s, RMSE: 0.832070992094558
* h256.256.512_lr0.005_dp0.8_bs128_aug1_ep300, Process time 29872s, RMSE: 0.8335073607999096

50 epochs:
* h128.128.256_lr0.005_dp0.8_bs64_aug1_ep50, Process time 7316s, RMSE: 0.8373085857776336
* h128.128.256_lr0.002_dp0.8_bs64_aug1_ep50, Process time 7255s, RMSE: 0.8505965955389251
* h128.128.256_lr0.005_dp0.8_bs128_aug1_ep50, Process time 6559s, RMSE: 0.8463281991349552
* h128.128.256_lr0.002_dp0.8_bs128_aug1_ep50, Process time 6550s, RMSE: 0.8666786635557636
* h128.128.128.256_lr0.005_dp0.8_bs64_aug1_ep50, Process time 5893s, RMSE: 0.8356554165814366
* h128.128.128.256_lr0.005_dp0.8_bs128_aug1_ep50, Process time 5892s, RMSE: 0.8426986399494479 
* h128.128.128.256_lr0.002_dp0.8_bs64_aug1_ep50, Process time 6084s, RMSE: 0.8461920729794343
* h128.128.128.256_lr0.002_dp0.8_bs128_aug1_ep50, Process time 5903s, RMSE: 0.8607027585932623

* h256.512_lr0.005_dp0.8_bs64_aug1_ep50, Process time 6166s, RMSE: 0.8420912025890906
* h256.512_lr0.002_dp0.8_bs64_aug1_ep50, Process time 6263s, RMSE: 0.8615509368380064
* h256.512_lr0.005_dp0.8_bs128_aug1_ep50, Process time 6450s, RMSE: 0.8581210973782695
* h256.512_lr0.002_dp0.8_bs128_aug1_ep50, Process time 6526s, RMSE: 0.8901831164436558
* h256.256.512_lr0.005_dp0.8_bs256_aug1_ep50, Process time 7761s, RMSE: 0.8505812808299635
* h256.256.512_lr0.001_dp0.8_bs256_aug1_ep50, Process time 8048s, RMSE: 0.9072564425137314
* h256.256.512_lr0.005_dp0.8_bs128_aug1_ep50, Process time 7313s, RMSE: 0.8383932386367762
* h256.256.512_lr0.001_dp0.8_bs128_aug1_ep50, Process time 7485s, RMSE: 0.8805132787428274
* h256.256.256.128_lr0.005_dp0.8_bs64_aug1_ep50, Process time 6832s, RMSE: 0.8826071700724554
* h256.256.256.128_lr0.002_dp0.8_bs64_aug1_ep50, Process time 6924s, RMSE: 0.8518409650015333
* h256.256.256.128_lr0.005_dp0.8_bs128_aug1_ep50, Process time 6670s, RMSE: 0.8516106274521363
* h256.256.256.128_lr0.002_dp0.8_bs128_aug1_ep50, Process time 6640s, RMSE: 0.876183417003062
* h256.256.256.256_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001, Process time 5681s, RMSE: 0.8247678613140474
* h256.256.256.256_lr0.005_dp0.8_bs32_aug0_ep50_wd0.00001_elu, Process time 5366s, RMSE: 0.82389932562926
* h256.256.256.256_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001_elu, Process time 5017s, RMSE: 0.8282986409964643
* h256.256.256.256_lr0.005_dp0.8_bs64_aug1_ep50_wd0.00001_elu, Process time 5136s, RMSE: 0.8324252316444559
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50, Process time 8286s, RMSE: 0.8292924646574259
* h256.256.256.512_lr0.002_dp0.8_bs64_aug1_ep50, Process time 8493s, RMSE: 0.8357998662652054
* h256.256.256.512_lr0.005_dp0.8_bs128_aug1_ep50, Process time 8877s, RMSE: 0.8342839521305517
* h256.256.256.512_lr0.002_dp0.8_bs128_aug1_ep50, Process time 9235s, RMSE: 0.8479914043369364
* h256.256.256.512_lr0.005_dp0.8_bs64_aug2_ep50, Process time 6194s, RMSE: 0.8314955237915321
* h256.256.256.512_lr0.005_dp0.6_bs64_aug1_ep50, Process time 5849s, RMSE: 0.8381541416170412
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_relu, Process time 5202s, RMSE: 2.165232282658598
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_wd0.001, Process time 6358s, RMSE: 0.8615138222322402
* h256.256.256.512_lr0.005_dp0.8_bs32_aug1_ep50, Process time 6601s, RMSE: 0.8286968416622136 
* h256.256.256.512_lr0.005_dp0.8_bs64_aug3_ep50, Process time 6700s, RMSE: 0.8325802227208278 
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_elu, Process time 5196s, RMSE: 0.829355669108844
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_wd0.00001, Process time 6092s, RMSE: 0.8286978067933191
* h256.256.256.512_lr0.005_dp0.8_bs32_aug1_ep50_wd0.00001_lrelu, Process time 5745s, RMSE: nan
* h256.256.256.512_lr0.005_dp0.9_bs32_aug1_ep50_wd0.00001, Process time 6752s, RMSE: 0.8358882991657209
* h256.256.256.512_lr0.005_dp0.8_bs32_aug1_ep50_wd0.00001_swish, Process time 6438s, RMSE: nan
* h256.256.256.512_lr0.005_dp0.8_bs32_aug1_ep50_wd0.00001_relu6, Process time 5806s, RMSE: 1.5870073433725758
* h256.256.256.512_lr0.005_dp0.8_bs32_aug1_ep50_wd0.00001, Process time 8147s, RMSE: 0.8278444900229919
* h256.256.256.512_lr0.005_dp0.5_bs64_aug1_ep50_wd0.00001, Process time 7146s, RMSE: 0.8458491087391462
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_wd0.00001_sigmoid, Process time 5855s, RMSE: 2.6946873485097624
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_wd0.00001_tanh, Process time 5879s, RMSE: 2.6892056428673965
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_wd0.00001_elu, Process time 6614s, RMSE: 0.829774144023921
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_wd0.00001_none, Process time 6069s, RMSE: nan
* h256.256.256.512_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001_elu, Process time 7231s, RMSE: 0.8246256263673024
* h256.256.256.512_lr0.005_dp0.8_bs64_aug1_ep50_wd0.00001_elu_cons, Process time 7623s, RMSE: nan
* h256.256.256.512_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001_elu_skp, Process time 5056s, RMSE: 0.8256991938919997
* h256.256.256.512_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001_elu_np0.2, Process time 5475s, RMSE: 0.8249846126834408
* h256.256.256.512_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001_elu_np0.4, Process time 5404s, RMSE: 0.8249381947311919
* h256.256.256.512_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001_elu_np0.6, Process time 5336s, RMSE: 0.8244053170054507
* h256.256.256.256.512_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001, Process time 8364s, RMSE: 0.8252511286962865
* h256.256.256.256.512_lr0.005_dp0.8_bs32_aug0_ep50_wd0.00001_elu, Process time 7876s, RMSE: 0.8208981665592793
* h256.256.256.256.512_lr0.005_dp0.8_bs64_aug0_ep50_wd0.00001_elu, Process time 6841s, RMSE: 0.8232621691107652
* h256.256.256.256.512_lr0.004_dp0.8_bs64_aug0_ep50_wd0.00001_elu, Process time 7459s, RMSE: 0.8250180634812827

* h512.12.12_lr0.005_dp0.8_bs256_aug1_ep50, Process time 8197s, RMSE: nan
* h512.128.12_lr0.005_dp0.8_bs256_aug1_ep50, Process time 5436s, RMSE: 1.0141237434398267
* h512.128.12_lr0.01_dp0.8_bs256_aug1_ep50, Process time 7107s, RMSE: 1.669601678946766
* h512.128.12_lr0.005_dp0.8_bs128_aug1_ep50, Process time 7304s, RMSE: 1.9644852247335254
* h512.128.12_lr0.01_dp0.8_bs128_aug1_ep50, Process time 8256s, RMSE: 2.2515750498298295
* h512.256.128_lr0.005_dp0.8_bs64_aug1_ep50, Process time 8330s, RMSE: nan
* h512.256.128_lr0.002_dp0.8_bs64_aug1_ep50, Process time 9167s, RMSE: 0.8532720004964209
* h512.256.128_lr0.005_dp0.8_bs128_aug1_ep50, Process time 7964s, RMSE: 0.8514060297120034
* h512.256.128_lr0.002_dp0.8_bs128_aug1_ep50, Process time 7808s, RMSE: 0.8771772641279514
* h512.512.512.1024_lr0.005_dp0.8_bs256_aug1_ep50, Process time 7631s, RMSE: 0.8408605201075207
* h512.512.512.1024_lr0.001_dp0.8_bs256_aug1_ep50, Process time 9081s, RMSE: 0.8851269050050165
* h512.512.512.1024_lr0.005_dp0.8_bs128_aug1_ep50, Process time 8221s, RMSE: 0.8390253067561858
* h512.512.512.1024_lr0.001_dp0.8_bs128_aug1_ep50, Process time 9201s, RMSE: 0.85770380406226
* h512.512.512_lr0.005_dp0.8_bs256_aug1_ep50, Process time 6015s, RMSE: 0.8490931741924439
* h512.512.512_lr0.001_dp0.8_bs256_aug1_ep50, Process time 6013s, RMSE: 0.8994081473635861
* h512.512.512_lr0.005_dp0.8_bs128_aug1_ep50, Process time 6276s, RMSE: 0.8435522435037354
* h512.512.512_lr0.001_dp0.8_bs128_aug1_ep50, Process time 6347s, RMSE: 0.8727244815655271
* h512.512.1024_lr0.005_dp0.8_bs256_aug1_ep50, Process time 7118s, RMSE: 0.8515680809349422
* h512.512.1024_lr0.001_dp0.8_bs256_aug1_ep50, Process time 8119s, RMSE: 0.9003549734935724
* h512.512.1024_lr0.005_dp0.8_bs128_aug1_ep50, Process time 8581s, RMSE: 0.8412896519786348
* h512.512.1024_lr0.001_dp0.8_bs128_aug1_ep50, Process time 8284s, RMSE: 0.8725194757170811

* h1024.512.512_lr0.005_dp0.8_bs256_aug1_ep50, Process time 8366s, RMSE: 0.848893039387492
* h1024.512.512_lr0.001_dp0.8_bs256_aug1_ep50, Process time 8487s, RMSE: 0.9071224941082034
* h1024.512.512_lr0.005_dp0.8_bs128_aug1_ep50, Process time 8055s, RMSE: 0.8402467531636137
* h1024.512.512_lr0.001_dp0.8_bs128_aug1_ep50, Process time 7878s, RMSE: 0.8799644549035793
* h2048.1024.512_lr0.005_dp0.8_bs256_aug1_ep50, Process time 9201s, RMSE: 0.8531879835494411
* h2048.1024.512_lr0.001_dp0.8_bs256_aug1_ep50, Process time 9343s, RMSE: 0.9083370416845887
* h2048.1024.512_lr0.005_dp0.8_bs128_aug1_ep50, Process time 9071s, RMSE: 0.8457299178975725
* h2048.1024.512_lr0.001_dp0.8_bs128_aug1_ep50, Process time 9159s, RMSE: 0.8802430695913057

Some observations:

• It looks that LR=0.005 is the winner, with 0.002 sometimes giving good results
• Smaller BS yields better accuracy (which makes sense), but there is no a massive difference in training time
• The architectures that seems to work better are inverted pyramids (256,256, 512) instead of traditional autoencoder shape that are pyramids (512, 256, 128). Could this behavior be because the input is super sparse, so it's better a small input size? However, it is odd that adding bigger internal layers seems to work better.

Did you have time to do some experiments on your side?

[okuchaiev]

Thanks @miguelgfierro for sharing your experiments! I think the main reason “inverse” pyramids work better is because then first layer is not as heavy as in the “classical” and therefore overfitting is less likely which is important for smaller datasets like this. BTW, is this movielense 1M or 20M? I’ll set up some experiments to run over the weekend to see if we can improve this RMSE. What was the best you’ve got using ALS on the same dataset?

[miguelgfierro]

Hey, I updated the initial comment with more results.

I'm using movielense 20M.

The best result I got for now is: h256.256.256.256.512_lr0.005_dp0.8_bs32_aug0_ep50_wd0.00001_elu, Process time 7876s, RMSE: 0.8208981665592793

I did experiments with Movielens 20M in databricks with sparkml ALS and also using the implementation with GPU of spotlight. Using rank 12 I got 0.80.

The only problem is that I haven't used the exact same partition of the dataset in the 3 experiments, however, using two different partitions with ALS in databricks and spotlight I got similar results. One of my next steps is to create a common partition repeat the experiments on the 3 frameworks.

[vaidoorya]

Hi,

Can you please help me understand how can i set up the environment on GCP? I am somehow not able to set it up properly.

2. can you also share a sample of final .txt file , folder structure , data formats used etc. I am running into multiple errors such as

--

"invalid literal for int() with base10: '1,2003,Dianosaur plane"

Please help!!

[okuchaiev]

@vaidoorya apparently your input is in the wrong format. Follow the Readme with Netflix data and then make sure your data follows the same format

[vaidoorya]

I was’nt able to prepare the data using the code, some error kept on coming – can you atleast let me know the format used to run the codes?

From: Oleksii Kuchaiev [mailto:notifications@github.com] Sent: Saturday, March 3, 2018 9:47 AM To: NVIDIA/DeepRecommender DeepRecommender@noreply.github.com Cc: Vaidoorya Agarwal (Big Data Analytics) vaidoorya.agarwal@wipro.com; Mention mention@noreply.github.com Subject: Re: [NVIDIA/DeepRecommender] Movielens experiments (#11)

This mail has been sent from an external source. Treat hyperlinks and attachments in this email with caution

@vaidooryahttps://github.com/vaidoorya apparently your input is in the wrong format. Follow the Readme with Netflix data and then make sure your data follows the same format

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/NVIDIA/DeepRecommender/issues/11#issuecomment-370116269, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ANG4r4U1caGtmQbhyWq9uvI3bXNXexDrks5tahlVgaJpZM4SB7TH.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com