Cl Experiments On OpeNTF

[rezaBarzgar]

This issue is for logging the results of the OpeNTF with 2 different loss-based curriculum learning methods (Data Parameters, SuperLoss) on 4 datasets (IMDb, GitHub, DBLP, USPT) by 2 models that have state-of-the-art results (fnn_emb, bnn_emb).

In These experiments, hyperparameters should remain the same.

• Learning rate: 0.001
• epoch: 10
• batch size: 128
• Early Stopping: True with patience 5
• weight decay in Cl methods: 0.9

[rezaBarzgar]

@hosseinfani Dr. Fani,

If you approve, I plan to assign this task to Marco. I will ask him to run it on Compute Canada and share the results with us. I believe it will be a good learning experience for Marco to become familiar with the process of running large-scale projects on cloud resources. I appreciate your thoughts on this.

Thank you

[hosseinfani]

@rezaBarzgar perfect. thank you.

[rezaBarzgar]

@MarcoKurepa There is comprehensive documentation available on using Compute Canada in the MS Teams -> SCS - HF Research Group / General / files / Library / ComputeCanada Guide.docx folder. Once you create an account, either Dr. Fani or I will send you an invitation link. Also, you can update the file with your experience if you feel it needs to be mentioned.

• For these experiments, please make sure to use the latest version of OpeNTF. I merged my fork's changes with the main branch.
• Desired Results:
  • [ ] Dataset: IMBd (IMPORTANT)
    • [ ] model: fnn_emb bnn_emb
    • [ ] param.py: loss = 'normal'
    • [ ] param.py: loss = 'SL'
    • [ ] param.py: loss = 'DP'
  • [ ] DBLP (IMPORTANT)
    • [ ] model: fnn_emb bnn_emb
    • [ ] param.py: loss = 'normal'
    • [ ] param.py: loss = 'SL'
    • [ ] param.py: loss = 'DP'
  • [ ] GitHub
    • [ ] model: fnn_emb bnn_emb
    • [ ] param.py: loss = 'normal'
    • [ ] param.py: loss = 'SL'
    • [ ] param.py: loss = 'DP'
  • [ ] USPT
    • [ ] model: fnn_emb bnn_emb
    • [ ] param.py: loss = 'normal'
    • [ ] param.py: loss = 'SL'
    • [ ] param.py: loss = 'DP'

These results are for a research paper that we aim to submit at ECIR 2024

[MarcoKurepa]

On it!

[MarcoKurepa]

@hosseinfani Please confirm me on Compute Canada.

[hosseinfani]

@MarcoKurepa already done!

[MarcoKurepa]

Response from Compute Canada support is already in. Gmail - Error 4 on Cluster Beluga.pdf

[hosseinfani]

@MarcoKurepa I asked Mohammad to tar his file. Hopefully, we can solve the issue soon.

[MarcoKurepa]

https://sharcnet-ca.zoom.us/meeting/register/tZ0rduuspzktH92Cq3-9AWe-HaUBmPyCBe78

[MarcoKurepa]

I changed computecanada.sh by removing #SBATCH --gpus-per-node=1 because it had a redundancy conflict causing an error with #SBATCH --gres=gpu:v100:1.

[MarcoKurepa]

The job is now pending I will update you when it has been completed.

[hosseinfani]

@MarcoKurepa I think you need to change the email notification to your own. I've been receiving some notification from computecanada :D

[MarcoKurepa]

@MarcoKurepa

I think you need to change the email notification to your own. I've been receiving some notification from computecanada :D

Apologies for that! I'll look into it🙂

[MarcoKurepa]

Sorry if you got any more notifs today, I just realized they were coming from the slurm script! You shouldn't receive any more emails now:)

[hosseinfani]

@MarcoKurepa no worries. correct, I don't receive them anymore. tnx.

[rezaBarzgar]

@MarcoKurepa Hey Marco, how are the experiments going on? Also, Have you started the write-up? I just wanted to remind you that these tasks are the priority gently. Please update me regularly even if you have not made any progress or you are busy with some other things.

[MarcoKurepa]

@MarcoKurepa Hey Marco, how are the experiments going on? Also, Have you started the write-up? I just wanted to remind you that these tasks are the priority gently. Please update me regularly even if you have not made any progress or you are busy with some other things.

Hey Reza, sorry for the lack of updates. I have mostly been working on the computecanada cluster, although to be frank I've been having a lot of hiccups with it especially in getting the correct versions of the libraries. At the moment I am still trying to squash errors relating to runnign the script but I have an .sh ready to run once I've gotten to the point where I can run a sample dataset in the terminal.

I'll make sure to keep you posted moving forward.

[rezaBarzgar]

Results' sheets: https://docs.google.com/spreadsheets/d/1oatYd2uMXOah8RQJsvaT2NIwxhh6CfssaneVLuYczQY/edit#gid=1683104529

[MarcoKurepa]

While I was working on the CL Visualization issue, I made an error which resulted in VSCode crashing and I lost the progress made on training CL on the Github dataset. However, in the 48~ hours that it did run ti only finished 2 epoches (still the first fold), so I think it'd be better to run it off Reza's computer regardless. Also I would've needed to restart the run either way to include the expert loss logging script, so no real progress was lost, just a mild annoyance in the end.

@rezaBarzgar