experiment: testing pipelines

[RuiFilipeCampos]

e4bd5de

[RuiFilipeCampos]

There's definitely some memory leakage going on

[RuiFilipeCampos]

cleanup tasks claim to be successful

[RuiFilipeCampos]

• blue = 200
• x axis = step

[RuiFilipeCampos]

model size is not helping at all

[RuiFilipeCampos]

https://stackoverflow.com/questions/17541452/celery-does-not-release-memory

[RuiFilipeCampos]

They keep converging to the same loss value, regardless of size.

I'm going to increase the warmup period to 15k

[RuiFilipeCampos]

I'm gonna need to consider other sources of memory leakage

[RuiFilipeCampos]

if the results backend is being flushed (twice), then how come flower is able to print them out

[RuiFilipeCampos]

fixed seed has been confirmed

[RuiFilipeCampos]

memory leak appears to be solved

[RuiFilipeCampos]

batch size appears to be the critical bit here, blue is 1024 and the slope has converged to a higher value

[RuiFilipeCampos]

the batch size has definitely improved the case

the new run is using a mini batch size of 4x1024, should take quite a while to get the results out though

[RuiFilipeCampos]

gpu usage peaking at 70%, no leakage over the 6h

[RuiFilipeCampos]

cpu mem percentage, no leakage here too

[RuiFilipeCampos]

something fishy going on with disk memory tho

i reckon it must be the .npy files somehow

[RuiFilipeCampos]

I'm gonna choose (256dim,16heads,6blocks,50ctx) as my base case. It fits in the 16gb with room for a large batch size.

The convergence is slow but it is there and it is stable.

There's several paths for optimization:

• rework the data (priority)
• hyper parameter tuning - probably good to check the lamma paper for reference
• scale the hardware - federated learning, TPU instances, etc

[RuiFilipeCampos]

https://arxiv.org/pdf/2302.13971.pdf

• https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html
  • "β1 = 0.9, β2 = 0.95"
• cosine learning rate schedule, final learning rate is equal to 10% of the maxi- mal learning rate
• weight decay of 0.1 and gradient clipping of 1.0
• 2000 warmup
• vary the learning rate and batch size with the size of the model

[RuiFilipeCampos]

will get the pipeline working and merge this right afterwards, pr is way too big rn