Log tflops and other fixes

[RaymondLi0]

This should be merged after #32

• change FIM tokens
• log TFLOPS, fixes #31 (code from https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/e52bdabbde3c6895aceb76c1bced295c2646121f/megatron/training.py#L749)

[RaymondLi0]

Thank you for the suggestion @NouamaneTazi ! In https://github.com/bigcode-project/Megatron-LM/pull/33/commits/b18ecf6b332c67e88a20b017a0714172702229b5 I adjusted the formula that's in the comments, could you confirm that this is correct?

[NouamaneTazi]

LGTM! You might wanna check the # TODO: maybe tp_size factor missing here depending on how you implemented MQA

[RaymondLi0]

I left the todo in the comments. Let's merge this and address this later.