search

logix-project / logix

AI Logging for Interpretability and Explainability🔬
Apache License 2.0
74 stars 6 forks source link

Flattening the tensors before unflattening #105

Closed eatpk closed 3 months ago

eatpk commented 3 months ago

Summary

Flattening the tensor before unflattening to minimize the data cloning from numpy mmap to torch.Tensor.

Related Issues

N/A

Test Plan

performance tested, improvement by 50%.

sangkeun00 commented 3 months ago

Thanks! I will check the performance improvement on my GPT benchmark. If I notice more than 10-20% improvements, I will merge this PR.

sangkeun00 commented 3 months ago

This PR gave me 5-15% improvement in my experiments. Still an improvement! You can merge this PR.