BlackSamorez / tensor_parallel

Automatically split your PyTorch models on multiple GPUs for training & inference
MIT License
629 stars 39 forks source link

Unpersistent buffers meta loading fix #57

Closed BlackSamorez closed 1 year ago

BlackSamorez commented 1 year ago

Right now if a model contains any meta tensors it will fully be placed on meta device.

Normally all the data will be loaded when dispatching the model but unpersistent buffer will not be loaded and will remain on meta.

This PR fixes this behavior.

BlackSamorez commented 1 year ago