search

chervonij / DFL-Colab

DeepFaceLab fork which provides IPython Notebook to use DFL with Google Colab
1.05k stars 265 forks source link

Use numpy array #32

Closed Tapiocapioca closed 3 years ago

Tapiocapioca commented 3 years ago

Hello chervonij,

I want ask you if is possible load data at your DFL-Colab Notebook as numpy array (.npy format) and use flow method instead of flow_from_directory. For paid users Google Colab provides 25GB RAM, so even for big data-sets is possible load the entire data into memory. I saw in other groups, not deepfakes or similar, the speed up using numpy array is aroud 2x faster, with the same data generation steps. If one iteration before was like 1 second become like 0,5 second. Muche more faster than data stored in colab local disk like it is now.

I am not expert of Colab or Pyton, but I thought make sense ask you about.

chervonij commented 3 years ago

@Tapiocapioca What do you mean? DFL already use numpy. Also, keep faces on storage instead of RAM, is much safer if you work with remote server, like Colab. And after all, you can use packed format in DFL to increase speed of loading your dataset to RAM, but it's not speed up iteration time.

Tapiocapioca commented 3 years ago

So I didn't understand nothing, I told you I am not expert, I tried to check the code but.... He he. Nothing so my request is useless.

chervonij commented 3 years ago

ok then.