harujoh / KelpNet

Pure C# machine learning framework
Apache License 2.0
242 stars 28 forks source link

The training is slow and can't put the GPU under heavy load. #46

Closed xiejiang2014 closed 2 years ago

xiejiang2014 commented 2 years ago

The training is slow and can't put the GPU under heavy load. cpu load 18% I7 9700K gpu load 3% RTX 2070

release mode.

How should I improve the performance? Thanks.

    [STAThread]
    static void Main()
    {
          OpenCL.Initialize(ComputeDeviceTypes.Gpu, 0);

          //Test6を連結してFashion-MNISTを学習
          Sample14.Run();
    }
harujoh commented 2 years ago

I ran the code you provided in my environment and I think the load you provided is probably normal.

CPU load 14% I9 11900K GPU load 7% GTX 1060

If you wish to further accelerate the training, you can increase the number of batches to appropriately increase the load and increase the accuracy and training speed.

However, please keep in mind that if the GPU memory is insufficient, your software may stall with exceptions or risk force termination with unexplained errors.

The number of batches can be changed with the BATCH_DATA_COUNT constant value.

class Sample14
{
    const int BATCH_DATA_COUNT = 20;

It is very difficult to advise exactly how many batches is a good number.

Some people say "the more the merrier," while others say "the more the merrier, the worse the generalization performance."

So, depending on the task you are trying to apply and the machine you are using, try to fine-tune this value.

xiejiang2014 commented 2 years ago

Thanks, I will try more.