TRAINING savings on minimal LOSS for better LORA

[Maelstrom2014]

Hi, Genious Kijai! Why not to save lora every new minimal value? Or several last minimals? If its not possible with every step, can minimal loss be taken every epoch? And saved minimal? Sorry if its a stupid question, but I make the same with training on RVC (voices). Thanx for your answer!

[kijai]

Hey, I've actually thought of this before, I just don't know yet what would be a good rule to do that. Technically it's simple to implement saving based on the loss values.

[Maelstrom2014]

It was a simple task for a first sight. I making research hwo to do it better. But for SURE it's better to chage your vay to draw everaged LOSS. This is how Tensorboard make it:

def smooth2(scalars, weight):
    """
    EMA implementation according to
    https://github.com/tensorflow/tensorboard/blob/34877f15153e1a2087316b9952c931807a122aa7/tensorboard/components/vz_line_chart2/line-chart.ts#L699
    """
    last = 0
    smoothed = []
    num_acc = 0
    for next_val in scalars:
        last = last * weight + (1 - weight) * next_val
        num_acc += 1
        # de-bias
        debias_weight = 1
        if weight != 1:
            debias_weight = 1 - math.pow(weight, num_acc)
        smoothed_val = last / debias_weight
        smoothed.append(smoothed_val)

    return smoothed

and this is without weight (previous tends to it with weight ->1) loss_cumsum=np.cumsum(loss)/np.arange(1, len(loss)+1) cumsum - green smooth2 - darkblue (smooth2_loss=smooth2(loss,.999) ) cyan - right way (without shift) YOUR gaph with window_size=200: loss_conv = np.convolve(loss1, np.ones(window_size) / window_size, mode='full') red dots its 5 minimals

[kijai]

Yeah I'd like to improve the loss graphs, problem just had been I have no previous training experience so I don't really know this stuff, multiple graphs in one seems like a good idea indeed.

[SencneS]

I'm not a python coder, but I have an idea in my mind how to make that. The node "Display Anything" from rgthree, outputs numbers from the "Loss_List" in the "Visualize Loss" node you have. I actually use those numbers, display them in Excel and sort to get the less loss save. But it reports it on every single step from every single loop. Which is not a bad thing...

In my mind there are a few ways to tackle this. 1) Every time it saves a loop, it saves the last number in the "Visualize Loss" list (this would be the one it just did) and appends that number to end of the name in that file. -- The reasons would be we can visually see which one of the saves has the lowest number. 2) You could kind of automate it by having it make a save every single step it makes, but only keeps the lowest value it comes across. You could add a "keep the lowest 5" so it will always over-right the highest numbered file in the save folder. But remember the lowest number it comes across - when Visualize Loss spits out a lower number than that it remembers it saves that file over the highest number in the save folder. So in the save folder it looks like this 0.42054999.safetensors 0.42091628.safetensors 0.40922244.safetensors 0.40924419.safetensors 0.40749725.safetensors

Lets say it does a step and produce a 0.49100515 - It doesn't save it. But the next step it produces a 0.4010055, It saves that that one and deleted 0.42091628.safetensors.

This way it doesn't really matter how many step, it's more about how much time you want to let it run. Since it's only going keep 5 files you can just have it loop forever, until the user cancels the job.

But yeah I'm not a coder, I think the quickest dirties way is to simple include that the last number from the Loss_list when saving a state into the file name. That way the users can just visually see which one of the save sates has the lowest number of loss.