Way to "load balance" archiving with multiple plotters

[FVierra27]

I have three plotters running. Occasionally, when they finish at the same time, all three plotters start archiving to the first drive with space. I think if there was a way to select the drive with the most space instead of the first drive with space it could balance the load a bit better?

[graemes]

If you use the 'index' option in the archive configuration section you can specify different destination directories for each plotter.

[randellhodges]

Assuming you have more disks with free space than plotters, you can try to merge my changes by hand.

https://github.com/ericaltendorf/plotman/pull/642

It doesn't balance really, it just looks at the .plot.xxxx.plot.yyyy temp files rsync creates and if it finds one (a copy from a plotter is in place) then it'll look at the next disk. So, assuming you have 6 disks with free space and 3 plotters, if each finishes at about the same time, each one will copy to a different archive disk.

It is a rough PR, more of a proof of concept. I'm using it. It'll probably be invalidated with upcoming changes.

[guloooo128]

Use multiple rsync modules to balance the plotter. Each plotter uses different modules.

[Saviola8x]

I have same issue when have man plotters and 1 rsynced server with many avaiable disk, i must edit the source of archive.py to :

• sorting the disks with most free disk in top of list (in default sorted by plots disk name).
• plotman config with index 0 then rsync will use the disk which has most free disk.

You can try with edit file /ROOT_DIR/chia-blockchain/venv/lib/python[VERSION]/site-packages/plotman/archive.py : from : (archdir, freespace) = sorted(available)[index] to : (archdir, freespace) = sorted(available,key=lambda x: x[1], reverse=True)[index]

This will change sort way from dir name to disk free (reverse sort, biggest number in top).

[altendky]

As mentioned earlier, just set archiving: index: to a different number on each plotter.

[FVierra27]

I have same issue when have man plotters and 1 rsynced server with many avaiable disk, i must edit the source of archive.py to :

• sorting the disks with most free disk in top of list (in default sorted by plots disk name).
• plotman config with index 0 then rsync will use the disk which has most free disk.

You can try with edit file /ROOT_DIR/chia-blockchain/venv/lib/python[VERSION]/site-packages/plotman/archive.py : from : (archdir, freespace) = sorted(available)[index] to : (archdir, freespace) = sorted(available,key=lambda x: x[1], reverse=True)[index]

This will change sort way from dir name to disk free (reverse sort, biggest number in top).

Indexing to different drives with each plotter has been working for me.

[Saviola8x]

@FVierra27 Did you open issue with asking about load balancing to archiving to most space disk ? @altendky I'm known about "index:" but I'm replying to this thread which that function can't be done.

[altendky]

This issue is about handling multiple plotters using plotman's archiving feature and avoiding having two or more plotters archive to a single drive at a time. This is important because that would reduce the total throughput to the disk because the drive heads would likely end up seeking back and forth a bunch to write the two files at the same time. The index: configuration is for this. It isn't super fancy, but it is useful.

If your issue is about free space, then it is separate. But, the purpose of the archiving feature is exactly to fill all the disks. It may do it in a different order than you have in mind, but it does generally work. If you have a problem with it, maybe start with a separate discussion here on GitHub or join the #plotman channel on Chia's keybase team.