Make ipfs add slower or a seperate process

[fusir]

So the issue I'm having relates running a small video site. The problem comes when I want to add a video. The daemon runs full tilt and interupts the ability to serve content to the gateway and others.

So the issue is I want to slow down the addition of the file. There is no rush. If I cgroup the ipfs add so that it can only run at 80% cpu it does nothing because it is the ipfs daemon that actually does the processing.

We need some kind of way to tell it we want a file added but we are in no rush. Slowing it down even a little may have it running in 95% speed but leave a considerable remander for what little else has to be done for serving data.

So possible solutions are:

1. Tune down the addition code universally
2. Seperate the process or execute the child process in a way that will be labeled so it can be reniced or cgrouped by the user
3. Give us a configuration option for a slower add
4. Add a command line option to ipfs add for a slower add
5. Automatically renice the child process within the code
6. Move high cpu processing to the actual ipfs add call itself so it can be niced and cgrouped

In my opinion nice does a lot less than I think the developers intended so I don't think you can just nice the child process and expect it to work. From my experience it seems that coding something to be less agressive so that you have a better behaving process in the beginning is a lot better than renicing an out of control process. Nice really is a placebo. If there were a way for you to make the code less agressive I think that would be better because it would likely finish within 10-20% time and play better with the system.

[Stebalien]

I agree that an option to do this would be useful but, if you want to do it now, you can limit the rate at which the file is read in by using a tool like pv. That is, you can add the file by running:

> pv -L 1M my_video | ipfs add

(ideally, we'd have some form of QoS in the daemon but that's tricker)

[Stebalien]

Actually, I'm guessing your problem isn't adding, it's telling the network that you've added the file. I'd try adding files with ipfs add --local and manually announcing the pieces to the network:

> HASH="$(ipfs add --local --pin myfile)"
> ipfs dht provide "$HASH"
> ipfs refs "$HASH" | while read hash; do ipfs dht provide "$HASH" && sleep 10; done

Note: This will probably take a while. If you want, you may be able to get away with only announcing the root hash ($HASH).

[Kubuxu]

To replicate what add does, one would probably use refs -r instead of just refs.