file upload support on new rmb

[muhamadazmy]

The new rmb can support file upload as follows:

• an rmb-peer reserves commands with special prefixes rmb.
• the peer itself need to handle the following commands
  • rmb.file.upload > returns a uid that need to be included in the following calls
  • rmb.file.write > (uid, offset, data)
  • The offset is needed in case messages are received out of order although that should not happen
  • make rmb.file.write calls as much as needed
  • make rmb.file.done(uid)

The UID is then can be used to pass to other services running behind rmb.

[muhamadazmy]

Specifications

The idea behind this feature is:

• A client can send a request to it's local rmb-peer to upload a file to a remove twin (DST)
  • it uses a special command in the outgoing request (say rmb.file.upload) the data in this case then can be path to the local file.
  • so let's imagine it's like that rmb.file.upload()
• The local rmb received this command over it's msgbus.system.local redis channel.
• Normally, what rmb does is to directly forward this message to remote twin. but since this is a special command (anything that is prefixed with rmb) it need to be handled locally.
• the special handler then (which should work in the background) will make the proper calls to the remote twin
• Once file is uploaded a response is initiated "from local rmb" to the client to tell it everything is complete, or report any possible error.

Peer file upload support

Only peers that are started with --upload-dir accepts file upload calls, hence a peer can choose if it can accept a upload call or not.

All peers though can accept requests to upload files from their local services. Hence the local peer will try to upload the file, and the remote peer can either accept or reject this request based on its configuration.

On accepting a local rmb.file.upload call with the path of the file the local rmb will spawn a routine to do the upload as follows:

• It calls remove rmb.file.upload, the arguments should be the full file size
• If remote rmb allows upload it should be able to respond back with success, and a special uuid that will be used as the upload id for the rest of the session
• an erro can be returned on doing rmb.file.upload this can be either permission denied (say upload is not enabled) or if there is no enough space to receive the full size.
• if successful, local rmb, can then continue with the upload as follows
  • create a request envelope, with all necessary fields, plus using the proper command rmb.file.write where the binary data is the chunk of the file to write (we need to find a proper file chunk size that works the best on the wire)
  • we can use the envelope tag to cramp the uuid of the upload, can be either <uuid>:<offset> or more complex string like uuid=<uuid> offset=<offset>
  • We are avoiding encoding both the actual chunk + arguments in data because this will force us to use some sort of encoding, and decoding in memory of both sender and receiver that we can avoid by using the tags fields for extra arguments, hence the data filed can only carry the actual chunk to upload.
  • on receive, the remote rmb will write the received chunk in the correct offset of the file
• Last call from the local rmb is to close the upload (mark there are no more parts)
• Once close is successful (receive a success response from the remote rmb) a response is created back to the local client

[muhamadazmy]

Already started a new branch support-built-in-fn that already has some clean up of the code.