Maintain blacklisted fogs during writes to ensure same fogs are not requested in case of write failures

[skmonga]

It might happen that replica identification returns a set of fogs but some of the fogs are not able to serve the write request due to several reasons (edge failures, edge storage below the disk watermark, etc). In such a case, during the retry phase, the api for replica identification must support a list of blacklisted fogs which will not be picked for writing again and will only search through the rest of the fogs.

[simmhan]

The client will, for that API call session, maintain fogs that do not have adequate edge count capacity at a given quadrant. If a fog that it contacts does not have edges at that quad, the fog will return an alternative fog which it thinks has capacity. The client will pass the blacklist as part of this exchange so that the alternative fog is not part of this.

[skmonga]

The client will initially get the list of fogs which will take part in writing of a data block. Initially the client will not be aware of any fogs which doesn't have adequate edge count capacity as we are doing it on an api call session basis. While performing the write operation, it might fail and client on an error response can populate the list of blacklisted fogs and the fog it communicated with can return an alternative fog. The client can then communicate with the alternative fog passing the blacklisted fogs as a parameter. However, the writes by the client are parallel and the blacklisted fogs list is building up as the writes are performed, so need to take care of parallelism in the write operation.