Closed iychoi closed 10 years ago
By default, Syndicate waits 10 seconds to connect and 300 seconds on transfer before timing out (see md_default_conf() in libsyndicate/libsyndicate.cpp). This might be too long for you. You can override them in a config file, using METADATA_CONNECT_TIMEOUT="xxx" and TRANSFER_TIMEOUT="xxx", respectively.
By default, the Gateway methods are meant to fail fast (no retry), so the application can decide how to handle the error. However, if you want to add network-level retries in your branch, all the relevant methods are in UG/network.cpp (particularly fs_entry_download_manifest(), fs_entry_download_block(), fs_entry_post_write(), and fs_entry_send_write_or_coordinate()).
Okey, reducing TRANSFER_TIMEOUT value will be helpful. But still I'm not sure where to put retries. UG-IPC or H-Syndicate(Hadoop-Syndicate connector) can potentially have this. Let me figure out how NFS works for this.
It seems NFS has built-in retries for requests as it uses UDP for transport protocol. Also providing retries on Syndicate UG(Both UG-FUSE and UG-IPC) level would provide better compatibility to other user programs. How do you think?
I think we can add support for retrying up to a certain number of times (probably no more than 5 times), if they can be done in rapid succession (i.e. no more than a few seconds between attempts). I'd want the UG to handle transient message loss transparently, but longer-lived failures (such as a chronic partition or offline gateway) shouldn't hang the system.
Get it, I'll look into this after M.S. Thesis :-)
UGs now re-try manifest requests between one another, as of db4f18e44032d8379ba6b4116150a42a72366f67.
Metadata reads from the MS will now be retried at most max_read_metadata_retry (in struct md_syndicate_conf), and manifest reads from gateways will now be retried at most max_read_retry (in struct md_syndicate_conf). You can set default values in md_default_conf() in libsyndicate/libsyndicate.cpp.
Alright thanks :+1:
When network is not stable, requests to other Gateways sometimes are lost. Hence, further operations are hanging.
Need timeout and retry logic is necessary.