Implement file lease recovery.

[shvachko]

File leases have expiration periods. LeaseManager monitors leases and when any of them expires due to client inactivity, LeaseManager schedules lease recovery. Lease recovery is handled by the BlockManagement layer via the standard HDFS block recovery procedure.

[pjeli]

Created a pull request for this work.

I added LeaseMonitor class (creates Runnable for monitoring leases which is held by the NamespaceProcessor).

Each LeaseManager can only have 1 LeaseMonitor.

I also implemented NamespaceProcessor and NamespaceAgent's "recoverLease" calls. They both force the file to recover and close.

Block and file recovery is handled by BlockManagementAgent.

The code is overall pretty small. Please take a look.

[milandesai]

I reviewed this and posted specific nits as annotations on #177. I checked unit tests, synchronization, and lease recovery logic, and they all look good except that when the NamespaceProcessor shuts down, it should properly shut down the LeaseMonitor the way HDFS does (by setting a boolean, then interrupting, then joining the thread). Currently we simply interrupt the monitor, but that may not necessarily do anything if the thread is sleeping.

[pjeli]

Thanks Milan. I updated my branch with a new commit to address your concerns.

[milandesai]

+1

[pjeli]

Thanks Milan. This is committed now as 9d3b51e4e05027affaa9ea50db6f3f72b4a46c9c.