`initFederatorBackend` doesn't clean-up stale entries.

[maihde]

The initFederatorBackend adds new buckets to the global DNS on startup; but if a bucket is removed while MinIO is stopped it doesn't remove the bucket. This can leave stranded entries in etcd that show up but cannot be deleted using the S3 API. This is because handlers associated with removing the bucket.

Expected Behavior

In federated mode, MinIO cleans up DNS entries that are associated with it but no longer exist.

Current Behavior

Buckets that have been removed while the MinIO server is stopped remain in DNS and cannot be deleted via the S3 API.

Possible Solutions

• Change initFederatorBackend to look for abandoned DNS entries and clean them up. Note care must be taken to ensure that you don't clean out entries associated with other servers. I have prototype code for this if you are interested in my solution.
• Change DeleteBucket handlers to clean-up stray entries even if the local directory no longer exists. Note care must be taken to ensure that you don't clean out entries associated with other servers.

Steps to Reproduce (for bugs)

1. Start two MinIO servers in federated mode
2. Create a bucket on one of the server
3. Stop the MinIO server
4. Remove the directory associated with the bucket
5. The bucket will still show up in the web browser list on the left and via mc ls
6. Attempting to delete the bucket in the web browser or via mc will throw an error that The specified bucket does not exist which is true; but the bucket remains in the list.

[harshavardhana]

Can you send the prototype solution?