search

Rice-Comp413-2016 / RDFS

The Rice Comp413 class (2016-2017) implementation of HDFS. (This will eventually be put under an open source license, which one TBD).
4 stars 2 forks source link

Periodically write DataNode statistics to ZooKeeper #57

Closed nichhk closed 7 years ago

nichhk commented 7 years ago

In particular, the NameNode needs to know 1) the free disk space on a DataNode and 2) the current number of transmissions on a DataNode when selecting target DataNodes for replication.

nichhk commented 7 years ago

blocked by #58 and #59

nichhk commented 7 years ago

We probably want to use @ap35's timer for this.

efrederic commented 7 years ago

We should send stats (number of xmits, available disk space, etc) from the datanode to zookeeper every 3(?) seconds. The stats will be added to the existing DataNodePayload struct in zk_dn_client.h

DataNodePayload fields: uint32_t ipcPort; uint32_t xferPort; uint64_t disk_bytes; //total space on disk uint64_t free_bytes; //free space on disk uint32_t xmits; //current number of xmits