I think something similar has been implemented by FB.
NN/DN/HDFS enhancement need to support file 'registration' (now we create, open, delete) - we need 'register' as well . Agent creates all replicas locally on K data nodes (K - is replication factor), agent register new file with DNs and NN. How? This new API will allow to reduce or even eliminate network traffic during HBase compaction.
Check HBase JIRA.
I think something similar has been implemented by FB. NN/DN/HDFS enhancement need to support file 'registration' (now we create, open, delete) - we need 'register' as well . Agent creates all replicas locally on K data nodes (K - is replication factor), agent register new file with DNs and NN. How? This new API will allow to reduce or even eliminate network traffic during HBase compaction. Check HBase JIRA.