Currently there are multiple configuration parameters to specify local/hdfs disk locations for checkpointing, saving tset, slipping over to disk during networking operations:
We have:
twister2.checkpointing.store.fs.dir and twister2.checkpointing.store.hdfs.dir in checkpoint.yaml
twister2.data.fs.root and twister2.data.hdfs.root in data.yaml
twister2.network.ops.persistent.dirs in network.yaml
Can we have a single parameter for local and hdfs disk locations? That would simplify configuring twister2 for users.
We can create a directory per job inside these directories not to get conflicts for concurrent jobs.
Currently there are multiple configuration parameters to specify local/hdfs disk locations for checkpointing, saving tset, slipping over to disk during networking operations:
We have: twister2.checkpointing.store.fs.dir and twister2.checkpointing.store.hdfs.dir in checkpoint.yaml twister2.data.fs.root and twister2.data.hdfs.root in data.yaml twister2.network.ops.persistent.dirs in network.yaml
Can we have a single parameter for local and hdfs disk locations? That would simplify configuring twister2 for users.
We can create a directory per job inside these directories not to get conflicts for concurrent jobs.