zhangys-lucky
commented
3 years ago here has a demo: examples/common/util.py
dataset = ds_data_files.shard(num_shards=tn.core.shard_num(), index=tn.core.self_shard_id())Open pengyuan opened 3 years ago
zhangys-lucky
commented
3 years ago here has a demo: examples/common/util.py
dataset = ds_data_files.shard(num_shards=tn.core.shard_num(), index=tn.core.self_shard_id())
when tensorflow read dataset from kafka, how to implement each partition read by one worker?
something like hovorod: a. get the total worker number and worker index by rank() b. then apply shard() method in tensorflow