msr-fiddle / pipedream

MIT License
375 stars 117 forks source link

Infinite loop problem in convert_graph_to_model.py #45

Open jayhpark530 opened 4 years ago

jayhpark530 commented 4 years ago

I had a problem using the optimizer for the resnet101 model. Following the README, the following commands were executed sequentially.

python optimizer_graph_hierarchical.py \ -f ../profiler/image_classification/profiles/resnet101/graph.txt \ --activation_compression_ratio 1 \ -o resnet101_partitioned \ --all_num_machines 4 4 \ --network_bandwidths 15750000000 7000000000 \ --memory_size 16000000000

python convert_graph_to_model.py \ -f resnet101_partitioned/gpus=16.txt \ -n RESNET101Partitioned \ -a resnet101 \ -o ../runtime/image_classification/models/gpus=16

In the resnet101 (or resnet152) model, an infinite loop occurs in convert_graph_to_model.py. (resnet50 or vgg16 had no problem.)

When I checked it, an infinite loop occurs in the populate_depths function in pipedream/graph/graph.py.

Is my execution wrong? Can you confirm what's the problem?

nirandaperera commented 4 years ago

I had a similar issue for densenet121

deepakn94 commented 4 years ago

Not sure -- haven't run this code in a while. The populate_depths function itself is not too important -- you could perhaps try commenting it out and seeing what happens

nirandaperera commented 4 years ago

commenting out populate_depths worked actually