search

radical-cybertools / radical.repex.at

This is the github location for RepEx developed by the RADICAL team in conjunction with the York Lab.
Other
4 stars 3 forks source link

not all CUs in MD step finished concurrently #34

Open haoyuanchen opened 9 years ago

haoyuanchen commented 9 years ago

I think we've discussed this issue in the last meeting. Here I'm just gonna show an example of it. This is from the first MD step in a TUU simulation with 256 replicas:

-rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000000/phos_trans_us.RST.0.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000001/phos_trans_us.RST.1.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000002/phos_trans_us.RST.2.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000003/phos_trans_us.RST.3.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000004/phos_trans_us.RST.4.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000005/phos_trans_us.RST.5.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000006/phos_trans_us.RST.6.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000007/phos_trans_us.RST.7.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000008/phos_trans_us.RST.8.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000009/phos_trans_us.RST.9.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000010/phos_trans_us.RST.10.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000011/phos_trans_us.RST.11.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000012/phos_trans_us.RST.12.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000013/phos_trans_us.RST.13.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000014/phos_trans_us.RST.14.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000015/phos_trans_us.RST.15.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000016/phos_trans_us.RST.16.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000017/phos_trans_us.RST.17.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000018/phos_trans_us.RST.18.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000019/phos_trans_us.RST.19.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000020/phos_trans_us.RST.20.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000021/phos_trans_us.RST.21.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000022/phos_trans_us.RST.22.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000023/phos_trans_us.RST.23.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000024/phos_trans_us.RST.24.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000025/phos_trans_us.RST.25.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000026/phos_trans_us.RST.26.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000027/phos_trans_us.RST.27.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000028/phos_trans_us.RST.28.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000029/phos_trans_us.RST.29.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000030/phos_trans_us.RST.30.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000031/phos_trans_us.RST.31.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000032/phos_trans_us.RST.32.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000033/phos_trans_us.RST.33.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000034/phos_trans_us.RST.34.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000035/phos_trans_us.RST.35.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:06 unit.000036/phos_trans_us.RST.36.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000037/phos_trans_us.RST.37.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000038/phos_trans_us.RST.38.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000039/phos_trans_us.RST.39.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000040/phos_trans_us.RST.40.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000041/phos_trans_us.RST.41.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000042/phos_trans_us.RST.42.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000043/phos_trans_us.RST.43.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000044/phos_trans_us.RST.44.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000045/phos_trans_us.RST.45.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000046/phos_trans_us.RST.46.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000047/phos_trans_us.RST.47.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000048/phos_trans_us.RST.48.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000049/phos_trans_us.RST.49.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000050/phos_trans_us.RST.50.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000051/phos_trans_us.RST.51.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000052/phos_trans_us.RST.52.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000053/phos_trans_us.RST.53.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000054/phos_trans_us.RST.54.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000055/phos_trans_us.RST.55.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000056/phos_trans_us.RST.56.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000057/phos_trans_us.RST.57.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000058/phos_trans_us.RST.58.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000059/phos_trans_us.RST.59.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000060/phos_trans_us.RST.60.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000061/phos_trans_us.RST.61.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000062/phos_trans_us.RST.62.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000063/phos_trans_us.RST.63.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000064/phos_trans_us.RST.64.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000065/phos_trans_us.RST.65.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000066/phos_trans_us.RST.66.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000067/phos_trans_us.RST.67.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000068/phos_trans_us.RST.68.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000069/phos_trans_us.RST.69.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000070/phos_trans_us.RST.70.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:06 unit.000071/phos_trans_us.RST.71.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000072/phos_trans_us.RST.72.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000073/phos_trans_us.RST.73.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000074/phos_trans_us.RST.74.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000075/phos_trans_us.RST.75.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000076/phos_trans_us.RST.76.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000077/phos_trans_us.RST.77.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000078/phos_trans_us.RST.78.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000079/phos_trans_us.RST.79.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000080/phos_trans_us.RST.80.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000081/phos_trans_us.RST.81.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000082/phos_trans_us.RST.82.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000083/phos_trans_us.RST.83.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000084/phos_trans_us.RST.84.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000085/phos_trans_us.RST.85.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000086/phos_trans_us.RST.86.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000087/phos_trans_us.RST.87.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000088/phos_trans_us.RST.88.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000089/phos_trans_us.RST.89.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:06 unit.000090/phos_trans_us.RST.90.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000091/phos_trans_us.RST.91.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000092/phos_trans_us.RST.92.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000093/phos_trans_us.RST.93.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000094/phos_trans_us.RST.94.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000095/phos_trans_us.RST.95.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000096/phos_trans_us.RST.96.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000097/phos_trans_us.RST.97.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000098/phos_trans_us.RST.98.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000099/phos_trans_us.RST.99.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000100/phos_trans_us.RST.100.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000101/phos_trans_us.RST.101.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000102/phos_trans_us.RST.102.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000103/phos_trans_us.RST.103.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000104/phos_trans_us.RST.104.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000105/phos_trans_us.RST.105.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000106/phos_trans_us.RST.106.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000107/phos_trans_us.RST.107.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000108/phos_trans_us.RST.108.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000109/phos_trans_us.RST.109.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000110/phos_trans_us.RST.110.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000111/phos_trans_us.RST.111.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000112/phos_trans_us.RST.112.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000113/phos_trans_us.RST.113.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000114/phos_trans_us.RST.114.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000115/phos_trans_us.RST.115.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000116/phos_trans_us.RST.116.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000117/phos_trans_us.RST.117.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000118/phos_trans_us.RST.118.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000119/phos_trans_us.RST.119.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000120/phos_trans_us.RST.120.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000121/phos_trans_us.RST.121.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000122/phos_trans_us.RST.122.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:06 unit.000123/phos_trans_us.RST.123.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000124/phos_trans_us.RST.124.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000125/phos_trans_us.RST.125.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000126/phos_trans_us.RST.126.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000127/phos_trans_us.RST.127.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000128/phos_trans_us.RST.128.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000129/phos_trans_us.RST.129.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000130/phos_trans_us.RST.130.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000131/phos_trans_us.RST.131.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000132/phos_trans_us.RST.132.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000133/phos_trans_us.RST.133.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000134/phos_trans_us.RST.134.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000135/phos_trans_us.RST.135.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000136/phos_trans_us.RST.136.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000137/phos_trans_us.RST.137.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000138/phos_trans_us.RST.138.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000139/phos_trans_us.RST.139.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000140/phos_trans_us.RST.140.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000141/phos_trans_us.RST.141.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000142/phos_trans_us.RST.142.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000143/phos_trans_us.RST.143.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000144/phos_trans_us.RST.144.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000145/phos_trans_us.RST.145.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000146/phos_trans_us.RST.146.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000147/phos_trans_us.RST.147.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000148/phos_trans_us.RST.148.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000149/phos_trans_us.RST.149.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000150/phos_trans_us.RST.150.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000151/phos_trans_us.RST.151.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000152/phos_trans_us.RST.152.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000153/phos_trans_us.RST.153.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:06 unit.000154/phos_trans_us.RST.154.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000155/phos_trans_us.RST.155.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000156/phos_trans_us.RST.156.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000157/phos_trans_us.RST.157.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000158/phos_trans_us.RST.158.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000159/phos_trans_us.RST.159.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000160/phos_trans_us.RST.160.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000161/phos_trans_us.RST.161.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000162/phos_trans_us.RST.162.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000163/phos_trans_us.RST.163.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000164/phos_trans_us.RST.164.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000165/phos_trans_us.RST.165.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000166/phos_trans_us.RST.166.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000167/phos_trans_us.RST.167.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000168/phos_trans_us.RST.168.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000169/phos_trans_us.RST.169.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000170/phos_trans_us.RST.170.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000171/phos_trans_us.RST.171.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000172/phos_trans_us.RST.172.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000173/phos_trans_us.RST.173.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000174/phos_trans_us.RST.174.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000175/phos_trans_us.RST.175.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000176/phos_trans_us.RST.176.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000177/phos_trans_us.RST.177.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000178/phos_trans_us.RST.178.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000179/phos_trans_us.RST.179.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000180/phos_trans_us.RST.180.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000181/phos_trans_us.RST.181.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000182/phos_trans_us.RST.182.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000183/phos_trans_us.RST.183.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000184/phos_trans_us.RST.184.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000185/phos_trans_us.RST.185.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000186/phos_trans_us.RST.186.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000187/phos_trans_us.RST.187.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000188/phos_trans_us.RST.188.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000189/phos_trans_us.RST.189.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000190/phos_trans_us.RST.190.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000191/phos_trans_us.RST.191.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000192/phos_trans_us.RST.192.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000193/phos_trans_us.RST.193.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000194/phos_trans_us.RST.194.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:06 unit.000195/phos_trans_us.RST.195.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000196/phos_trans_us.RST.196.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000197/phos_trans_us.RST.197.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000198/phos_trans_us.RST.198.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000199/phos_trans_us.RST.199.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000200/phos_trans_us.RST.200.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000201/phos_trans_us.RST.201.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000202/phos_trans_us.RST.202.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000203/phos_trans_us.RST.203.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000204/phos_trans_us.RST.204.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000205/phos_trans_us.RST.205.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000206/phos_trans_us.RST.206.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000207/phos_trans_us.RST.207.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000208/phos_trans_us.RST.208.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000209/phos_trans_us.RST.209.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:07 unit.000210/phos_trans_us.RST.210.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000211/phos_trans_us.RST.211.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000212/phos_trans_us.RST.212.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000213/phos_trans_us.RST.213.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000214/phos_trans_us.RST.214.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000215/phos_trans_us.RST.215.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000216/phos_trans_us.RST.216.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000217/phos_trans_us.RST.217.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000218/phos_trans_us.RST.218.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000219/phos_trans_us.RST.219.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000220/phos_trans_us.RST.220.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000221/phos_trans_us.RST.221.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000222/phos_trans_us.RST.222.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000223/phos_trans_us.RST.223.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000224/phos_trans_us.RST.224.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000225/phos_trans_us.RST.225.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000226/phos_trans_us.RST.226.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000227/phos_trans_us.RST.227.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000228/phos_trans_us.RST.228.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000229/phos_trans_us.RST.229.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000230/phos_trans_us.RST.230.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000231/phos_trans_us.RST.231.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000232/phos_trans_us.RST.232.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000233/phos_trans_us.RST.233.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000234/phos_trans_us.RST.234.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000235/phos_trans_us.RST.235.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000236/phos_trans_us.RST.236.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:07 unit.000237/phos_trans_us.RST.237.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000238/phos_trans_us.RST.238.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000239/phos_trans_us.RST.239.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000240/phos_trans_us.RST.240.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000241/phos_trans_us.RST.241.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000242/phos_trans_us.RST.242.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000243/phos_trans_us.RST.243.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000244/phos_trans_us.RST.244.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000245/phos_trans_us.RST.245.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000246/phos_trans_us.RST.246.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000247/phos_trans_us.RST.247.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:02 unit.000248/phos_trans_us.RST.248.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:07 unit.000249/phos_trans_us.RST.249.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:07 unit.000250/phos_trans_us.RST.250.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:10 unit.000251/phos_trans_us.RST.251.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:07 unit.000252/phos_trans_us.RST.252.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:07 unit.000253/phos_trans_us.RST.253.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:06 unit.000254/phos_trans_us.RST.254.out -rw------- 1 chen1990 G-803297 14529 Jun 22 13:07 unit.000255/phos_trans_us.RST.255.out

We can see that most CUs finished concurrently but there're several ones finished 4-8 minutes later than others, which delayed the entire simulation quite a bit.

Thanks! Haoyuan

antonst commented 9 years ago

Yes Haoyuan, I have seen this behaviour before. Not sure that is the reason. I assume this was observed when submitting to development queue? What happens when you submit to regular queue?

antonst commented 9 years ago

It might be the case that some cores are occupied by something else. So the simple way to check for this would be to allocate more cores than there are replicas, e.g. if you have 64 replicas allocate 80 cores (+1 node). I will investigate this.

haoyuanchen commented 9 years ago

I haven't submitted to the regular queue before. I did that 3 hours ago and it's still queueing. Will let you know the result once it starts running.

haoyuanchen commented 9 years ago

I exactly repeated the job on regular queue and I observed similar issue.

haoyuanchen commented 9 years ago

Allocating extra cores seems to help. I ran a 4x4x4 TUU simulation with 64 cores first, which had that sort of 2-3 minute delays for several replicas in every cycle that we've seen before. I then repeated the same simulation with 72 cores and there's no such delay in any cycle.

Branch: tuu-opt3

antonst commented 9 years ago

Yes, allocating extra cores helps, but does not eliminate the problem. For larger replica counts and when specifying more cycles I still see this sometimes.

ibethune commented 7 years ago

Not touched for 2 years -> Backburner