fqinwen
commented
3 years ago I have found the reason. It is because that the place of hadoop dir is not enough : )
Closed fqinwen closed 3 years ago
fqinwen
commented
3 years ago I have found the reason. It is because that the place of hadoop dir is not enough : )
Hello, I am using "./all_run.sh -e kmeans-
date +%m%d%H%M-s kmeans_baseline" to run kmeans_baseline, but it displays "No results after 10 minutes. Sleeping for 60 seconds" in several hours. I see the cpu and memory monitoring data of virtual machines, it looks like nothing runs in the virtual machines.Log is like this: `<11/06/2020 10:37:41 INFO common.osgcloud_common kmeans-11061037 KMeans baseline START time...20201106023741UTC MainThread osgcloud_kmeans_baseline.py:58 main 11/06/2020 10:37:41 INFO common.osgcloud_common retrieved CB api info from /tmp/cb_api_cbuser MainThread osgcloud_common.py:481 get_api_conn_info 11/06/2020 10:37:41 INFO common.osgcloud_common Searching for CB code at paths "/.././../../" MainThread osgcloud_common.py:510 setPath 11/06/2020 10:37:41 INFO common.osgcloud_common Found CB code at "/home/cbuser/osgcloud/cbtool" MainThread osgcloud_common.py:524 setPath 11/06/2020 10:37:41 INFO common.osgcloud_common Connecting to API daemon (http://10.10.10.74:7070)... MainThread osgcloud_kmeans_baseline.py:68 main 11/06/2020 10:37:41 INFO common.osgcloud_common Checking connection to cloud "MYZSTACK".... MainThread osgcloud_common.py:546 check_cloud_connection 11/06/2020 10:37:41 INFO common.osgcloud_common The "ZStack Elastic Compute Cloud", named "MYZSTACK" is reachable from this host MainThread osgcloud_common.py:566 check_cloud_connection 11/06/2020 10:37:41 INFO common.osgcloud_common Checking connection to cloud "MYZSTACK".... MainThread osgcloud_common.py:546 check_cloud_connection 11/06/2020 10:37:41 INFO common.osgcloud_common The "ZStack Elastic Compute Cloud", named "MYZSTACK" is reachable from this host MainThread osgcloud_common.py:566 check_cloud_connection 11/06/2020 10:37:41 INFO common.osgcloud_common ...Setting parameter vm_defaults:update_frequency to value 5 MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:41 INFO common.osgcloud_common ...Setting parameter vm_defaults:notification_channel to value EXPERIMENT MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:41 INFO common.osgcloud_common ...Setting parameter vm_defaults:sticky_app_status to value True MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_defaults:notification to value True MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_defaults:update_attempts to value 60 MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_defaults:recreate_attempts to value 1 MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_defaults:sla_provisioning_abort to value True MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_templates:cassandra to value size:speccloud-ins,remote_dir_name:cbtool,login:cbuser,imageid1:12B40B3D-5EEF-568E-8E84-5A0D1EB4FB36 MainThread osgcloud_common.py:244 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_templates:hadoopmaster to value size:speccloud-ins,remote_dir_name:cbtool,login:cbuser,imageid1:090D1E4B-739D-579D-B335-7972BAD0D9B8 MainThread osgcloud_common.py:244 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_templates:hadoopslave to value size:speccloud-ins,remote_dir_name:cbtool,login:cbuser,imageid1:090D1E4B-739D-579D-B335-7972BAD0D9B8 MainThread osgcloud_common.py:244 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_templates:seed to value size:speccloud-ins,remote_dir_name:cbtool,login:cbuser,imageid1:12B40B3D-5EEF-568E-8E84-5A0D1EB4FB36 MainThread osgcloud_common.py:244 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter vm_templates:ycsb to value size:speccloud-ins,remote_dir_name:cbtool,login:cbuser,imageid1:12B40B3D-5EEF-568E-8E84-5A0D1EB4FB36 MainThread osgcloud_common.py:244 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter logstore:expid_change_restart to value True MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter ai_defaults:attach_parallelism to value 10 MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter aidrs_defaults:daemon_parallelism to value 20 MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter mon_defaults:collect_from_host to value False MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common ...Setting parameter mon_defaults:collect_from_guest to value False MainThread osgcloud_common.py:249 setCommonConfig 11/06/2020 10:37:42 INFO common.osgcloud_common Running baseline load 5 times... MainThread osgcloud_kmeans.py:116 measure 11/06/2020 10:37:42 INFO common.osgcloud_common App error threshold 1 MainThread osgcloud_kmeans.py:117 measure 11/06/2020 10:37:42 INFO common.osgcloud_common Will wait for 5 runs per iteration... MainThread osgcloud_kmeans.py:118 measure 11/06/2020 10:37:42 INFO common.osgcloud_common ------------------------------------------------------ MainThread osgcloud_kmeans.py:119 measure 11/06/2020 10:37:42 INFO common.osgcloud_common Creating new K-Means Experiment ID of kmeans-11061037-KMEANS-BASELINE-0-20201106023741UTC... MainThread osgcloud_kmeans.py:131 measure 11/06/2020 10:37:47 INFO common.osgcloud_common ...Success. MainThread osgcloud_kmeans.py:133 measure 11/06/2020 10:37:47 INFO common.osgcloud_common Setting application parameters for workloadhadoop... MainThread osgcloud_kmeans.py:136 measure 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter load_level to value 1 MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter hadoopslave_data_dir to value /hadoopstore/hdfs/datanode MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter num_of_clusters to value 5 MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter load_factor to value 1000000 MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter hadoopmaster_data_dir to value /hadoopstore/hdfs/datanode MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter dfs_name_dir to value /usr/local/hadoop_store/hdfs/namenode MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter regenerate_data to value True MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter java_home to value /usr/lib/jvm/java-8-openjdk-arm64/ MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter load_profile to value kmeans MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter workload to value hadoop MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter dimensions to value 20 MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter samples_per_inputfile to value 500000 MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter dfs_data_dir to value /usr/local/hadoop_store/hdfs/datanode MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter vapp_pattern to value simplehd MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter num_maps to value 8 MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter sut to value hadoopmaster->5_x_hadoopslave MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter max_iteration to value 5 MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter hadoop_home to value /usr/local/hadoop MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:47 INFO common.osgcloud_common ...Setting parameter num_reds to value 4 MainThread osgcloud_common.py:180 setKMeansConfig 11/06/2020 10:37:48 INFO common.osgcloud_common ...Setting parameter load_duration to value 5 MainThread osgcloud_common.py:183 setKMeansConfig 11/06/2020 10:37:48 INFO common.osgcloud_common Setting Virtual Application Submitter for parameters hadoop MainThread osgcloud_common.py:188 setKMeansConfig 11/06/2020 10:37:48 INFO common.osgcloud_common ...Success. MainThread osgcloud_kmeans.py:138 measure 11/06/2020 10:37:48 INFO common.osgcloud_common Baseline run number0 MainThread osgcloud_kmeans.py:143 measure 11/06/2020 10:37:48 INFO common.osgcloud_common Creating kmeans application instance... MainThread osgcloud_kmeans.py:144 measure 11/06/2020 10:42:50 INFO common.osgcloud_common ...Success. Application instance name: ai_1 MainThread osgcloud_kmeans.py:147 measure 11/06/2020 10:42:50 INFO common.osgcloud_common ...List of Virtual Machines for this application instance: MainThread osgcloud_kmeans.py:148 measure 11/06/2020 10:42:50 INFO common.osgcloud_common ...Name | Role | Harness UUID | Deploy Time MainThread osgcloud_kmeans.py:149 measure 11/06/2020 10:42:50 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:42:50 INFO common.osgcloud_common ...Instance vm_1 | role = hadoopmaster | UUID = 289E2220-BEA1-53EB-8045-449FCEE246EA | Instance DeployTime = 107 | DeployTimeIncludingApp = 301 MainThread osgcloud_kmeans.py:175 measure 11/06/2020 10:42:50 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:42:50 INFO common.osgcloud_common ...Instance vm_2 | role = hadoopslave | UUID = 077C1836-EE41-50DF-BA12-F3428754D26E | Instance DeployTime = 96 | DeployTimeIncludingApp = 290 MainThread osgcloud_kmeans.py:175 measure 11/06/2020 10:42:50 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:42:50 INFO common.osgcloud_common ...Instance vm_5 | role = hadoopslave | UUID = 390954A4-F7CB-5A23-A9B6-8133CBF80CC1 | Instance DeployTime = 96 | DeployTimeIncludingApp = 290 MainThread osgcloud_kmeans.py:175 measure 11/06/2020 10:42:50 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:42:50 INFO common.osgcloud_common ...Instance vm_3 | role = hadoopslave | UUID = BFD6DBB6-924E-5FC7-90CF-3BCEA2AA0A98 | Instance DeployTime = 85 | DeployTimeIncludingApp = 279 MainThread osgcloud_kmeans.py:175 measure 11/06/2020 10:42:50 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:42:50 INFO common.osgcloud_common ...Instance vm_6 | role = hadoopslave | UUID = 030AC9C2-9CF7-5F81-8F21-8EAB30A1CBD3 | Instance DeployTime = 85 | DeployTimeIncludingApp = 279 MainThread osgcloud_kmeans.py:175 measure 11/06/2020 10:42:50 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:42:50 INFO common.osgcloud_common ...Instance vm_4 | role = hadoopslave | UUID = C1D0F25F-9E59-5378-A18E-05F6D45E6E50 | Instance DeployTime = 107 | DeployTimeIncludingApp = 301 MainThread osgcloud_kmeans.py:175 measure 11/06/2020 10:42:50 INFO common.osgcloud_common Waiting forever minutes for application to complete. CTRL-C to abort. MainThread osgcloud_kmeans.py:182 measure 11/06/2020 10:42:50 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:42:50 INFO common.osgcloud_common ...No results after 1 minutes. Sleeping for 60 seconds. MainThread osgcloud_kmeans.py:206 measure 11/06/2020 10:43:51 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:43:51 INFO common.osgcloud_common ...No results after 2 minutes. Sleeping for 60 seconds. MainThread osgcloud_kmeans.py:206 measure 11/06/2020 10:44:51 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:44:51 INFO common.osgcloud_common ...No results after 3 minutes. Sleeping for 60 seconds. MainThread osgcloud_kmeans.py:206 measure 11/06/2020 10:45:51 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:45:51 INFO common.osgcloud_common ...No results after 4 minutes. Sleeping for 60 seconds. MainThread osgcloud_kmeans.py:206 measure 11/06/2020 10:46:51 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:46:51 INFO common.osgcloud_common ...No results after 5 minutes. Sleeping for 60 seconds. MainThread osgcloud_kmeans.py:206 measure 11/06/2020 10:47:51 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 10:47:51 INFO common.osgcloud_common ...No results after 6 minutes. Sleeping for 60 seconds. MainThread osgcloud_kmeans.py:206 measure
......
11/06/2020 12:28:06 DEBUG root mongodb_datastore_adapter.py/MongodbMgdConn.connect TEST_cbuser - A connection to MongoDB running on host 10.10.10.74, port 27017, database metrics, with a timeout of 240s was established. MainThread code_instrumentation.py:143 _cblog 11/06/2020 12:28:06 INFO common.osgcloud_common ...No results after 106 minutes. Sleeping for 60 seconds. MainThread osgcloud_kmeans.py:206 measure>` Do you know the reason?
I would be really glad if you could help me with that.
Thank you.