search

benchflow / data-transformers

Spark scripts utilised to transform data to the BenchFlow internal formats
Other
0 stars 0 forks source link

Stats Transformer Has Errors when Net is not Host #84

Closed VincenzoFerme closed 7 years ago

VincenzoFerme commented 7 years ago

An execution log reporting the error:

8/20/2016 5:55:54 PMReceived message: {"minio_key":"dfe8/BenchFlow.wfmsTest.1.1/BenchFlow.wfmsTest.1.1.1/benchflow.collector.stats.test_BenchFlow.wfmsTest.1.1.1/stats/stats/outyet.test_BenchFlow.wfmsTest.1.1.1","trial_id":"BenchFlow.wfmsTest.1.1.1","experiment_id":"BenchFlow.wfmsTest.1.1","container_id":"141e371cdd782f46e6c1bab5bdf79e5c1687371e1ad0dc24e91e46e6db575848","container_name":"outyet.test","host_id":"RBHO:FXNA:VSSR:UUY6:NUMZ:2H2G:ZRJZ:RAWJ:7YYI:T2FR:5DHG:3GWG","collector_name":"stats"}
8/20/2016 5:55:54 PMstats topic, submitting script data-transformers/transformers/statsTransformer.py, minio location: dfe8/BenchFlow.wfmsTest.1.1/BenchFlow.wfmsTest.1.1.1/benchflow.collector.stats.test_BenchFlow.wfmsTest.1.1.1/stats/stats/outyet.test_BenchFlow.wfmsTest.1.1.1, trial id: BenchFlow.wfmsTest.1.1.1
8/20/2016 5:55:54 PMTransformer work request queued for script stats, camunda, 7.5.0, BenchFlow.wfmsTest.1.1.1, 141e371cdd782f46e6c1bab5bdf79e5c1687371e1ad0dc24e91e46e6db575848, RBHO:FXNA:VSSR:UUY6:NUMZ:2H2G:ZRJZ:RAWJ:7YYI:T2FR:5DHG:3GWG, trial
8/20/2016 5:55:54 PMReceived work request for script stats, camunda, 7.5.0, BenchFlow.wfmsTest.1.1.1, 141e371cdd782f46e6c1bab5bdf79e5c1687371e1ad0dc24e91e46e6db575848, RBHO:FXNA:VSSR:UUY6:NUMZ:2H2G:ZRJZ:RAWJ:7YYI:T2FR:5DHG:3GWG, trial
8/20/2016 5:55:54 PMDispatching work request for script stats, camunda, 7.5.0, BenchFlow.wfmsTest.1.1.1, 141e371cdd782f46e6c1bab5bdf79e5c1687371e1ad0dc24e91e46e6db575848, RBHO:FXNA:VSSR:UUY6:NUMZ:2H2G:ZRJZ:RAWJ:7YYI:T2FR:5DHG:3GWG, trial
8/20/2016 5:55:54 PMTransformer worker submitting script:  [--master local[*] --jars /usr/spark/pyspark-cassandra-assembly-0.3.5.jar --driver-memory 12g --executor-memory 12g --conf spark.executor.heartbeatInterval=300000s --conf spark.storage.blockManagerSlaveTimeoutMs=300000s --conf spark.core.connection.ack.wait.timeout=300000s --conf spark.cassandra.connection.host=cassandra --driver-class-path /usr/spark/pyspark-cassandra-assembly-0.3.5.jar --py-files /app/data-transformers/commons/commons.py,/app/data-transformers/transformations/dataTransformations.py,/usr/spark/pyspark-cassandra-assembly-0.3.5.jar /app/data-transformers/transformers/statsTransformer.py {"cassandra_keyspace":"benchflow","minio_host":"minio","minio_port":"9000","minio_access_key":"CYNQML6R7V12MTT32W6P","minio_secret_key":"SQ96V5pg02Z3kZ/0ViF9YY6GwWzZvoBmElpzEEjn","file_bucket":"runs","file_path":"dfe8/BenchFlow.wfmsTest.1.1/BenchFlow.wfmsTest.1.1.1/benchflow.collector.stats.test_BenchFlow.wfmsTest.1.1.1/stats/stats/outyet.test_BenchFlow.wfmsTest.1.1.1","trial_id":"BenchFlow.wfmsTest.1.1.1","experiment_id":"BenchFlow.wfmsTest.1.1","config_file":"data-transformers.configuration.yml","container_id":"141e371cdd782f46e6c1bab5bdf79e5c1687371e1ad0dc24e91e46e6db575848","host_id":"RBHO:FXNA:VSSR:UUY6:NUMZ:2H2G:ZRJZ:RAWJ:7YYI:T2FR:5DHG:3GWG"}]
8/20/2016 5:55:54 PM/apps/bin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/sbin
8/20/2016 5:55:59 PMCould not save on alluxio file BenchFlow.wfmsTest.1.1.1_141e371cdd782f46e6c1bab5bdf79e5c1687371e1ad0dc24e91e46e6db575848_environment_data
8/20/2016 5:56:05 PMTraceback (most recent call last):
8/20/2016 5:56:05 PM  File "/app/data-transformers/transformers/statsTransformer.py", line 268, in <module>
8/20/2016 5:56:05 PM    main()
8/20/2016 5:56:05 PM  File "/app/data-transformers/transformers/statsTransformer.py", line 250, in main
8/20/2016 5:56:05 PM    query = statsRDD.map(f).reduce(lambda a, b: a+b)
8/20/2016 5:56:05 PMNameError: global name 'statsRDD' is not defined
8/20/2016 5:56:08 PM
[Stage 0:>                                                        (0 + 0) / 240]
[Stage 0:>                                                       (0 + 48) / 240]
[Stage 0:>                                                       (0 + 49) / 240]
[Stage 0:>                                                       (1 + 66) / 240]
[Stage 0:====================>                                  (88 + 49) / 240]
[Stage 0:==========================>                           (117 + 48) / 240]
[Stage 0:==========================================>           (189 + 48) / 240]

[Stage 2:===================================>                  (156 + 49) / 240]
8/20/2016 5:56:08 PMScript data-transformers/transformers/statsTransformer.py processed
8/20/2016 5:56:08 PMTransformer script completed successfully: stats, camunda, 7.5.0, BenchFlow.wfmsTest.1.1.1, 141e371cdd782f46e6c1bab5bdf79e5c1687371e1ad0dc24e91e46e6db575848, RBHO:FXNA:VSSR:UUY6:NUMZ:2H2G:ZRJZ:RAWJ:7YYI:T2FR:5DHG:3GWG, trial