cerquide
commented
10 years ago I have perceived the same effect when running PageRank, although I have not measured times. This will point towards a bug/problem in GraphX, right?
Open GuillermoBlasco opened 10 years ago
cerquide
commented
10 years ago I have perceived the same effect when running PageRank, although I have not measured times. This will point towards a bug/problem in GraphX, right?
GuillermoBlasco
commented
10 years ago http://apache-spark-user-list.1001560.n3.nabble.com/GraphX-Is-it-normal-to-shuffle-write-15GB-while-the-data-is-only-30MB-td11774.html http://apache-spark-user-list.1001560.n3.nabble.com/Question-about-RDD-cache-unpersist-materialization-td7374.html http://apache-spark-user-list.1001560.n3.nabble.com/Pregel-runs-slower-and-slower-when-each-Pregel-has-data-dependency-td7648.html
Each line outputs the error and the system time in milliseconds
Guillermos-MacBook-Air:gmX guillermoblascojimenez$ $SPARK_HOME/bin/spark-submit --class "com.blackbox.gmx.example.Student" --master local[4] target/scala-2.10/gmx_2.10-1.0.jar 14/10/06 11:30:00 WARN SparkConf: SPARK_JAVA_OPTS was detected (set to ' -Dspark.serializer=org.apache.spark.serializer.KryoSerializer -Dspark.kryo.registrator=org.apache.spark.graphx.GraphKryoRegistrator '). This is deprecated in Spark 1.0+.
Please instead use:
14/10/06 11:30:00 WARN SparkConf: Setting 'spark.executor.extraJavaOptions' to ' -Dspark.serializer=org.apache.spark.serializer.KryoSerializer -Dspark.kryo.registrator=org.apache.spark.graphx.GraphKryoRegistrator ' as a work-around. 14/10/06 11:30:00 WARN SparkConf: Setting 'spark.driver.extraJavaOptions' to ' -Dspark.serializer=org.apache.spark.serializer.KryoSerializer -Dspark.kryo.registrator=org.apache.spark.graphx.GraphKryoRegistrator ' as a work-around. Cluster Graph built Cluster with 8 clusters error: 0.12519177621599598 at 1412587808098 error: 0.0372313362941709 at 1412587809062 error: 0.06300571554158453 at 1412587809978 error: 0.00161212143632129 at 1412587810791 error: 0.030923425188297375 at 1412587811548 error: 0.054924511754993896 at 1412587812446 error: 0.00179216227496699 at 1412587813092 error: 0.026453902880653097 at 1412587813676 error: 0.04633838053832956 at 1412587814227 error: 0.0019397498390229576 at 1412587814756 error: 0.021979344335079948 at 1412587815193 error: 0.037895937786007076 at 1412587815570 error: 0.0020447103515861204 at 1412587815960 error: 0.017800920542429398 at 1412587816370 error: 0.030117038195337953 at 1412587816803 error: 0.0021008989559054185 at 1412587817336 error: 0.014122736707891216 at 1412587817809 error: 0.023341273358217927 at 1412587818336 error: 0.0021067342348015045 at 1412587818805 error: 0.011044122733230993 at 1412587819282 error: 0.017722184448606144 at 1412587819783 error: 0.002064930329880744 at 1412587820272 error: 0.008574193904316924 at 1412587820734 error: 0.013257235603442087 at 1412587821255 error: 0.001981613649191978 at 1412587821748 error: 0.006658818172795672 at 1412587822271 error: 0.009836377066600989 at 1412587822832 error: 0.0018651133405955605 at 1412587823427 error: 0.005209808956860952 at 1412587824059 error: 0.007292594390134931 at 1412587824722 error: 0.001724712954617012 at 1412587825355 error: 0.004129219619470362 at 1412587826043 error: 0.00544330835573178 at 1412587826734 error: 0.0015695776468153963 at 1412587827453 error: 0.0033256497207461593 at 1412587828197 error: 0.004118071455019422 at 1412587828965 error: 0.001407971738427338 at 1412587829776 error: 0.0027226952114126094 at 1412587830663 error: 0.003173069003239549 at 1412587831625 error: 0.001246791472562367 at 1412587832493 error: 0.002261411859234501 at 1412587833425 error: 0.002495589621008917 at 1412587834445 error: 0.0010913759412709664 at 1412587835472 error: 0.0018990935164854544 at 1412587836638 error: 0.0020022487072031683 at 1412587837774 error: 9.455283197521791E-4 at 1412587838964 error: 0.0016063215660626236 at 1412587840208 error: 0.0016341730447027739 at 1412587841567 error: 8.116733361830232E-4 at 1412587842921 error: 0.0013636151992905824 at 1412587844582 error: 0.0013513514689513575 at 1412587846075 error: 6.9108589586185E-4 at 1412587847592 error: 0.0011584153631472333 at 1412587849197 error: 0.0011274015433960385 at 1412587850789 error: 5.841414925669533E-4 at 1412587852685 error: 9.826983437212836E-4 at 1412587854452 error: 9.452949549260945E-4 at 1412587856312 error: 4.905556028834128E-4 at 1412587858777 error: 8.312518158165064E-4 at 1412587860722 error: 7.941479333414236E-4 at 1412587862791 error: 4.0959345835718075E-4 at 1412587865178 error: 7.005185711507701E-4 at 1412587867307 error: 6.66960775445E-4 at 1412587869639 error: 3.402422239565994E-4 at 1412587872412 error: 5.878721798916212E-4 at 1412587875131 error: 5.591084200205985E-4 at 1412587877876 error: 2.8134470297948023E-4 at 1412587880482 error: 4.911941301720584E-4 at 1412587883528 error: 4.673795723369543E-4 at 1412587886294 error: 2.316978488437011E-4 at 1412587889442 error: 4.08646498442017E-4 at 1412587892523 error: 3.8939242588711693E-4 at 1412587895810 error: 1.9012137346270506E-4 at 1412587898961 error: 3.3856260888576244E-4 at 1412587902726 error: 3.2325556409674224E-4 at 1412587906439 error: 1.5550232517912334E-4 at 1412587910030 error: 2.7940323852191685E-4 at 1412587913892 error: 2.673806875499431E-4 at 1412587917908 error: 1.2682124391892125E-4 at 1412587921811 error: 2.2974531031923326E-4 at 1412587926020 error: 2.203845929616536E-4 at 1412587931047 error: 1.0316480149530829E-4 at 1412587936686 error: 1.8828371244058144E-4 at 1412587943826 error: 1.810405714242484E-4 at 1412587949923 error: 8.372897270365223E-5 at 1412587955998 error: 1.5383589335611405E-4 at 1412587962615 error: 1.4825513370506739E-4 at 1412587970265 error: 6.78159206745734E-5 at 1412587976948 error: 1.2534443071747666E-4 at 1412587983611 error: 1.210562958977216E-4 at 1412587990095 error: 5.482700395990345E-5 at 1412587997221 error: 1.0187594691457726E-4 at 1412588004837 error: 9.858610511237596E-5 at 1412588012438 error: 4.425365933934775E-5 at 1412588020448 error: 8.261639376011903E-5 at 1412588030284 error: 8.009390808734914E-5 at 1412588040433 error: 3.5667389872503174E-5 at 1412588049176 error: 6.686345694268794E-5 at 1412588057937 error: 6.492900014604511E-5 at 1412588067124 error: 2.8709682698803304E-5 at 1412588076591 Calibrated error: 0.12283588778298232 at 1412588095804 error: 0.06734689401269783 at 1412588096008 error: 0.08202427250069447 at 1412588096210 error: 0.009125868090623329 at 1412588096437 error: 0.02370906736888987 at 1412588096708 error: 0.061729871366266405 at 1412588097013 error: 0.005303302363843675 at 1412588097318 error: 0.0132107241097436 at 1412588097586 error: 0.04947564926318184 at 1412588097884 error: 0.0027130234216898195 at 1412588098134 error: 0.00777109400968799 at 1412588098444 error: 0.03557238065169925 at 1412588098725 error: 0.0012684945620320513 at 1412588099028 error: 0.004707108373747947 at 1412588099362 error: 0.02296395393643073 at 1412588099722 error: 5.628445113616308E-4 at 1412588100087 error: 0.0027959802023483143 at 1412588100402 error: 0.013543437291183753 at 1412588100735 error: 2.4419009522826145E-4 at 1412588101171 error: 0.0015876828700689084 at 1412588101513 error: 0.007451464602969695 at 1412588101932 error: 1.0561038573350697E-4 at 1412588102317 error: 8.595522937915052E-4 at 1412588102785 error: 0.00389643723865832 at 1412588103344 error: 4.59926284801572E-5 at 1412588103875 error: 4.468585100072085E-4 at 1412588104572 error: 0.001965049017029566 at 1412588105137 error: 2.024341879241442E-5 at 1412588105706 error: 2.251356055541293E-4 at 1412588106258 error: 9.662161650241316E-4 at 1412588106926 error: 9.007157354916817E-6 at 1412588107478 Map calibrated
If you plot the differences in a graph it actually scares...
https://drive.google.com/file/d/0BzzgOwBUb-lyaE5SaEk1ZkhSYVE/view?usp=sharing
https://drive.google.com/file/d/0BzzgOwBUb-lyLTNPeTVad1FpVGs/view?usp=sharing