How should I use Spark version of datagen to access LiveGraph

[ggssh]

#!/bin/bash
DEFAULT_HADOOP_HOME=$HOME/hadoop-2.6.0 #change to your hadoop folder
DEFAULT_LDBC_SNB_DATAGEN_HOME=`pwd` #change to your ldbc_snb_datagen folder

# allow overriding configuration from outside via environment variables
# i.e. you can do
#     HADOOP_HOME=/foo/bar LDBC_SNB_DATAGEN_HOME=/baz/quux ./run.sh
# instead of changing the contents of this file
HADOOP_HOME=${HADOOP_HOME:-$DEFAULT_HADOOP_HOME}
LDBC_SNB_DATAGEN_HOME=${LDBC_SNB_DATAGEN_HOME:-$DEFAULT_LDBC_SNB_DATAGEN_HOME}

export HADOOP_HOME
export LDBC_SNB_DATAGEN_HOME
export HADOOP_CLIENT_OPTS="-Xmx64G"

echo ===============================================================================
echo Running generator with the following parameters:
echo -------------------------------------------------------------------------------
echo LDBC_SNB_DATAGEN_HOME: $LDBC_SNB_DATAGEN_HOME
echo HADOOP_HOME: $HADOOP_HOME
echo HADOOP_CLIENT_OPTS: $HADOOP_CLIENT_OPTS
echo ===============================================================================

#  --- for OSX, add this line ---
# zip -d $LDBC_SNB_DATAGEN_HOME/target/ldbc_snb_datagen-0.2.7-jar-with-dependencies.jar META-INF/LICENSE

$HADOOP_HOME/bin/hadoop jar ldbc_snb_datagen-0.2.7-jar-with-dependencies.jar $LDBC_SNB_DATAGEN_HOME/params-foreign-key.ini

rm -f m*personFactors*
rm -f .m*personFactors*
rm -f m*activityFactors*
rm -f .m*activityFactors*
rm -f m0friendList*
rm -f .m0friendList*

The current version used in run.sh is Hadoop. How should I modify it?

[jiguanglizipao]

The experiments used Hadoop version. I think spark version can generate the same dataset if it works.2023年5月29日 20:19，Yizhe Yuan @.***>写道：

!/bin/bash

DEFAULT_HADOOP_HOME=$HOME/hadoop-2.6.0 #change to your hadoop folder DEFAULT_LDBC_SNB_DATAGEN_HOME=pwd #change to your ldbc_snb_datagen folder

allow overriding configuration from outside via environment variables

i.e. you can do

HADOOP_HOME=/foo/bar LDBC_SNB_DATAGEN_HOME=/baz/quux ./run.sh

instead of changing the contents of this file

HADOOP_HOME=${HADOOP_HOME:-$DEFAULT_HADOOP_HOME} LDBC_SNB_DATAGEN_HOME=${LDBC_SNB_DATAGEN_HOME:-$DEFAULT_LDBC_SNB_DATAGEN_HOME}

export HADOOP_HOME export LDBC_SNB_DATAGEN_HOME export HADOOP_CLIENT_OPTS="-Xmx64G"

echo =============================================================================== echo Running generator with the following parameters: echo ------------------------------------------------------------------------------- echo LDBC_SNB_DATAGEN_HOME: $LDBC_SNB_DATAGEN_HOME echo HADOOP_HOME: $HADOOP_HOME echo HADOOP_CLIENT_OPTS: $HADOOP_CLIENT_OPTS echo ===============================================================================

--- for OSX, add this line ---

zip -d $LDBC_SNB_DATAGEN_HOME/target/ldbc_snb_datagen-0.2.7-jar-with-dependencies.jar META-INF/LICENSE

$HADOOP_HOME/bin/hadoop jar ldbc_snb_datagen-0.2.7-jar-with-dependencies.jar $LDBC_SNB_DATAGEN_HOME/params-foreign-key.ini

rm -f mpersonFactors rm -f .mpersonFactors rm -f mactivityFactors rm -f .mactivityFactors rm -f m0friendList rm -f .m0friendList

The current version used in run.sh is Hadoop. How should I modify it?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>