search

byzer-org / byzer-build

Apache License 2.0
7 stars 9 forks source link

Running byzer-lang container on local mode #38

Closed chncaesar closed 2 years ago

chncaesar commented 2 years ago

Please see README.md for usage Test if DRIVER_MEMORY and MASTER are effective

## Start container 
docker run -d --name lang -e DRIVER_MEMORY=8g -e MASTER=local[*] -p9003:9003 byzer/byzer-lang:3.1.1-2.2.2
## Check container memory setting
docker exec -it lang bash
ps -eaf | grep -i 8g

Result:

root        135      7 99 02:51 ?        00:00:30 /work/jdk1.8.0_151/bin/java -cp /home/deploy/kolo-lang/libs/ansj_seg-5.1.6.jar:/home/deploy/kolo-lang/libs/azure-blob_3.2-1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-assert-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-excel-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-ext-ets-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-mllib-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-shell-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/nlp-lang-1.7.8.jar:/home/deploy/kolo-lang/libs/streamingpro-mlsql-spark_3.0_2.12-2.2.2.jar:/work/spark-3.1.1-bin-hadoop3.2/conf/:/work/spark-3.1.1-bin-hadoop3.2/jars/*:/etc/hadoop -Xmx8g org.apache.spark.deploy.SparkSubmit --master local[*] --deploy-mode client --conf spark.driver.memory=8g --conf spark.kryoserializer.buffer.max=1024m --conf spark.sql.hive.thriftServer.singleSession=true --conf spark.executor.extraClassPath=/home/deploy/kolo-lang/libs/ansj_seg-5.1.6.jar:/home/deploy/kolo-lang/libs/azure-blob_3.2-1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-assert-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-excel-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-ext-ets-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-mllib-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-shell-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/nlp-lang-1.7.8.jar:/home/deploy/kolo-lang/libs/streamingpro-mlsql-spark_3.0_2.12-2.2.2.jar --conf spark.scheduler.mode=FAIR --conf spark.kryoserializer.buffer=256k --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.driver.extraClassPath=/home/deploy/kolo-lang/libs/ansj_seg-5.1.6.jar:/home/deploy/kolo-lang/libs/azure-blob_3.2-1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-assert-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-excel-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-ext-ets-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-mllib-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/mlsql-shell-3.0_2.12-0.1.0-SNAPSHOT.jar:/home/deploy/kolo-lang/libs/nlp-lang-1.7.8.jar:/home/deploy/kolo-lang/libs/streamingpro-mlsql-spark_3.0_2.12-2.2.2.jar --class streaming.core.StreamingApp --jars /home/deploy/kolo-lang/libs/ansj_seg-5.1.6.jar,/home/deploy/kolo-lang/libs/azure-blob_3.2-1.0-SNAPSHOT.jar,/home/deploy/kolo-lang/libs/mlsql-assert-3.0_2.12-0.1.0-SNAPSHOT.jar,/home/deploy/kolo-lang/libs/mlsql-excel-3.0_2.12-0.1.0-SNAPSHOT.jar,/home/deploy/kolo-lang/libs/mlsql-ext-ets-3.0_2.12-0.1.0-SNAPSHOT.jar,/home/deploy/kolo-lang/libs/mlsql-mllib-3.0_2.12-0.1.0-SNAPSHOT.jar,/home/deploy/kolo-lang/libs/mlsql-shell-3.0_2.12-0.1.0-SNAPSHOT.jar,/home/deploy/kolo-lang/libs/nlp-lang-1.7.8.jar,/home/deploy/kolo-lang/libs/streamingpro-mlsql-spark_3.0_2.12-2.2.2.jar --name mlsql /home/deploy/kolo-lang/libs/streamingpro-mlsql-spark_3.0_2.12-2.2.2.jar -streaming.name mlsql -streaming.platform spark -streaming.rest true -streaming.driver.port 9003 -streaming.spark.service true -streaming.thrift false -streaming.enableHiveSupport true -streaming.datalake.path /mlsql/_delta -streaming.plugin.clzznames tech.mlsql.plugins.ds.MLSQLExcelApp,tech.mlsql.plugins.assert.app.MLSQLAssert,tech.mlsql.plugins.shell.app.MLSQLShell,tech.mlsql.plugins.ext.ets.app.MLSQLETApp