search

marvin-ai / marvin-engine-executor

Marvin AI has been accepted into the Apache Foundation and is now available at https://github.com/apache/incubator-marvin
http://www.marvin-ai.org
Apache License 2.0
45 stars 10 forks source link

Predictors reload action are locking the state machine in the "reloading" state forever #36

Closed takabayashi closed 6 years ago

takabayashi commented 6 years ago

/usr/lib/jvm/java-8-oracle/bin/java -DmarvinConfig.engineHome=/home/taka/marvin/engines/iris-species-engine -javaagent:/home/taka/development/intelliJ-idea/lib/idea_rt.jar=35995:/home/taka/development/intelliJ-idea/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/deploy.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-oracle/jre/lib/javaws.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfxswt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-oracle/jre/lib/plugin.jar:/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/home/taka/marvin/marvin-engine-executor/target/scala-2.12/classes:/home/taka/.ivy2/cache/aopalliance/aopalliance/jars/aopalliance-1.0.jar:/home/taka/.ivy2/cache/xmlenc/xmlenc/jars/xmlenc-0.52.jar:/home/taka/.ivy2/cache/xml-apis/xml-apis/jars/xml-apis-1.3.04.jar:/home/taka/.ivy2/cache/xerces/xercesImpl/jars/xercesImpl-2.9.1.jar:/home/taka/.ivy2/cache/org.xerial.snappy/snappy-java/bundles/snappy-java-1.0.4.1.jar:/home/taka/.ivy2/cache/org.tukaani/xz/jars/xz-1.0.jar:/home/taka/.ivy2/cache/org.sonatype.sisu.inject/cglib/jars/cglib-2.2.1-v20090111.jar:/home/taka/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:/home/taka/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.10.jar:/home/taka/.ivy2/cache/org.scala-lang.modules/scala-parser-combinators_2.12/bundles/scala-parser-combinators_2.12-1.0.4.jar:/home/taka/.ivy2/cache/org.scala-lang.modules/scala-java8-compat_2.12/bundles/scala-java8-compat_2.12-0.8.0.jar:/home/taka/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.3.jar:/home/taka/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.3.jar:/home/taka/.ivy2/cache/org.reactivestreams/reactive-streams/jars/reactive-streams-1.0.0.jar:/home/taka/.ivy2/cache/org.mortbay.jetty/jetty-util/jars/jetty-util-6.1.26.jar:/home/taka/.ivy2/cache/org.mortbay.jetty/jetty-sslengine/jars/jetty-sslengine-6.1.26.jar:/home/taka/.ivy2/cache/org.fusesource.leveldbjni/leveldbjni-all/bundles/leveldbjni-all-1.8.jar:/home/taka/.ivy2/cache/org.codehaus.jettison/jettison/bundles/jettison-1.1.jar:/home/taka/.ivy2/cache/org.codehaus.jackson/jackson-xc/jars/jackson-xc-1.9.13.jar:/home/taka/.ivy2/cache/org.codehaus.jackson/jackson-mapper-asl/jars/jackson-mapper-asl-1.9.13.jar:/home/taka/.ivy2/cache/org.codehaus.jackson/jackson-jaxrs/jars/jackson-jaxrs-1.9.13.jar:/home/taka/.ivy2/cache/org.codehaus.jackson/jackson-core-asl/jars/jackson-core-asl-1.9.13.jar:/home/taka/.ivy2/cache/org.clapper/grizzled-slf4j_2.12/jars/grizzled-slf4j_2.12-1.3.0.jar:/home/taka/.ivy2/cache/org.apache.zookeeper/zookeeper/jars/zookeeper-3.4.6.jar:/home/taka/.ivy2/cache/org.apache.htrace/htrace-core/jars/htrace-core-3.1.0-incubating.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-yarn-server-common/jars/hadoop-yarn-server-common-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-yarn-client/jars/hadoop-yarn-client-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-shuffle/jars/hadoop-mapreduce-client-shuffle-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-jobclient/jars/hadoop-mapreduce-client-jobclient-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-core/jars/hadoop-mapreduce-client-core-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-common/jars/hadoop-mapreduce-client-common-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/jars/hadoop-mapreduce-client-app-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/jars/hadoop-hdfs-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-common/jars/hadoop-common-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-client/jars/hadoop-client-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-auth/jars/hadoop-auth-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.hadoop/hadoop-annotations/jars/hadoop-annotations-2.7.4.jar:/home/taka/.ivy2/cache/org.apache.directory.server/apacheds-kerberos-codec/bundles/apacheds-kerberos-codec-2.0.0-M15.jar:/home/taka/.ivy2/cache/org.apache.directory.server/apacheds-i18n/bundles/apacheds-i18n-2.0.0-M15.jar:/home/taka/.ivy2/cache/org.apache.directory.api/api-util/bundles/api-util-1.0.0-M20.jar:/home/taka/.ivy2/cache/org.apache.directory.api/api-asn1-api/bundles/api-asn1-api-1.0.0-M20.jar:/home/taka/.ivy2/cache/org.apache.curator/curator-recipes/bundles/curator-recipes-2.7.1.jar:/home/taka/.ivy2/cache/org.apache.curator/curator-framework/bundles/curator-framework-2.7.1.jar:/home/taka/.ivy2/cache/org.apache.curator/curator-client/bundles/curator-client-2.7.1.jar:/home/taka/.ivy2/cache/org.apache.commons/commons-math3/jars/commons-math3-3.1.1.jar:/home/taka/.ivy2/cache/org.apache.commons/commons-compress/jars/commons-compress-1.4.1.jar:/home/taka/.ivy2/cache/org.apache.avro/avro/jars/avro-1.7.4.jar:/home/taka/.ivy2/cache/log4j/log4j/bundles/log4j-1.2.17.jar:/home/taka/.ivy2/cache/junit/junit/jars/junit-3.8.1.jar:/home/taka/.ivy2/cache/javax.xml.stream/stax-api/jars/stax-api-1.0-2.jar:/home/taka/.ivy2/cache/javax.xml.bind/jaxb-api/jars/jaxb-api-2.2.2.jar:/home/taka/.ivy2/cache/javax.servlet.jsp/jsp-api/jars/jsp-api-2.1.jar:/home/taka/.ivy2/cache/javax.servlet/servlet-api/jars/servlet-api-2.5.jar:/home/taka/.ivy2/cache/javax.inject/javax.inject/jars/javax.inject-1.jar:/home/taka/.ivy2/cache/javax.activation/activation/jars/activation-1.1.jar:/home/taka/.ivy2/cache/io.spray/spray-json_2.12/bundles/spray-json_2.12-1.3.3.jar:/home/taka/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-resolver/jars/netty-resolver-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-handler-proxy/jars/netty-handler-proxy-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-codec-socks/jars/netty-codec-socks-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-codec-http2/jars/netty-codec-http2-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-codec-http/jars/netty-codec-http-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.1.12.Final.jar:/home/taka/.ivy2/cache/io.netty/netty-all/jars/netty-all-4.0.23.Final.jar:/home/taka/.ivy2/cache/io.netty/netty/bundles/netty-3.7.0.Final.jar:/home/taka/.ivy2/cache/io.grpc/grpc-stub/jars/grpc-stub-1.5.0.jar:/home/taka/.ivy2/cache/io.grpc/grpc-protobuf-lite/jars/grpc-protobuf-lite-1.5.0.jar:/home/taka/.ivy2/cache/io.grpc/grpc-protobuf/jars/grpc-protobuf-1.5.0.jar:/home/taka/.ivy2/cache/io.grpc/grpc-netty/jars/grpc-netty-1.5.0.jar:/home/taka/.ivy2/cache/io.grpc/grpc-core/jars/grpc-core-1.5.0.jar:/home/taka/.ivy2/cache/io.grpc/grpc-context/jars/grpc-context-1.5.0.jar:/home/taka/.ivy2/cache/commons-net/commons-net/jars/commons-net-3.1.jar:/home/taka/.ivy2/cache/commons-lang/commons-lang/jars/commons-lang-2.6.jar:/home/taka/.ivy2/cache/commons-io/commons-io/jars/commons-io-2.4.jar:/home/taka/.ivy2/cache/commons-httpclient/commons-httpclient/jars/commons-httpclient-3.1.jar:/home/taka/.ivy2/cache/commons-configuration/commons-configuration/jars/commons-configuration-1.6.jar:/home/taka/.ivy2/cache/commons-collections/commons-collections/jars/commons-collections-3.2.2.jar:/home/taka/.ivy2/cache/commons-cli/commons-cli/jars/commons-cli-1.2.jar:/home/taka/.ivy2/cache/commons-beanutils/commons-beanutils-core/jars/commons-beanutils-core-1.8.0.jar:/home/taka/.ivy2/cache/com.typesafe.akka/akka-stream_2.12/jars/akka-stream_2.12-2.4.19.jar:/home/taka/.ivy2/cache/com.typesafe.akka/akka-parsing_2.12/jars/akka-parsing_2.12-10.0.10.jar:/home/taka/.ivy2/cache/com.typesafe.akka/akka-http_2.12/jars/akka-http_2.12-10.0.10.jar:/home/taka/.ivy2/cache/com.typesafe.akka/akka-http-spray-json_2.12/jars/akka-http-spray-json_2.12-10.0.10.jar:/home/taka/.ivy2/cache/com.typesafe.akka/akka-http-core_2.12/jars/akka-http-core_2.12-10.0.10.jar:/home/taka/.ivy2/cache/com.typesafe.akka/akka-actor_2.12/jars/akka-actor_2.12-2.4.19.jar:/home/taka/.ivy2/cache/com.typesafe/ssl-config-core_2.12/bundles/ssl-config-core_2.12-0.2.1.jar:/home/taka/.ivy2/cache/com.typesafe/config/bundles/config-1.3.0.jar:/home/taka/.ivy2/cache/com.trueaccord.scalapb/scalapb-runtime_2.12/jars/scalapb-runtime_2.12-0.6.2.jar:/home/taka/.ivy2/cache/com.trueaccord.scalapb/scalapb-runtime-grpc_2.12/jars/scalapb-runtime-grpc_2.12-0.6.2.jar:/home/taka/.ivy2/cache/com.trueaccord.lenses/lenses_2.12/jars/lenses_2.12-0.4.12.jar:/home/taka/.ivy2/cache/com.thoughtworks.paranamer/paranamer/bundles/paranamer-2.8.jar:/home/taka/.ivy2/cache/com.sun.xml.bind/jaxb-impl/jars/jaxb-impl-2.2.3-1.jar:/home/taka/.ivy2/cache/com.sun.jersey.contribs/jersey-guice/jars/jersey-guice-1.9.jar:/home/taka/.ivy2/cache/com.sun.jersey/jersey-server/bundles/jersey-server-1.9.jar:/home/taka/.ivy2/cache/com.sun.jersey/jersey-json/bundles/jersey-json-1.9.jar:/home/taka/.ivy2/cache/com.sun.jersey/jersey-core/bundles/jersey-core-1.9.jar:/home/taka/.ivy2/cache/com.sun.jersey/jersey-client/bundles/jersey-client-1.9.jar:/home/taka/.ivy2/cache/com.lihaoyi/sourcecode_2.12/jars/sourcecode_2.12-0.1.3.jar:/home/taka/.ivy2/cache/com.lihaoyi/fastparse_2.12/jars/fastparse_2.12-0.4.3.jar:/home/taka/.ivy2/cache/com.lihaoyi/fastparse-utils_2.12/jars/fastparse-utils_2.12-0.4.3.jar:/home/taka/.ivy2/cache/com.google.protobuf/protobuf-java-util/bundles/protobuf-java-util-3.3.1.jar:/home/taka/.ivy2/cache/com.google.protobuf/protobuf-java/bundles/protobuf-java-3.3.1.jar:/home/taka/.ivy2/cache/com.google.instrumentation/instrumentation-api/jars/instrumentation-api-0.4.3.jar:/home/taka/.ivy2/cache/com.google.inject/guice/jars/guice-3.0.jar:/home/taka/.ivy2/cache/com.google.guava/guava/bundles/guava-19.0.jar:/home/taka/.ivy2/cache/com.google.errorprone/error_prone_annotations/jars/error_prone_annotations-2.0.19.jar:/home/taka/.ivy2/cache/com.google.code.gson/gson/jars/gson-2.7.jar:/home/taka/.ivy2/cache/com.google.code.findbugs/jsr305/jars/jsr305-3.0.0.jar:/home/taka/.ivy2/cache/com.google.api.grpc/proto-google-common-protos/jars/proto-google-common-protos-0.1.9.jar:/home/taka/.ivy2/cache/com.fasterxml.jackson.module/jackson-module-scala_2.12/bundles/jackson-module-scala_2.12-2.9.0.jar:/home/taka/.ivy2/cache/com.fasterxml.jackson.module/jackson-module-paranamer/bundles/jackson-module-paranamer-2.9.0.jar:/home/taka/.ivy2/cache/com.fasterxml.jackson.core/jackson-databind/bundles/jackson-databind-2.9.0.jar:/home/taka/.ivy2/cache/com.fasterxml.jackson.core/jackson-core/bundles/jackson-core-2.9.0.jar:/home/taka/.ivy2/cache/com.fasterxml.jackson.core/jackson-annotations/bundles/jackson-annotations-2.9.0.jar:/home/taka/.ivy2/cache/asm/asm/jars/asm-3.1.jar:/home/taka/.ivy2/cache/io.jvm.uuid/scala-uuid_2.12/jars/scala-uuid_2.12-0.2.3.jar:/home/taka/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.232.jar:/home/taka/.ivy2/cache/com.amazonaws/aws-java-sdk-kms/jars/aws-java-sdk-kms-1.11.232.jar:/home/taka/.ivy2/cache/com.amazonaws/aws-java-sdk-s3/jars/aws-java-sdk-s3-1.11.232.jar:/home/taka/.ivy2/cache/com.amazonaws/jmespath-java/jars/jmespath-java-1.11.232.jar:/home/taka/.ivy2/cache/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/bundles/jackson-dataformat-cbor-2.6.7.jar:/home/taka/.ivy2/cache/commons-codec/commons-codec/jars/commons-codec-1.9.jar:/home/taka/.ivy2/cache/joda-time/joda-time/jars/joda-time-2.8.1.jar:/home/taka/.ivy2/cache/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.2.jar:/home/taka/.ivy2/cache/org.apache.httpcomponents/httpcore/jars/httpcore-4.4.4.jar:/home/taka/.ivy2/cache/software.amazon.ion/ion-java/bundles/ion-java-1.0.2.jar:/home/taka/.ivy2/cache/commons-beanutils/commons-beanutils/jars/commons-beanutils-1.9.2.jar:/home/taka/.ivy2/cache/commons-digester/commons-digester/jars/commons-digester-1.8.1.jar:/home/taka/.ivy2/cache/commons-logging/commons-logging/jars/commons-logging-1.2.jar:/home/taka/.ivy2/cache/commons-validator/commons-validator/jars/commons-validator-1.5.1.jar:/home/taka/.ivy2/cache/org.everit.json/org.everit.json.schema/bundles/org.everit.json.schema-1.5.1.jar:/home/taka/.ivy2/cache/org.json/json/bundles/json-20160810.jar org.marvin.executor.api.GenericHttpAPI [INFO] [12/05/2017 19:50:50.165] [MarvinExecutorSystem-akka.actor.default-dispatcher-2] [akka://MarvinExecutorSystem/user/acquisitorActor/batchActionProxy] org.marvin.executor.proxies.BatchActionProxy actor initialized... [INFO] [12/05/2017 19:50:50.165] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/acquisitorActor/artifactSaver] org.marvin.manager.ArtifactFSSaver actor initialized... [INFO] [12/05/2017 19:50:50.171] [MarvinExecutorSystem-akka.actor.default-dispatcher-5] [akka://MarvinExecutorSystem/user/tpreparatorActor/batchActionProxy] org.marvin.executor.proxies.BatchActionProxy actor initialized... [INFO] [12/05/2017 19:50:50.174] [MarvinExecutorSystem-akka.actor.default-dispatcher-8] [akka://MarvinExecutorSystem/user/trainerActor/batchActionProxy] org.marvin.executor.proxies.BatchActionProxy actor initialized... [INFO] [12/05/2017 19:50:50.176] [MarvinExecutorSystem-akka.actor.default-dispatcher-6] [akka://MarvinExecutorSystem/user/tpreparatorActor/artifactSaver] org.marvin.manager.ArtifactFSSaver actor initialized... [INFO] [12/05/2017 19:50:50.176] [MarvinExecutorSystem-akka.actor.default-dispatcher-6] [akka://MarvinExecutorSystem/user/trainerActor/artifactSaver] org.marvin.manager.ArtifactFSSaver actor initialized... [INFO] [12/05/2017 19:50:50.177] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/pipelineActor/artifactSaver] org.marvin.manager.ArtifactFSSaver actor initialized... [INFO] [12/05/2017 19:50:50.184] [MarvinExecutorSystem-akka.actor.default-dispatcher-9] [akka://MarvinExecutorSystem/user/evaluatorActor/batchActionProxy] org.marvin.executor.proxies.BatchActionProxy actor initialized... [INFO] [12/05/2017 19:50:50.184] [MarvinExecutorSystem-akka.actor.default-dispatcher-10] [akka://MarvinExecutorSystem/user/evaluatorActor/artifactSaver] org.marvin.manager.ArtifactFSSaver actor initialized... [INFO] [12/05/2017 19:50:50.201] [MarvinExecutorSystem-akka.actor.default-dispatcher-11] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] org.marvin.manager.ArtifactFSSaver actor initialized... [INFO] [12/05/2017 19:50:50.202] [MarvinExecutorSystem-akka.actor.default-dispatcher-13] [akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy] org.marvin.executor.proxies.OnlineActionProxy actor initialized... [INFO] [12/05/2017 19:50:50.208] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor] Starting to process reload [no save] to predictor. Protocol: []. [INFO] [12/05/2017 19:50:50.477] [MarvinExecutorSystem-akka.actor.default-dispatcher-13] [akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy] Start the reload remote procedure to predictor. Protocol [] [INFO] [12/05/2017 19:50:50.884] [MarvinExecutorSystem-akka.actor.default-dispatcher-13] [akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy] Reload remote procedure to predictor Done with [Reloaded]. Protocol [] [INFO] [12/05/2017 19:50:50.886] [MarvinExecutorSystem-akka.actor.default-dispatcher-13] [akka://MarvinExecutorSystem/user/predictorFSM] Reloaded the model with success. [INFO] [12/05/2017 19:50:51.302] [MarvinExecutorSystem-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(MarvinExecutorSystem)] Server online at http://localhost:8000/ [INFO] [12/05/2017 19:51:36.277] [MarvinExecutorSystem-akka.actor.default-dispatcher-4] [akka://MarvinExecutorSystem/user/predictorFSM] Received a message to reload the model. [INFO] [12/05/2017 19:51:36.277] [MarvinExecutorSystem-akka.actor.default-dispatcher-4] [akka://MarvinExecutorSystem/user/predictorActor] Starting to process reload to predictor. Protocol: [pipeline_235af8ce-429a-46af-952e-3229da1fcc1b]. [INFO] [12/05/2017 19:51:36.282] [MarvinExecutorSystem-akka.actor.default-dispatcher-10] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] Receive message and starting to working... [INFO] [12/05/2017 19:51:36.283] [MarvinExecutorSystem-akka.actor.default-dispatcher-10] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] Copying files from /tmp/marvin/iris_species/v0.0.1/model/pipeline_235af8ce-429a-46af-952e-3229da1fcc1b to /home/taka/marvin/data/.artifacts/iris_species/model [INFO] [12/05/2017 19:51:36.284] [MarvinExecutorSystem-akka.actor.default-dispatcher-10] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] File /home/taka/marvin/data/.artifacts/iris_species/model saved! [INFO] [12/05/2017 19:51:36.287] [MarvinExecutorSystem-akka.actor.default-dispatcher-10] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] Receive message and starting to working... [INFO] [12/05/2017 19:51:36.287] [MarvinExecutorSystem-akka.actor.default-dispatcher-10] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] Copying files from /tmp/marvin/iris_species/v0.0.1/metrics/pipeline_235af8ce-429a-46af-952e-3229da1fcc1b to /home/taka/marvin/data/.artifacts/iris_species/metrics [INFO] [12/05/2017 19:51:36.288] [MarvinExecutorSystem-akka.actor.default-dispatcher-10] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] File /home/taka/marvin/data/.artifacts/iris_species/metrics saved! [INFO] [12/05/2017 19:51:36.292] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor] Reload to predictor completed [ListBuffer(Done, Done)] ! Protocol: [pipeline_235af8ce-429a-46af-952e-3229da1fcc1b] [INFO] [12/05/2017 19:51:36.292] [MarvinExecutorSystem-akka.actor.default-dispatcher-11] [akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy] Start the reload remote procedure to predictor. Protocol [pipeline_235af8ce-429a-46af-952e-3229da1fcc1b] [INFO] [12/05/2017 19:51:36.296] [MarvinExecutorSystem-akka.actor.default-dispatcher-11] [akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy] Reload remote procedure to predictor Done with [Reloaded]. Protocol [pipeline_235af8ce-429a-46af-952e-3229da1fcc1b] [INFO] [12/05/2017 19:51:36.296] [MarvinExecutorSystem-akka.actor.default-dispatcher-11] [akka://MarvinExecutorSystem/user/predictorFSM] Reloaded the model with success. [INFO] [12/05/2017 19:52:18.422] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor] Starting to process reload to predictor. Protocol: [pipeline_235af8ce-429a-46af-952e-3229da1fcc1b]. [INFO] [12/05/2017 19:52:18.422] [MarvinExecutorSystem-akka.actor.default-dispatcher-10] [akka://MarvinExecutorSystem/user/predictorFSM] Received a message to reload the model. [INFO] [12/05/2017 19:52:18.424] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] Receive message and starting to working... [INFO] [12/05/2017 19:52:18.424] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] Copying files from /tmp/marvin/iris_species/v0.0.1/model/pipeline_235af8ce-429a-46af-952e-3229da1fcc1b to /home/taka/marvin/data/.artifacts/iris_species/model [INFO] [12/05/2017 19:52:18.424] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] File /home/taka/marvin/data/.artifacts/iris_species/model saved! [INFO] [12/05/2017 19:52:18.425] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] Receive message and starting to working... [INFO] [12/05/2017 19:52:18.425] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] Copying files from /tmp/marvin/iris_species/v0.0.1/metrics/pipeline_235af8ce-429a-46af-952e-3229da1fcc1b to /home/taka/marvin/data/.artifacts/iris_species/metrics [INFO] [12/05/2017 19:52:18.426] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor/artifactSaver] File /home/taka/marvin/data/.artifacts/iris_species/metrics saved! [INFO] [12/05/2017 19:52:18.426] [MarvinExecutorSystem-akka.actor.default-dispatcher-20] [akka://MarvinExecutorSystem/user/predictorActor] Reload to predictor completed [ListBuffer(Done, Done)] ! Protocol: [pipeline_235af8ce-429a-46af-952e-3229da1fcc1b] [INFO] [12/05/2017 19:52:18.426] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy] Start the reload remote procedure to predictor. Protocol [pipeline_235af8ce-429a-46af-952e-3229da1fcc1b] [INFO] [12/05/2017 19:52:18.435] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy] Reload remote procedure to predictor Done with [Reloaded]. Protocol [pipeline_235af8ce-429a-46af-952e-3229da1fcc1b] [INFO] [12/05/2017 19:52:18.435] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/deadLetters] Message [org.marvin.executor.statemachine.Reloaded] from Actor[akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy#1243896337] to Actor[akka://MarvinExecutorSystem/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [12/05/2017 19:52:32.482] [MarvinExecutorSystem-akka.actor.default-dispatcher-3] [akka://MarvinExecutorSystem/user/predictorFSM] Engine is reloading, not possible to perform event Reload(pipeline_235af8ce-429a-46af-952e-3229da1fcc1b) in state Reloading/ToReload(pipeline_235af8ce-429a-46af-952e-3229da1fcc1b) [INFO] [12/05/2017 19:52:32.483] [MarvinExecutorSystem-akka.actor.default-dispatcher-16] [akka://MarvinExecutorSystem/deadLetters] Message [akka.actor.Status$Failure] from Actor[akka://MarvinExecutorSystem/user/predictorFSM#397137975] to Actor[akka://MarvinExecutorSystem/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

lucasbm88 commented 6 years ago

The problem is being caused because the OnlineAction is using an actor method (forward) inside a future. By that reason akka does not have the actor context to get the sender, and the sender is being set to DeadLetters.

MarvinExecutorSystem/deadLetters] Message [org.marvin.executor.statemachine.Reloaded] from Actor[│
akka://MarvinExecutorSystem/user/predictorActor/onlineActionProxy#497269889] to Actor[akka://Marv│
inExecutorSystem/deadLetters] was not delivered. [1] dead letters encountered.

References: https://stackoverflow.com/a/43396848

I'm still studying the best way to fix this.