search

pinterest / orion

Management and automation platform for Stateful Distributed Systems
Apache License 2.0
103 stars 27 forks source link

build and deploy issue #21

Open aymec opened 3 years ago

aymec commented 3 years ago

Hi,

I greatly enjoyed your article Lessons Learned from Running Apache Kafka at Scale at Pinterest. Thank you very much.
That is where I discovered this project, looking for an upgrade over CMAK.
Orion looks very promising, though I met a few issues when building and starting the components. Please let me know if I should split this or report it differently.

I have used the master branch, commit f94201ff9e106eff38c14e6e20c0674fafade584. It builds Orion version 0.0.33.

  1. The build-deployment.sh expects to be run from an Ubuntu 18.04 LTS, from the node debian repo addition for bionic. Only Ubuntu is mentioned in the QuickStart, not the version.
    Replacing bionic with $(grep -oP -m1 "CODENAME=\K(\w+)" /etc/os-release) in the script solves this, and works for debian distros as well.
    With this change, I have successfully built Orion on Ubuntu 20.04 (Focal Fossa), debian stretch and debian buster.
    For buster, a hack is necessary to install openjdk-8-jdk, since it is not available in the official repos for that version.

  2. The script needs to be run as sudo, to be able to add the node repo key, configure the apt sources, etc. but then it also builds the project as sudo, when the call to mvn clean Package is made.
    I am not sure how to solve that cleanly. Maybe adding the sudo where needed in the script ?

  3. The orion-agent debian package built has a dependency to openjdk-8-jdk instead of openjdk-8-jre
    I have not looked in the sources where to change that.

  4. To run the orion server, I have had to modify the run_orion.sh with a different JAVA_HOME value, which for openjdk-8-jre seems incorrect.
    On Ubuntu 20.04, debian stretch and debian buster, the JAVA_HOME is /usr/lib/jvm/java-8-openjdk-amd64.
    Also, the script could check if JAVA_HOME is already set.

  5. The run_orion.sh script needs to be run from /opt/orion-server/ otherwise it reports an error in /var/log/orion/orion-stdout.log: Error: Could not find or load main class com.pinterest.orion.server.OrionServer
    I believe it comes from settings the ORION_HOME parameter as follow ORION_HOME=$(pwd $DIR/..)

  6. After solving that, a new issue comes up when starting the server, the server outputs an exception in /var/log/orion/orion-stdout.log
    Commenting all endpointEnabled in the config file /opt/orion-server/conf/kafka-server.yaml allowed to start the server, but I guess I just lost some functions.

Exception details

```bash io.dropwizard.configuration.ConfigurationParsingException: /opt/orion-server/conf/kafka-server.yaml has an error: * Unrecognized field at: plugins.actionConfigs.[8].endpointEnabled Did you mean?: - endpoint - enabled - configuration - class - key at io.dropwizard.configuration.ConfigurationParsingException$Builder.build(ConfigurationParsingException.java:279) at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:140) at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:89) at io.dropwizard.cli.ConfiguredCommand.parseConfiguration(ConfiguredCommand.java:126) at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:74) at io.dropwizard.cli.Cli.run(Cli.java:78) at io.dropwizard.Application.run(Application.java:93) at com.pinterest.orion.server.OrionServer.main(OrionServer.java:298) Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "endpointEnabled" (class com.pinterest.orion.core.configs.ActionConfig), not marked as ignorable (5 known properties: "class", "enabled", "endpoint", "configuration", "key"]) at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.pinterest.orion.server.config.OrionConf["plugins"]->com.pinterest.orion.server.config.OrionPluginConfig["actionConfigs"]->java.util.ArrayList[8]->com.pinterest.orion.core.configs.ActionConfig["endpointEnabled"]) at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61) at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:823) at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1153) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1589) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1567) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:258) at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:155) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:286) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27) at com.fasterxml.jackson.module.afterburner.deser.SettableObjectMethodProperty.deserializeAndSet(SettableObjectMethodProperty.java:47) at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:159) at com.fasterxml.jackson.module.afterburner.deser.SettableObjectMethodProperty.deserializeAndSet(SettableObjectMethodProperty.java:47) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:252) at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:155) at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3985) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2277) at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:127) ... 6 more ```

  1. Finally, a suggestion about the log folders. Since orion-server is installed in /opt/orion-server and orion-agent is installed in /opt/orion-agent, what about using /var/log/orion-server as the default path for the orion-server instead of /var/log/orion/ ?

This is actually where I stopped for now.

I have a question though about where the server and agents should be installed.
Is it recommended, or not, to install both the server and agent on the same machine ?
Should it be installed on one of the Kafka brokers or should it be installed separately ?

Thanks a lot for the great work.
Please let me know if I should submit pull requests for any of the points mentioned above.

kabochya commented 3 years ago

Hi @aymsite sorry for the late reply, great thanks for providing all the detailed feedback. We will try to address these points soon, but feel free to create PRs as well if you'd like to and I'd be happy to review them.

For your question, in our deployment we have Orion server deployed on a dedicated instance and manage about 2.5k brokers on it. For smaller deployments, it might be ok to deploy the server on a broker, but it will be problematic if that broker needs to be replaced.

Please let us know if there are any further problems. Once again I apologize for the extremely delayed response and appreciate the notes you provided.