search

divolte / divolte-collector

Divolte Collector
https://divolte.io/
Apache License 2.0
282 stars 77 forks source link

How to update the divolte-collector.conf and reload the service #187

Closed ddehghan closed 6 years ago

ddehghan commented 6 years ago

Thanks for this great service.

We are running Divolte in a docker container and it would speed of development and testing of the config file if we can just copy the file into the container and have it be reloaded automatically.

Reading the documentations it seems like this is done for the geo2ip database file. Is that true?

Thanks again.

Also we have a bug to report on Divolte 0.6:

When we start the service in the container and run it again we get this error:

2018-01-26 03:28:39.509Z [main] INFO  [AppInfoParser]: Kafka version : 0.10.2.1
2018-01-26 03:28:39.509Z [main] INFO  [AppInfoParser]: Kafka commitId : e89bffd6b2eff799
2018-01-26 03:28:39.511Z [main] WARN  [AppInfoParser]: Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=divolte.collector
    at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:58)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:204)
    at io.divolte.server.kafka.KafkaFlushingPool.<init>(KafkaFlushingPool.java:45)
    at io.divolte.server.Server.lambda$new$6(Server.java:92)
    at com.google.common.collect.CollectCollectors.lambda$toImmutableMap$1(CollectCollectors.java:62)
    at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at io.divolte.server.Server.<init>(Server.java:89)
    at io.divolte.server.Server.<init>(Server.java:60)
    at io.divolte.server.Server.main(Server.java:199)

It seems that when kafka-producer was initialized then stopping the service does not clean up something so restarting it will cause this error.

The service actually works afterward and sends the data to kafka successfully.

friso commented 6 years ago

Dynamic config reload is tricky to implement in the way Divolte is setup internally. We had some plans previously to perhaps make it work, but I have to say that these days most deployments are immutable and as such a config reload without building a new container doesn't typically happen. It would probably be a better path to create separate tools for testing configurations and mappings.

As for the Kafka exception, I haven't seen that before. Perhaps it's a good idea to take it on the mailing list and then also share the relevant parts of your configuration. It seems more specific to how Kafka is setup for you than to anything that Divolte does.

asnare commented 6 years ago

As @friso noted this isn't something we're planning on working on for now.

The GeoIP database is a special-case: this is routinely updated in the background while Divolte is running so we support that. (It was fairly easy to implement and the impact of re-loading it is very narrow in scope.)