Multiple carbon-cache instances don't work

[ravbaba]

I'v been trying to add an extra carbon-cache instance but it looks like this repo supports only one carbon-cache. I added this to carbon.conf file:

[cache:b]
LINE_RECEIVER_PORT = 2103
PICKLE_RECEIVER_PORT = 2104
CACHE_QUERY_PORT = 7102

and this to relay section

DESTINATIONS = 127.0.0.1:2004:a, 127.0.0.1:2104:b

here is my docker-compose.yaml file

version: '2'
services:
  graphite:
    image: graphiteapp/graphite-statsd
    container_name: graphite
    restart: always
    volumes:
      - ./conf:/opt/graphite/conf
      - ./storage:/opt/graphite/storage
    environment:
      RELAY: 1
      GRAPHITE_CARBONLINK_HOSTS: "127.0.0.1:7002:a, 127.0.0.1:7102:b"
    ports:
      - "80:80"
      - "81:81"
      - "2003-2004:2003-2004"
      - "2103-2104:2103-2104"
      - "2023-2024:2023-2024"
      - "8125:8125/udp"
      - "8126:8126"

that what docker-compose logs -f graphite shows:

graphite    | 07/02/2019 10:56:41 :: [clients] Destination is down: 127.0.0.1:2104:b (4/5)
graphite    | 07/02/2019 10:56:41 :: [console] Stopping factory CarbonClientFactory(127.0.0.1:2104:b)
graphite    | 07/02/2019 10:56:41 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2104:b)
graphite    | 07/02/2019 10:56:41 :: [clients] CarbonClientFactory(127.0.0.1:2104:b)::startedConnecting (127.0.0.1:2104)
graphite    | 07/02/2019 10:56:41 :: [console] <twisted.internet.tcp.Connector object at 0x7f4653f5b3c8> will retry in 4 seconds
graphite    | 07/02/2019 10:56:41 :: [clients] CarbonClientFactory(127.0.0.1:2104:b)::clientConnectionFailed (127.0.0.1:2104) Connection was refused by other side: 111: Connection refused.

I think there should be a way to run all carbon-cache instances defined in carbon.conf file

example how to run a specific instance:

python3 /opt/graphite/bin/carbon-cache.py --instance=b start

Sorry if I am wrong and I missed something.

[piotr1212]

You'll have to add a service file to start it somewhere here https://github.com/graphite-project/docker-graphite-statsd/tree/master/conf/etc/service/carbon

[ravbaba]

@piotr1212 I tried this approach as well but I get the error when I try run /etc/service/carbon/run

/etc/service # /etc/service/carbon/run
Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-cache.py", line 28, in <module>
    from carbon.util import run_twistd_plugin  # noqa
  File "/opt/graphite/lib/carbon/util.py", line 15, in <module>
    from twisted.python.util import initgroups
ModuleNotFoundError: No module named 'twisted.python'
Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-cache.py", line 28, in <module>
    from carbon.util import run_twistd_plugin  # noqa
  File "/opt/graphite/lib/carbon/util.py", line 15, in <module>
    from twisted.python.util import initgroups
ModuleNotFoundError: No module named 'twisted.python'

I get the above error without changing anything.

[deniszh]

Adding it to the same file https://github.com/graphite-project/docker-graphite-statsd/blob/master/conf/etc/service/carbon/run will not work. You need to create separate one, e.g. etc/service/carbon-b/run That's why only one carbon instance is supported currently - supporting a variable amount of instances will complicate setup significantly.

[ravbaba]

@deniszh, yes, creating etc/service/carbon-b/run and adding

#!/bin/sh

rm -f /opt/graphite/storage/carbon-cache-*
exec python3 /opt/graphite/bin/carbon-cache.py --instance=b start --debug 2>&1 | tee -a /var/log/carbon.log

fixes my problem.

Indeed supporting multiple instances will complicate a setup but it would be nice to have the feature.

[ravbaba]

@piotr1212 I tried this approach as well but I get the error when I try run /etc/service/carbon/run

/etc/service # /etc/service/carbon/run
Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-cache.py", line 28, in <module>
    from carbon.util import run_twistd_plugin  # noqa
  File "/opt/graphite/lib/carbon/util.py", line 15, in <module>
    from twisted.python.util import initgroups
ModuleNotFoundError: No module named 'twisted.python'
Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-cache.py", line 28, in <module>
    from carbon.util import run_twistd_plugin  # noqa
  File "/opt/graphite/lib/carbon/util.py", line 15, in <module>
    from twisted.python.util import initgroups
ModuleNotFoundError: No module named 'twisted.python'

I get the above error without changing anything.

Please ignore the comment I didn't notice that virtualenv is used. running below commands inside container fixes the ModuleNotFoundError issue:

source /opt/graphite/bin/activate
/etc/service/carbon/run

[ravbaba]

Maybe someone with similar issue find it useful, I decided to run carbon cache instances with the following command:

docker-compose exec graphite sh -c "source /opt/graphite/bin/activate && python3 /opt/graphite/bin/carbon-cache.py --instance=b start"

you need to remember to run it again if you restart your container, in my case I do it with an ansbile task.

[hamelg]

Now, you can use a startup script to do the job.

To configure 2 instances.

Put this script in the file /graphite/conf/run_once/mycarbon-custom.

#!/bin/sh

for i in a b ; do
    S=/etc/service/carbon-$i
    [ -d $S ] && continue
    mkdir -p $S/log
    cp /etc/service/carbon/log/run $S/log
    sed "s/start/--instance=$i start/" /etc/service/carbon/run > $S/run
    chmod +x $S/run
done

Adapt the /graphite/conf/graphite/carbon.conf setup :

...
[cache:a]
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_PORT = 2004
CACHE_QUERY_PORT = 7002
[cache:b]
LINE_RECEIVER_PORT = 2005
PICKLE_RECEIVER_PORT = 2006
CACHE_QUERY_PORT = 7003
...

And at end, startup up your container with the adequate options.

docker run -d\
...
 -p 2013-2016:2003-2006\
 -v /graphite/conf/graphite:/opt/graphite/conf\
 -v /graphite/conf/run_once:/etc/run_once\
 -e GRAPHITE_CARBONLINK_HOSTS=127.0.0.1:7002:a,127.0.0.1:7003:b\
 -e CARBON_DISABLED=1\
...
 graphiteapp/graphite-statsd

The parameter CARBON_DISABLED=1 tells to runit to disable the default instance.