no log driver named 'sumologic' is registered

[rghunter]

Hello

I'm trying to get the logging driver installed on amzn-ami-2017.03.f-amazon-ecs-optimized. I followed the instructions (I ran docker plugin install store/sumologic/docker-logging-driver:1.0.2 --alias sumologic). Running docker plugins ls shows:

ID                  NAME                DESCRIPTION                 ENABLED
7d401749617c        sumologic:latest    Sumo Logic logging driver   true

(note NAME is sumologic:latest, not sumologic)

I can run a docker plugin inspect sumologic and i get back info.

However, when I try to actually use the driver... for example docker run --log-driver=sumologic --log-opt sumo-url=<redacted> chentex/random-logger

I get: docker: Error response from daemon: logger: no log driver named 'sumologic' is registered.

Same error if I try --log-driver=sumologic:latest too. Am I missing a step?

[dwkerwin]

Seeing the same exact thing, using amzn-ami-2017.03.g-amazon-ecs-optimized. With this version of the ECS AMI it has Docker version 17.03.2-ce, build 7392c3b/17.03.2-ce, and I believe it is this version of the Docker engine which is incompatible with this Sumologic logging driver. I also tested on another EC2 instance running Centos7 and the latest Docker 17.09.0-ce and it works fine there however ECS insists on using the ECS optimized AMI so that is not a solution. I suppose at some point Amazon will release a new ECS AMI with a newer Docker engine and this will work fine, but in the meantime it would be nice to have this plugin to be compatible with Docker 17.03. In the short term I have regrettably put in place the awslogs driver (cloudwatch logs).

[mhumeSF]

I am running ubuntu 16.04 with docker-17.09.0-ce and getting the same error when I set the log_driver in /etc/docker/daemon.json

Failed to set log opts: logger: no log driver named 'sumologic' is registered.

But if I run it as a flag it works

docker run --log-driver=sumologic --log-opt sumo-url=<redacted> alpine:latest echo Hello

[bin3377]

Following the ECS agent versions list, I believe the latest AMI (2017.03.g) should include the agent support sumologic logging driver

And if you want to use other linux distributions, please follow the section "To install the Amazon ECS container agent on a non-Amazon Linux EC2 instance" in this article with latest ECS agent build from https://github.com/aws/amazon-ecs-agent

[mhumeSF]

https://github.com/mhumeSF/docker-ce-ami

Posting an example vagrant project to show how setting sumologic as the logging driver with daemon.json is erroring.

These is daemon.json:

{
  "log-driver": "sumologic",
  "log-opts": {
    "sumo-url": "https://endpoint.sumologic.com/receiver/v1/http/token"
  }
}

I have an ubuntu instance running the ecs-agent and sumologic logging driver installed. It is working when no daemon.json file exists and launching task-definitions with sumologic set as the logging driver. It's only when setting the driver via the daemon.json.

Oct 24 22:19:53 ubuntu-xenial dockerd[11169]: Failed to set log opts: logger: no log driver named 'sumologic' is registered
Oct 24 22:19:53 ubuntu-xenial systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Oct 24 22:19:53 ubuntu-xenial systemd[1]: Failed to start Docker Application Container Engine.
Oct 24 22:19:53 ubuntu-xenial systemd[1]: docker.service: Unit entered failed state.
Oct 24 22:19:53 ubuntu-xenial systemd[1]: docker.service: Failed with result 'exit-code'.
Oct 24 22:19:53 ubuntu-xenial systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Oct 24 22:19:53 ubuntu-xenial systemd[1]: Stopped Docker Application Container Engine.
Oct 24 22:19:53 ubuntu-xenial systemd[1]: docker.service: Start request repeated too quickly.
Oct 24 22:19:53 ubuntu-xenial systemd[1]: Failed to start Docker Application Container Engine.

[dwkerwin]

I tested on the latest ECS AMI (2017.03.g), specifically amzn-ami-2017.03.g-amazon-ecs-optimized, and can confirm it does not work on that version.

[bin3377]

@dwkerwin okay... I didn't try the AMI but from the release note of 2017.03.g it should contains the change we submitted to ecs-agent. One possible reason is the plugin style logging driver requires minimum docker version 1.29. Can you check the docker version on the AMI?

[dwkerwin]

@bin3377 the docker version on the 2017.03.g AMI is Docker version 17.03.2-ce, build 7392c3b/17.03.2-ce

[njparisi]

I am experiencing the same behavior. I used a centos 7 ami and registered the instances with ECS via the latest version of the ecs agent. I installed Docker 17.09 and registered the 1.0.2 version of the plugin. I can specify the driver on the command line but cannot get it to register with the daemon using the daemon.json config file. I get the error "dockerd: Failed to set log opts: logger: no log driver named 'sumologic' is registered". There doesn't appear to be a way to pass in the command line options from a task definition since sumologic is not an option from the dropdown for logging drivers.

[mhumeSF]

@njparisi You can still register tasks via a json file with the config options set.

$: aws ecs register-task-definition --cli-input-json "file://app1-task-definition.json"

# app1-task-definition.json
{
    "family": "", 
    "containerDefinitions": [
...
            "logConfiguration": {
                "logDriver": "sumologic", 
                "options": {
                    "sumo-url": "<sumologic-endpoint>"
                }
            }
...
}

[njparisi]

@mhumeSF Yes, that worked. Thanks!

[bin3377]

@dwkerwin The docker 17.03.2-ce only support API version up to 1.27, so unfortunately the logging driver is still don't supported by ECS build-in AMI now. Currently the only way to use this driver with ECS is building the container host on top of other Linux flavors.

docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   7392c3b/17.03.2-ce
 Built:        Wed Aug  9 22:45:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   7392c3b/17.03.2-ce
 Built:        Wed Aug  9 22:45:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

[hoegertn]

The latest image now contains Docker version 17.06.2-ce, build 3dfb8343b139d6342acfd9975d7f1068b5b1c3d3 and it works if you specify the log-driver on container start. But if you create a daemon.json file it says Failed to set log opts: logger: no log driver named 'sumologic' is registered.

Any idea?

[dwkerwin]

Yes, the new ECS AMI is compatible with the sumo log driver plugin. But you need to indicate to ECS that sumologic is an available logging driver as well as installing the docker plugin:

echo ECS_AVAILABLE_LOGGING_DRIVERS='["sumologic","awslogs","syslog","json-file"]' >> /etc/ecs/ecs.config

docker plugin install store/sumologic/docker-logging-driver:1.0.2 --alias sumologic --grant-all-permissions

You can do this in the userdata for your launch config when your ECS hosts boot up.

[hoegertn]

Even after adding it to the ECS config it still says: no log driver named 'sumologic' is registered.

docker info does not show the plugin either:

Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog

But docker plugin ls does:

ID                  NAME                DESCRIPTION                 ENABLED
a1894ff3ff9a        sumologic:latest    Sumo Logic logging driver   true

[bin3377]

@hoegertn can you try on the ECS instance with manually launching container with the logging driver?

docker run --log-driver=sumologic --log-opt sumo-url=XXX -d hello-world

And after modify the configure in /etc/ecs/ecs.config, you probably need restart the ecs agent to make it works:

sudo service stop docker
sudo service start docker
sudo start ecs

I created a wiki page for full steps with 2017.09.a AMI. Hope it's helpful.

[hoegertn]

It works when you specify it on container launch, but this is not what I need in my cluster. This should not be part of every TaskDefinition but should be configured in the ECS configuration.

I will try restarting some services. (EDIT: Did not help)

[endofcake]

@hoegertn , on Amazon ECS optimised you can set default docker daemon options in /etc/sysconfig/docker and then restart docker service. I'd argue though that sumo driver needs to be specified in each task individually, as otherwise logs from different services with different log format will get dumped into one source category.

[hoegertn]

Changing the daemon options in /etc/sysconfig/docker raised the same error. I think logging is part of the infrastructure configuration and services should not think about that. All my services use the same log format so this is no problem in my setup.

I will try to install another logging-driver and check if it is an ECS or a driver problem.

[hoegertn]

Is there anything new here? I would really love to use the logging driver plugin on ECS

[bin3377]

Confirm it's repro on latest AMI. Filed an issue for docker engine: https://github.com/moby/moby/issues/35553

[ryangardner]

I've tested that I have installed the logging plugin properly (I'm able to docker run and get the hello-world output to my sumo collector via the plugin) but see the same issues as mentioned above.

Earlier in the thread it sounded like you could define task definitions that use the sumologic driver and they would work, but that's not the case for me any more.

In ECS, When I try to launch an instance that is configured with the 'sumologic' log driver, it wont even try to start because ECS adds on a requirement for the logging driver in the required-attributes;

 "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.sumologic"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    }
  ],

It wont launch because it wont find an instance that matches that. (the specific message in the events list of the services is:

service foo was unable to place a task because no container instance met all of its requirements. The closest matching container-instance 21804a2b-3ec1-4dff-aaaa-3167b7d8fe32 is missing an attribute required by your task. For more information, see the Troubleshooting section.

I presume that this is related to the docker info not showing the sumologic plugin in the list of logging drivers?

[bin3377]

@ryangardner could you please provide the AMI version you are using?

[ryangardner]

I'm on ami-fad25980

I'm doing some more double-checking now... that error doesn't make any sense because I see:

{
                    "name": "com.amazonaws.ecs.capability.logging-driver.sumologic"
                }, 

when I describe-instance on the ecs instance running it, so it seems like the required attributes should be satisfied.

I'm creating a very simple task definition for the hello-world image and I'll see if I can get that to work with the sumologic driver

[ryangardner]

Ok, I was wrong. I am able to get it to work with ECS using a simple hello-world logging config.

That earlier error was likely from when I had the wrong instances set to drain so only older ones without sumo were left to run tasks or something. sorry about that.

[waddles]

This was a bug in Docker (moby) raised in https://github.com/moby/moby/issues/35553, fixed in https://github.com/moby/moby/pull/36306 and released in Docker 18.03.0-ce.

Please close this issue.