search

amazon-archives / service-discovery-ecs-dns

ARCHIVED: Service Discovery via DNS with ECS.
Apache License 2.0
166 stars 77 forks source link

Is ecssd_agent using the correct AWS API endpoints? #44

Open bploetz opened 6 years ago

bploetz commented 6 years ago

I'm getting this error when the ecssd_agent attempts to register DNS records for my Docker service:

AccessDenied: The resource hostedzone/XXXXXXXXX can only be managed through servicediscovery.amazonaws.com (arn:aws:servicediscovery:us-east-1:385298791949:namespace/ns-n5xn65imrgpwaztt)\n\tstatus code: 403

Full logs:

Dec 29 22:42:50 ip-XX-X-XXX-XXX ecssd_agent[2088]: time="2017-12-29T22:42:50Z" level=error msg="AccessDenied: The resource hostedzone/XXXXXXXXX can only be managed through servicediscovery.amazonaws.com (arn:aws:servicediscovery:us-east-1:385298791949:namespace/ns-n5xn65imrgpwaztt)\n\tstatus code: 403, request id: 99baf104-ece9-11e7-b3f7-410794b4d19b"

Dec 29 22:42:50 ip-XX-X-XXX-XXX ecssd_agent[2088]: time="2017-12-29T22:42:50Z" level=info msg="Record _test._tcp.servicediscovery.internal created (1 1 9091 ip-XX-X-XXX-XXX.ec2.internal)"

Dec 29 22:42:50 ip-XX-X-XXX-XXX ecssd_agent[2088]: time="2017-12-29T22:42:50Z" level=error msg="Error creating DNS record"

The second log message indicting success seems to be a bug, as no records of any type (A or SRV) get created in the given hosted zone.

niteshldd commented 6 years ago

Hi I am also getting same error . @bploetz are you able to resolve

bploetz commented 6 years ago

@niteshldd I think to get around this I had to stop using ecs service discovery, and used route53 manually, which works fine. I wrote a blog post about it if you're interested: https://medium.com/ground-signal-engineering/ecs-service-discovery-with-lambda-dns-and-haproxy-1126ab381688

givenm commented 6 years ago

I ran into the same issue and I resolved it using aws cli after getting a hint from this post.

  1. execute aws servicediscovery list-services to see a list of services that needs to be deleted first before you can delete the private hosted zone
  2. If you have the services, execute aws servicediscovery delete-service --id srv-m7vxqqagemf6hiea to delete the service with id srv-m7vxqqagemf6hiea. Of-course the id will be different from yours and you will probably have more than 1 services registered if you have many microservices. Delete all of them.
  3. Now to see the private hosted zone, execute aws servicediscovery list-namespaces to see the service discovery private hosted zones that you want to delete.
  4. To delete the private hosted zone, execute aws servicediscovery delete-namespace --id ns-ytdqyhvchxbv3qer where "ns-ytdqyhvchxbv3qer" is the id you would have seen in step 3. When done, if you go back to the console, the hosted zone should be gone. Pooooooof