Cannot connect dockerised microservices after v.6.0.5

[kmyllyvi]

Regression

Potential Commit/PR that introduced the regression**

6.1.0?

Describe the regression

I was struggling a while after upgrading NestJS to latest 6.4.0 with microservices that run inside docker. After downgrading @nestjs/microservices to version 6.0.5 it all started working fine. Tried few releases in between but didn't manage to get it working. I did not change any code. Just following the example. With latest NestJS version I'm getting error connect ECONNREFUSED

Input Code

Code that works on v.6.0.5, nothing special, following microservice example from https://docs.nestjs.com/microservices/basics

server main.ts ->

    const msPort: number = 3001;
    const appAsMicroservice: INestMicroservice = await NestFactory.createMicroservice(ApplicationModule, {
        transport: Transport.TCP,
        options: { port: msPort }
    });
    await appAsMicroservice.listen(() => console.log('microservice is ready at: ' + msPort));

tried connecting the client with both "module way" ... ->

@Module({
    imports: [
        ClientsModule.register([
            {
                name: 'my_microservice',
                transport: Transport.TCP,
                options: { port: 3001, host: 'module_docker_name' }
            }
        ])
    ]

with @Inject ->

constructor(@Inject('module_docker_name') private readonly microserviceClient: ClientProxy) {}

... and alternatively using ClientProxyFactory.create() ->

        this.microserviceClient = ClientProxyFactory.create({
            transport: Transport.TCP,
            options: {
                host: 'module_docker_name',
                port: 3001
            }
        });

Both methods work on NestJS v.6.0.5 but not after that

Expected behavior/code

Microservices should work also between docker containers

Environment

Nest version: 6.0.5 -> 6.4.0

For Tooling issues:
- Node version: 8.12.0  
- Platform:  Mac Mojave 10.14.4

Others:

[kamilmysliwiec]

Microservices should work also between docker containers

They work inside Docker containers (nothing has changed here)

Please, provide a minimal repository which reproduces your issue.

[kmyllyvi]

If you say nothing has changed then I need to use more time on debugging what could cause the issue. Perhaps it's some other package dependency that breaks it then.

[kmyllyvi]

Okay, I have to apologise. Now trying to reproduce this with 3 docker microservices I realised I most likely had different nestJS versions for different servers. So I will close this issue with a remark that there might be issues when trying to connect microservice that do not share the same @nestjs/microservices version. Would you agree to that point?

[kmyllyvi]

Hi. I need to open this issue again because it is actually reproducing regularly. I have now made a demo project to demonstrate the issue: https://github.com/kmyllyvi/nest-microservice-demo

It's based on another project on git and I just modified it to use microservice connection between two docker services. It has some code that is irrelevant to this issue, but demoing microservice should be pretty straight forward. To test make a GET request to hostname:3000/orders for example http://localhost:3000/orders -> that should give 200 OK

So I first pushed with NestJS version 6.0.5 and everything works. When I upgrade to NestJS 6.4.0 I get: { "errno": "ECONNREFUSED", "code": "ECONNREFUSED", "syscall": "connect", "address": "172.26.0.3", "port": 4000 }

I can't tell if it's some package dependency issue or what.

[kamilmysliwiec]

Have you tried updating from 6.4.0 to 6.5.2? There as a regression (see release: https://github.com/nestjs/nest/releases/tag/v6.5.2), see discussion here https://github.com/nestjs/nest/pull/2428

[kmyllyvi]

I didn't. I will give it a shot.

[kmyllyvi]

No luck with 6.5.2 either. According to my testing it's really already breaking from 6.0.5 -> 6.1.0

[GabrielGil]

I was struggling to get my microservices working fine in Google Kubernetes Engine last week until, for some random enlightening I got to identify what to change in order to access a Dockerized microserviced app.

If no host is specified, NestJS will bind to localhost as defined in constants.ts.:

https://github.com/nestjs/nest/blob/a0a51302ed3d13476e5e19c65c3a3cc07bae6f42/packages/microservices/constants.ts#L2

Changing the host in the createMicroservice function to 0.0.0.0 seems to work just fine even with different Nest versions (6.5.3 connecting to 6.5.2).

const app = await NestFactory.createMicroservice(AppModule, {
    transport: Transport.TCP,
    options: {
      host: '0.0.0.0',
      port: 3000
    }
  });

[BorntraegerMarc]

Changing the host in the createMicroservice function to 0.0.0.0

this will only work if your container setup maps the internal docker port to the host.

Most docker containers will not expose and map ports to host. So IMO defining the host as docker_container_name should work.

[GabrielGil]

Thank you for your answer @BorntraegerMarc. I am not sure what you say is exactly like that. Using localhost as the host value in the app allows exposing the app to another port on the host machine using -p. I am not sure why the same method just rejects incoming connections in v6.5.2.

[BorntraegerMarc]

@kamilmysliwiec any update on the issue? As mentioned updating to 6.5.2 doesn't seem to fix the issue

[crdev13]

@kamilmysliwiec Hi I'm facing the same problem with a microservice( v6.5.3) inside docker, I have this error "Error: connect: ECONNREFUSED"

[BrunnerLivio]

I think this is not related to NestJS. Let me explain:

My research is based n @kmyllyvi awesome repository.

Research

Testing Orders Service

I started a listener to check whether NestJS sends messages:

# terminal 1
docker-compose up mongodb

# terminal 2
cd orders-app && npm install
MONGO_URI=mongodb://localhost:27000/nest-microservice PAYMENT_ENDPOINT=http://payments:4000/payments npm run start:dev

# terminal 3
nc -l 4000 

# terminal 4
curl localhost:3000

and I received in terminal 3 the following message:

92#{"pattern":{"cmd":"payments"},"data":"hi there","id":"d69b69c2-7b26-43d8-ba10-9f427ec55416"}

so far so good. Now save this message in message.txt.

Testing Payment Service

Lets send this message manually using netcat once payment microservices runs

# terminal 1
docker-compose up

# terminal 2
netcat localhost 4000 < ./message.txt

and I receive the message in the docker-compose log:

payments-container | payments ms api test got: hi there

Conclusion

What we can take out of this investigation:

• Orders Service sends the correct messages
• Payment Service can parse and receive the messages

Therefore this lets me to believe there is something wrong with the docker network setup, not NestJS. I have not found the actual network problem yet, but in case you want to help me investigate, my research may help you.

[kmyllyvi]

Hi. Thanks @BrunnerLivio for exploring the issue. Also very happy to hear my sample project has helped in the process :) Unfortunately at the moment I don't have time to investigate further myself, but perhaps at some point if there's still no resolution to this.

[kamilmysliwiec]

Thanks for your investigation @BrunnerLivio. Even though I'd love to help, it seems that this issue isn't specifically related to Nest. I'd suggest looking at our Discord channel. We've got a very helpful community, maybe they will know how to help you!

[mattwills8]

@GabrielGil you saved my life man, I'd racked up about 20 hours trying to figure out why my services couldn't communicate. I least im pretty familiar with every single page of the kubernetes docs now.....

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.