search

Azure / azure-sdk-for-js

This repository is for active development of the Azure SDK for JavaScript (NodeJS & Browser). For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/javascript/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-js.
MIT License
2.09k stars 1.2k forks source link

[Service Bus] TypeError: Cannot read property '_process' of undefined #9792

Closed ahsanhabib91 closed 4 years ago

ahsanhabib91 commented 4 years ago

Describe the bug I am running 2 instances of a node.js(with typescript) script where I listen for the incoming messages coming from 2 different sessions of Azure Service Bus. After couple of hours, one or both got crashed by raising the following error: /......./service-bus-quick-starts/node_modules/rhea/lib/connection.js:731 this.local_channel_map[k]._process(); ^ TypeError: Cannot read property '_process' of undefined image image

To Reproduce Steps to reproduce the behavior:

  1. I did not find a way how to reproduce it quickly, just to start-up scripts and wait

Expected behavior I set up Service Bus receivers in the following way: image And I expect that it will keep listening for incoming messages continuously.

Additional context I also was testing Azure Event Hubs javascript(with Typescript) client library("@azure/event-hubs": "^5.2.0",). It works absolutely fine. But the Service Bus client is doing the prolem.

ramya-rao-a commented 4 years ago

Thanks for reporting @ahsanhabib91

Are you able to reproduce this consistently? If so, then enabling the logs would definitely help in figuring out what is going on here. To enable logs, set the DEBUG env variable to azure:service-bus:error,azure-amqp-common:error,rhea-promise:error and then run your experiments again.

We saw a similar error in Event Hubs, see https://github.com/Azure/azure-sdk-for-js/issues/9311 which @chradek is looking at.

ahsanhabib91 commented 4 years ago

@ramya-rao-a I am able to reproduce it approximately 1 out of 2/3 times. Mostly It produces when the process becomes idle I guess. But that may not be the case always. I will enable the DEBUG mode and try to produce you more info if I find any.

chradek commented 4 years ago

@ahsanhabib91 When you encounter this, is there a lull in receiving messages? Or is there more or less a constant stream of messaging incoming?

ahsanhabib91 commented 4 years ago

@chradek As you can see from the code sample, I am trying to receive messages continuously. I guess its raising when the process becomes idle or there is no message to fetch, but the process still tries to fetch. Therefore, it's not like that I have received a null message. Instead, it keeps trying to fetch a message, then after a certain period of time, it gets crashed by producing the error. My scenario is, I have an Azure Function which is using the Event Hub trigger. After processing data from Event Hub, I am saving data to Azure Service Bus. Then using my client library(Node.js with Typescript), I am listening for incoming Service Bus messages(with 2 sessions). Usually, I am sending 10-100 messages(1KB-1.2KB in size) at a time to the Service bus using sendBatch().

ahsanhabib91 commented 4 years ago

Hello @chradek @ramya-rao-a !!! @ramya-rao-a as you aksed, I enable the DEBUG mode with the parameters you provided. So I was testing with the receiver.registerMessageHandler as I am facing problems with batch processing using the infinite while loop. Now this time I face a different type of problem. The process encounters the following error. After this, the client no longer receives any message despite the process is running.

image

image

ahsanhabib91 commented 4 years ago

@ramya-rao-a @chradek Should I move the issue described on my last comment to a different one?

ramya-rao-a commented 4 years ago

Hey @ahsanhabib91,

Yes, let's track the recent problem you are seeing separately. When logging the new issue, please share the rest of the logs, and any other details of your setup

chradek commented 4 years ago

@ahsanhabib91 We've just released service-bus 1.1.8 that fixes many of the TypeError issues we've encountered. https://www.npmjs.com/package/@azure/service-bus/v/1.1.8

Can you give this new version a try?

chradek commented 4 years ago

I'm closing this issue as we believe it is resolved in service-bus version 1.1.8. If you encounter the TypeError: _process of undefined exception after updating please open a new issue and we'll investigate.

NoelAbrahams commented 4 years ago

@chradek did this fix also go into the preview versions? If so, into which version did this fix go?

ramya-rao-a commented 4 years ago

@HarshaNalluru, Can you help with the above query?

HarshaNalluru commented 4 years ago

@NoelAbrahams, yes. All the fixes that were made to version 1.1.x have been ported to the preview versions too. Specifically, related to TypeError fixes, the observed issues coming from the internal "rhea" library have been addressed and the SDK has been updated to pick the latest "rhea" library as well as handling the errors better in the SDK.

All the observed issues related to the TypeError have been fixed on or before 7.0.0-preview.5.

NoelAbrahams commented 4 years ago

@HarshaNalluru, thanks for the update. The error from the rhea library was occurring in version 7.0.0-preview.3. We've now updated to 7.0.0-preview.7 and hopefully that should fix it.

HarshaNalluru commented 4 years ago

Good to know, @NoelAbrahams! Feel free to log an issue in case you encounter one. 🙂

otociulis commented 3 years ago

Hi,

we're using "@azure/service-bus": "^7.0.0", (latest at the moment of writing) and we are running into same issue when running our microservice utilizing service bus in the Azure Cloud - usually this happens after longer periods of inactivity:

 /home/azure-user/render-service/node_modules/rhea/lib/connection.js:731
             this.local_channel_map[k]._process();
                                       ^
 TypeError: Cannot read property '_process' of undefined
     at Connection._process (/home/azure-user/render-service/node_modules/rhea/lib/connection.js:731:39)
     at processTicksAndRejections (internal/process/task_queues.js:79:11)

Typescript version: 4.0.5 Node.js version: 12.18.4 Rhea version (from package-lock.json): 1.0.24

NoelAbrahams commented 3 years ago

I agree with @otociulis. This issue was never fixed. We are still seeing the error in the logs.

There is no indication in the conversation above that anyone investigated this bug thoroughly and fixed it.

We've just released service-bus 1.1.8 that fixes many of the TypeError issues we've encountered

I'm closing this issue as we believe it is resolved in service-bus version 1.1.8.

In the above 'fixes many of the TypeError issues we've encountered' and 'we believe it is resolved' are not the sort of statements that inspire confidence.

ramya-rao-a commented 3 years ago

Hey @otociulis and @NoelAbrahams,

Apologies for the delay in response. Comments on closed issues do not end up getting the same attention as open issues. We will do better there. I'll open a new issue so that we can continue the discussion and investigation there.

There is no indication in the conversation above that anyone investigated this bug thoroughly and fixed it.

@NoelAbrahams We have had multiple issues around TypeErrors thrown from rhea across Event Hubs and Service Bus SDKs which have similar logic around connection creation and recovery. The investigation most probably is documented better in one of the other issues and PRs like #9311, #8884, #10046 and #10051

I recall we were able to replicate the below TypeError cases and made improvements around how we worked with rhea connections.

Looks like the improvements were not enough to handle other cases which we were not able to replicate like the one being discussed here.

Let's continue this in #13500