Moderate load on local replica (or emulator) lead to canister calls hanging indefinitely

[megrogan]

Steps to reproduce:

1. Install dfx 0.7.0-beta.2 DFX_VERSION=0.7.0-beta.2 sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
2. Start dfx (issue happens much more quickly when running emulator) dxf start --emulator
3. Clone "open chat" https://github.com/dfinity-lab/open-chat/commit/04162d2a6004221d5ca8ceafb50fb022f6954187
4. Build and deploy "open chat" locally 3.1 cd into open-chat 3.2 npm install 3.3 dfx deploy
5. Open local "website" canister in browser, sign-in, and register a user name
6. Repeat step 4 in another browser - say Chrome incognito
7. Open chrome dev tools and watch the network tab (open chat polls for new messages)
8. After a few minutes you should see the query calls hanging

[megrogan]

Btw, the same issue occurs with just one instance of "open chat" it just takes longer

[p-shahi]

Has this been debugged and resolved? We noticed that a lot last week when working on the identity provider, and it almost derailed our Dom demo… (My gut feeling debugging seems to point that some component in dfx runs out of something like connections, as if it doesn’t know that it has closed connections and eventually doesn’t allow new ones.)

@nomeata

[taylorham]

Affecting CanCan as well. Seems to consistently happen to us after around 9 minutes of the replica being started, but that seems more of a coincidence than a cause.

@hansl has suggested this could also be related to a hang we're experiencing during the asset upload phase of dfx deploy/dfx canister install..., where the replica becomes completely unresponsive after uploading a portion of assets and we have to cancel the process. dfx stop then prints the "Stopping the replica... Stopped." output, but the dfx process is never stopped and has to be manually killed. Let me know if I should create another ticket for this one, @ericswanson-dfinity!

[nomeata]

I worked around it locally by not using dfx start but rather running icx-proxy; instructions in https://github.com/dfinity/idp-service/blob/main/Demo.md. This may be useful for people who need this working right now more reliably for a Demo.

[ericswanson-dfinity]

Trying to repro, I get stuck at step

5. Open local "website" canister in browser, sign-in, and register a user name

[sesi200]

This could also be releated to the issues mentioned by Mio here