search

n0-computer / iroh

peer-2-peer that just works
https://iroh.computer
Apache License 2.0
2.61k stars 165 forks source link

Iroh aborts under high load #2424

Open rklaehn opened 5 months ago

rklaehn commented 5 months ago

@ppodolsky on discord:

Well... I have left Iroh for the last night and it has been supposed to do intense inserts into docs, but unfortunately I have noted right now that after the last code update Iroh has started to restart sometimes. Looks like process being aborted, no any useful log lines and signs of disruption before abortion. And Iroh has restarted for 4 times in 8 hour period, I have never met these behaviour before latest commits.

I will try to watch Iroh today, but no idea where should I look at. btw, there is also an intense spam 2024-06-28T04:28:32.076983Z INFO poll_send:get_send_addrs: new connection type typ=mixed me=yyy node=xxx, approximately every second. Mentioned node xxx is always the same

Possibly related to the changes we have made to tokio task handling. I have improved error logging in https://github.com/n0-computer/iroh/pull/2423 already.

rklaehn commented 4 months ago

This branch adds some brute force println! to track down why the main loop aborts: https://github.com/n0-computer/iroh/tree/pasha-debug

rklaehn commented 4 months ago

@pashapodolsky we just had a discussion about panic (weirdly in storage).

I noticed that we only set panic=abort in release-optimized mode. But I think that is not relevant for you since you embed iroh. Are you building with panic=abort?

Do you have RUST_BACKTRACE=1 set when you are running your thing?