search

HangfireIO / stdump

Explore stack trace of a running managed process or from a minidump file
Other
20 stars 5 forks source link

Empty output from a tool #4

Closed FixRM closed 8 months ago

FixRM commented 9 months ago

Hello, @odinserj. I'm trying to collect logs from sample Hangfire, app but I got empty result. I tried to run my app in a console and as Windows service but no luck. I manually download last release from here. I'm running .Framework 4.7.1 on latest Windows 11. What am I doing wrong?

I put my app here: https://github.com/FixRM/Hangfirelogs/tree/main/HangfireDemo. We finally find a way to reproduce issue with leakage of worker threads that I reported earlier.

FixRM commented 9 months ago

We got a dump on another machine: https://github.com/FixRM/Hangfirelogs/blob/main/stdump_demo.txt. Should I open at issue in HF repo, or we can continue here? I can see that threads #27 and #28 are running a job, but UI show only one:

image

Is it storage, or UI or whatever issue?

odinserj commented 9 months ago

Can you show me all the state transitions of that active processing job? It is theoretically possible that it gets executed by multiple workers inside the same Hangfire Server, if storage implementation allows this, and I'd like to understand whether it's the case.

FixRM commented 9 months ago

Sure @odinserj, how can I do that? There was only one server in experiment, state transitions are not customized. Hangfire setup was basic: only storage + wokercount was set to two.

odinserj commented 9 months ago

Just wanted to check screenshot like the following one to see how many "Processing" entries it contains.

image

FixRM commented 9 months ago

Here it is

image

odinserj commented 9 months ago

Thanks, so my assumption was correct. Can you please include the full screenshot to understand the timings, e.g. when second transition to the Processing state was performed?

FixRM commented 9 months ago

I guess this one was queued last week. If you want I can clear DB and start experiment from scratch

image

FixRM commented 9 months ago

Hello @odinserj. Sorry for asking, what are our next steps?

odinserj commented 8 months ago

Sorry for the delayed response. For some reason, another worker from the same process is picking up a job, despite previous one is still running. Perhaps something is incorrect in the fault detection logic of the storage implementation, since it allows a background job to be fetched again when there was no any failure. Last week I started an infinite job using the Hangfire.Pro.Redis, here's the output so it's still running:

image

I see you are using Hangfire.Redis.StackExchange, a community-based implementation of Redis as a job storage. Please try raising an issue in their repository instead, with all of these screenshots.

FixRM commented 8 months ago

Thanks a lot!

odinserj commented 8 months ago

Not at all, Artem