search

TritonDataCenter / manta-thoth

Thoth is a Manta-based system for core and crash dump management
16 stars 7 forks source link

Thoth's check step can leave behind zombie processes #175

Open KodyKantor opened 5 years ago

KodyKantor commented 5 years ago

I discovered a number of very old zombie thoth processes while looking into a different issue.

[kkantor@headnode] $ ps auxwww | grep 'thoth info' | grep -v grep
thoth    32860  0.0  0.1 62476 43768 ?        S   Mar 20  3:17 node /opt/custom/thoth/node_modules/manta-thoth/bin/thoth info c5f121ed3033fd8783784972b7f29fa8
thoth    39408  0.0  0.1 62476 43724 ?        S   Mar 20  3:21 node /opt/custom/thoth/node_modules/manta-thoth/bin/thoth info b4e7f28b94790c12b6fa2df9e43b3b8c
thoth    43412  0.0  0.1 62476 43752 ?        S   Dec 06  1:02 node /opt/custom/thoth/node_modules/manta-thoth/bin/thoth info 9f6b7f021ad22d46fde19eda4bdc7d9c
thoth    51559  0.0  0.1 62476 43768 ?        S   Mar 16  3:49 node /opt/custom/thoth/node_modules/manta-thoth/bin/thoth info 4df61a2fc1c89b023641f5fa2ed35752
thoth    80222  0.0  0.1 62476 43764 ?        S   Mar 16  3:48 node /opt/custom/thoth/node_modules/manta-thoth/bin/thoth info 1126037f2ee0b7658f65aba806fa6337

I can 'thoth info' these cores locally and the command returns quickly with the expected data. I took a core of the last process in the list and uploaded it to thoth. It was given ID 9edd81bcb8c67a2ae2fd7c1d99b10e6a.

I left the thoth processes running. I'll either get back to further debug this, or this will be an embarrassing reminder that I never root caused this issue.

bahamat commented 5 years ago

@KodyKantor Do you know how/when these get kicked off?

KodyKantor commented 5 years ago

@bahamat Yes, I believe these are part of the sdc-thoth upload routine to check that objects were uploaded correctly: https://github.com/joyent/manta-thoth/blob/9fea183fb08a7b60c5a75544fc8a1b204ec21605/bin/sdc-thoth#L567