search

TritonDataCenter / manta-thoth

Thoth is a Manta-based system for core and crash dump management
16 stars 7 forks source link

TOOLS-2448 need tool for purging thoth #180

Closed jlevon closed 4 years ago

askfongjojo commented 4 years ago

Hi @jlevon,

Thanks for enhancing thoth to provide date range support! This looks pretty good and will certainly make daily thoth purge work faster.

The only thing to note is that when I ran thoth info over a period of time when there was a lot of dump files, I'd run into OOM or the following error:

[root@manta-client ~/manta-thoth]# bin/thoth info mtime=187d otime=180d
thoth: using database at thothdb.svc.aed35417-4c53-4d6c-a127-fd8a6e55723b.us-east-1.triton.zone:28015 (configured from Manta)
thoth: RangeError: Invalid string length

The instance used is a base-64-lts 16.4.1 image of a 16G RAM package on node v0.12.10. The issue is however not happening on my laptop which also has 16G memory but a newer node version (v10.16.2). Is there a recommended node version which I should use?

jlevon commented 4 years ago

Well TIL I learned that adding reviewers to a draft PR shows up for them, sorry about that!

I hit similar issues while testing in jenkins. I have a newer version with a bunch of changes there that I'm still working on testing. It also updates purged dump's indexed JSON so we skip it next time - which should further reduce the runtime hopefully.

I haven't seen the string length error however.

jlevon commented 4 years ago

@askfongjojo note that the purge script is now using:

export NODE_OPTIONS="--max-old-space-size=8192"

which should hopefully avoid the issues you were seeing with running out of memory in node (the default is 1Gb-ish, almost like we shouldn't be trying to use node.js as a systems language ;)

I think this is ready for a final review now!

jlevon commented 4 years ago

@askfongjojo I re-started my jenkins job and this seems to still be working OK, could you take one last look?