manta-thoth

Thoth is a Manta-based system for core and crash dump management for illumos-derived systems like SmartOS, OmniOS, DelphixOS, Nexenta, etc. -- though in principle it can operate on any system that generates ELF core files.

Installation

$ npm install git://github.com/joyent/manta-thoth.git#master

Setup

As with the node-manta CLI tools, you will need to set Manta environment variables that match your Manta account:

$ export MANTA_KEY_ID=`ssh-keygen -l -f ~/.ssh/id_rsa.pub | awk '{print $2}' | tr -d '\n'`
$ export MANTA_URL=https://us-east.manta.joyent.com
$ export MANTA_USER=bcantrill

You may also need to set your THOTH_USER environment variable if you are using a shared Thoth installation. For example, if your shared Thoth installation uses the 'thoth' Manta user:

$ export THOTH_USER=thoth

If $THOTH_USER is set, $MANTA_USER must have read and write access to /$THOTH_USER/stor/thoth.

While all of its canonical data resides in Manta, Thoth uses RethinkDB for metadata caching. If setting up a new $THOTH_USER, RethinkDB should be installed on a server, and then pointed to via $THOTH_USER/stor/thoth/config.json as described below.

Once RethinkDB is installed, the authKey should be set. On versions of RethinkDB of 2.3 or more recent, this can be done via the Data Explorer from the RethinkDB web interface:

r.db('rethinkdb').table('users').get('admin').update({password:'I<3dumps!'})

(Once this has been done, it's wise to disable web administration by uncommenting the no-http-admin line in the RethinkDB instances's configuration file.)

To initialize thoth, first store the RethinkDB credentials in Manta at $THOTH_USER/stor/thoth/config.json:

{
        "db": { "host": "my-thoth-server", "authKey": "I<3dumps!" }
}

Then, run thoth init:

$ thoth init
thoth: using database at my-thoth-server:28015 (configured from Manta)
thoth: created database 'bcantrill'
thoth: created table 'dumps'
thoth: created table 'analyzers'
thoth: created index 'time'

Now you can upload your first core dump:

$ thoth upload ./core.bc.24388
thoth: using database at my-thoth-server:28015 (configured from Manta)
thoth: creating 76998f82a450a8914037e4da838ec609
thoth: uploading core.bc.24388 to 76998f82a450a8914037e4da838ec609
thoth: core.bc.24388 [=======================>] 100%   3.83MB
thoth: creating job to uncompress 76998f82a450a8914037e4da838ec609
thoth: adding key to job 42b9feff-56d5-482a-b12b-da2099fd44ed
thoth: processing job 42b9feff-56d5-482a-b12b-da2099fd44ed
thoth: waiting for completion of job 42b9feff-56d5-482a-b12b-da2099fd44ed
thoth: job 42b9feff-56d5-482a-b12b-da2099fd44ed completed in 0h0m14s
thoth: creating job to process 76998f82a450a8914037e4da838ec609
thoth: adding key to job 3e434caf-0544-6e89-ed71-8fa1630adcde
thoth: processing 76998f82a450a8914037e4da838ec609
thoth: waiting for completion of job 3e434caf-0544-6e89-ed71-8fa1630adcde
thoth: job 3e434caf-0544-6e89-ed71-8fa1630adcde completed in 0h0m8s

This dump should appear in thoth ls output:

$ thoth ls
thoth: using database at my-thoth-server:28015 (configured from Manta)
NAME             TYPE  TIME                NODE/CMD         TICKET
76998f82a450a891 core  2015-12-04T13:10:26 bc               -

Running Thoth

Introduction

Thoth consists primarily of the thoth utility, a veneer on Manta that generates a hash unique to a core or crash dump, uploads that dump to a directory under $MANTA_USER/stor/thoth, loads the metadata associated with the dump into a RethinkDB-based querying database, and offers facilities to list, filter and (most importantly) debug those dumps.

If used with a Manta v1 installation, thoth uses Manta jobs. With Manta v2, jobs are not supported. In this case, things such as thoth debug run on the local machine.

Dump specifications

Most thoth subcommands operate on a dump specification: a dump's hash (or substring thereof) or a space-delimited set of constraints based on its properties. A constraint consists of a property name, a single equals sign, and the value (or globbed expression) to match. For example, to list all crash dumps from the node 95SY9R1:

$ thoth ls type=crash node=95SY9R1

Special token: `mtime`

The special token mtime denotes how long ago the dump was uploaded, with equality denoting recency. For example, to list all of the dumps uploaded in the last 6 hours:

$ thoth ls mtime=6h
thoth: using database at thoth-db:28015 (configured from Manta)
NAME             TYPE  TIME                NODE/CMD         TICKET
e1f5422b892d9394 core  2017-11-17T19:34:29 java             -
c04110bc8190a84e core  2017-11-17T19:39:52 node             -
b9379570b4a9a224 core  2017-11-17T19:39:52 node             -
5f1171019ce419cb core  2017-11-17T19:51:01 node             -
713f9e8b48559acd core  2017-11-17T19:55:57 node             -
d91719939666de40 core  2017-11-17T20:05:57 node             -
beaa65d3548ac96f core  2017-11-17T20:23:19 pg_prefaulter    -
5841ba86a2b198be core  2017-11-17T20:53:06 node             -
3d8921ce583dff68 core  2017-11-17T20:54:06 node             -
34a8661c049456b1 core  2017-11-17T21:14:09 node             -
6d75f1cd30898f48 core  2017-11-17T21:31:19 node             -
cc2328f7d8a6c4ad core  2017-11-17T21:41:15 node             -
b0b6f4ed9ab418ce core  2017-11-17T21:51:20 node             -
5d64e695505d15c9 core  2017-11-17T22:11:18 node             -
96d2271d81e4cd63 core  2017-11-17T22:21:17 node             -
71aff9c315553b03 core  2017-11-17T23:31:14 node             -
72f18495c7f54841 core  2017-11-18T00:21:17 node             -

Special token: `otime`

The special token otime can be used to match only dumps uploaded prior to that time. It can be combined with mtime to specify a range, for example, all dumps uploaded in the previous week could be listed with:

thoth ls mtime=2w otime=1w

Special token: `limit`

The special token limit denotes that the number of dumps specified should be limited to the parameter, allowing a smaller number of dumps to be examined. (Exactly which dumps will be returned is unspecified.) For example, to get the ID of at most five dumps from commands that begin with "system":

$ thoth info cmd=systemd* limit=5 | json -ga id
thoth: using database at thoth-db:28015 (configured from Manta)
00103a107b5db8f79ebc77782b707d07
0071f6c50b39f1a917ba21a957f43e3f
0021e9c447815c1f7a91e1af2672543b
00d7ae803e01365798654c4dbeea5b28
012c3f942d0b7de6b7dbc8eed8798b86

Special token: `undefined`

The special token undefined denotes a property that isn't set. For example, to list all dumps that were added in the last one hundred days that begin with svc that don't have a ticket:

$ thoth ls mtime=100d cmd=svc* ticket=undefined
thoth: using database at thoth-db:28015 (configured from Manta)
NAME             TYPE  TIME                NODE/CMD         TICKET
0ecc8338c5949ea7 core  2017-08-10T01:57:56 svc.startd       -
925de938d529e58b core  2017-08-18T03:51:27 svcs             -
1d16db174473d8b5 core  2017-08-18T04:53:30 svcs             -
2b4b3f5931e4b945 core  2017-08-18T05:39:14 svcs             -
c5761bf75ea51a3f core  2017-08-18T08:27:01 svcs             -
2204949c1735126b core  2017-08-18T15:08:35 svcs             -
bc987a441a10da48 core  2017-08-24T17:44:41 svc.startd       -
d7ba3510178394c3 core  2017-09-06T12:53:45 svcs             -
48157650dc2d4204 core  2017-09-07T00:24:59 svccfg           -
c48278b2930f991c core  2017-09-07T01:09:49 svccfg           -
14918d63fb7239da core  2017-09-26T01:26:44 svc.startd       -
9a29ead38c89930a core  2017-10-01T08:22:37 svc.configd      -
d36a11c974f7f03d core  2017-10-01T08:22:37 svc.startd       -
463412ce271ec7ec core  2017-10-02T15:39:23 svc.startd       -

Special specification: `dump=stdin`

The special specification dump=stdin denotes that dump identifiers should be read from standard input, e.g.:

$ cat /tmp/dumps
3f7a8bde5a907afab7f966b9963c7d10
3260a5e49918260ccdc1f94830c937c1
f12ea8712e8b2586f062b03808b1c292
5aaa91149e94a91f66c76b00ec1de521
04a681f27ffcd19952d8efb75006c490
$ cat /tmp/dumps | thoth ls dump=stdin
thoth: using database at thoth-db:28015 (configured from Manta)
thoth: reading dump identifiers from stdin
NAME             TYPE  TIME                NODE/CMD         TICKET
3260a5e49918260c core  2017-11-16T22:30:22 pg_prefaulter    -
5aaa91149e94a91f core  2017-11-17T11:07:43 pg_prefaulter    -
04a681f27ffcd199 core  2017-11-17T14:12:27 pg_prefaulter    -
3f7a8bde5a907afa core  2017-11-17T14:42:03 pg_prefaulter    -
f12ea8712e8b2586 core  2017-11-17T17:22:21 pg_prefaulter    -

Subcommands

thoth operates by specifying a subcommand. Many subcommands kick off Manta jobs when using v1, and the job ID is presented in the command line (allowing Manta tools like mjob to be used to observe or debug behavior). In general, success is denoted by an exit status 0 and failure by an exit status of 1 -- but some subcommands can exit with other status codes (notably, info). The following subcommands are supported:

upload

Takes the name of a core or crash dump to upload. It will generate a hash unique to the dump, upload the dump, and kick off a Manta job to postprocess it:

$ thoth upload core.19972
thoth: creating 3e166b93871e7747c799008f58bd30b9
thoth: uploading core.19972 to 3e166b93871e7747c799008f58bd30b9
thoth: core.19972    [=======================>] 100%   1.94MB
thoth: creating job to uncompress 3e166b93871e7747c799008f58bd30b9
thoth: adding key to job 84b7f163-ecda-49bd-ba8e-ffc5efd8da62
thoth: processing job 84b7f163-ecda-49bd-ba8e-ffc5efd8da62
thoth: waiting for completion of job 84b7f163-ecda-49bd-ba8e-ffc5efd8da62
thoth: job 84b7f163-ecda-49bd-ba8e-ffc5efd8da62 completed in 0h0m4s
thoth: creating job to process 3e166b93871e7747c799008f58bd30b9
thoth: adding key to job da3c0bf5-b04f-445b-aee7-af43ea3d17c0
thoth: processing 3e166b93871e7747c799008f58bd30b9
thoth: waiting for completion of job da3c0bf5-b04f-445b-aee7-af43ea3d17c0
thoth: job da3c0bf5-b04f-445b-aee7-af43ea3d17c0 completed in 0h0m2s

If using Manta v2, a kernel crash dump is not uncompressed after uploading (and only minimal information is collected in thoth info for the dump). The analyzer process-dump can be used to do this post-upload.

info

Returns the JSON blob associated with the specified dump.

$ thoth info 3e166b93871e7747c799008f58bd30b9
{
    "name": "/bcantrill/stor/thoth/3e166b93871e7747c799008f58bd30b9",
    "dump": "/bcantrill/stor/thoth/3e166b93871e7747c799008f58bd30b9/core.19972",
    "pid": "19972",
    "cmd": "utmpd",
    "psargs": "/usr/lib/utmpd",
    "platform": "joyent_20130418T192128Z",
    "node": "headnode",
    "version": "1",
    "time": 1366869350,
    "stack": [ "libc.so.1`__pollsys+0x15()", "libc.so.1`poll+0x66()", "wait_for_pids+0xe3()", "main+0x379()", "_start+0x83()" ],
    "type": "core",
    "properties": {}
}

Trent Mick's excellent json is recommended to post-process these blobs; here's an example of printing out the stack traces of dumps that match a particular ticket:

$ thoth info ticket=OS-2359 | json -ga dump stack
thoth: created job 8ba4fae1-ce47-43fa-af24-3ad2916d48f1
thoth: waiting for completion of job 8ba4fae1-ce47-43fa-af24-3ad2916d48f1
thoth: job 8ba4fae1-ce47-43fa-af24-3ad2916d48f1 completed in 0h0m19s
/thoth/stor/thoth/baef9f79a473580347b6338574007953/core.svc.startd.23308 [
  "libc.so.1`_lwp_kill+0x15()",
  "libc.so.1`raise+0x2b()",
  "libc.so.1`abort+0x10e()",
  "utmpx_postfork+0x44()",
  "fork_common+0x186()",
  "fork_configd+0x8d()",
  "fork_configd_thread+0x2ca()",
  "libc.so.1`_thrp_setup+0x88()",
  "libc.so.1`_lwp_start()"
]
/thoth/stor/thoth/ba137fd783fd3ffb725fe8d70b3bb62f/core.svc.startd.27733 [
  "libc.so.1`_lwp_kill+0x15()",
  "libc.so.1`raise+0x2b()",
  "libc.so.1`abort+0x10e()",
  "utmpx_postfork+0x44()",
  "fork_common+0x186()",
  "fork_configd+0x8d()",
  "fork_configd_thread+0x2ca()",
  "libc.so.1`_thrp_setup+0x88()",
  "libc.so.1`_lwp_start()"
]
...

Note that for the info subcommand, a dump specification can also consist of a local dump -- in which case the hash of that dump will be determined locally, and the corresponding dump information will be retrieved (if it exists). This is a useful way of determining if a dump has already been uploaded to thoth: an exit status of 0 denotes that the information was found; an exit status of 2 denotes that the dump was not found.

$ thoth info core.that.i.already.uploaded > /dev/null ; echo $?
0
$ thoth info core.that.i.have.never.seen.before > /dev/null ; echo $?
2

debug

Results in an interactive debugging session debugging the specified dump.

If using Manta v2, the dump is downloaded locally into /var/tmp/thoth/cache. It is not deleted, so running again will be much quicker; a simple rm is sufficient to clean up any unwanted local dumps.

ls

Lists the dumps that match the dump specification, or all dumps if no dump specification is provided. By default, the dumps are listed in time order from oldest to newest.

A dump abbreviation, the dump type, the time, the node or command, and the ticket are provided for each dump -- but ls will additionally display any property provided. For example, to list the stack trace in addition for all dumps in the last three days from the pg_prefaulter command:

$ thoth ls mtime=3d cmd=pg_prefaulter stack

object

For a given local dump, provides the hashed name of the object.

$ thoth object core.19972
3e166b93871e7747c799008f58bd30b9

This can be used to automate uploads of dumps.

report

Gives a JSON report of the given property across the given dump specification. For example, here's a report of platform for cores from the command svc.startd:

$ thoth report cmd=svc.startd platform
{
  "joyent_20130625T221319Z": 47,
  "joyent_20130613T200352Z": 57
}

set

Sets a user property, which will appear in the properties field of the JSON blob retrieved via info. The value for the property can be a string:

$ thoth set 086d664357716ae7 triage bmc
$ thoth info 086d664357716ae7 | json properties.triage
bmc

Or specified as a JSON object via stdin:

$ thoth set cmd=svc.configd triage <<EOF
{
    "category": "SMF",
    "engineer": "bmc"
}
EOF
$ thoth info 086d664357716ae7 | json properties.triage.engineer
bmc

unset

Unsets a user property. Once a property is unset, it can be searched for in a dump specification by using the special token undefined.

ticket

Sets a ticket on a dump, a field of arbitrary alphanumeric characters purely for being able to associate the dump with a defect tracking system.

unticket

Unsets a ticket on a dump.

analyze

On the specified dumps, runs the specified analyzer, as uploaded via the analyzer subcommand. An analyzer is a shell script that runs against a given dump. The following shell variables are made available in the context of an analyzer:

$THOTH_DUMP: The path of the dump. (This is set to the same value as $MANTA_INPUT_FILE.)
$THOTH_INFO: The path of a local file that contains the JSON info for the dump. The json utility may be used to parse this file.
$THOTH_TYPE: The type of the dump (either crash or core).
$THOTH_NAME: The full name (that is, hash) of the dump.
thoth_set: A shell function that will set the specified property to the specified value or, if no value is specified, standard input.
thoth_unset: A shell function that will unset the specified property on the dump being analyzed.
thoth_ticket: A shell function that will set the ticket on the dump being analyzed to the ticket specified.
thoth_unticket: A shell function that will unset the ticket on the dump being analyzed.

For example, here is an analyzer that looks for a particular stack pattern and -- if it is found -- diagnoses it to be a certain ticket.

#
# This analyzer only applies to core files
#
if [[ "$THOTH_TYPE" != "core" ]]; then
    exit 0
fi

#
# This is only relevant for svc.startd
#
if [[ `cat $THOTH_INFO | json cmd` != "svc.startd" ]]; then
    exit 0
fi

#
# This is only OS-2359 if we have utmpx_postfork in our stack
#
if ( ! mdb -e ::stack $THOTH_DUMP | grep utmpx_postfork > /dev/null ); then
    exit 0
fi

#
# We have a winner! Set the ticket.
#
thoth_ticket OS-2359
echo $THOTH_NAME: successfully diagnosed as OS-2359

Here's an analyzer that sets an fmri property to be that of the SMF_FMRI environment variable:

if [[ "$THOTH_TYPE" != "core" ]]; then
    exit 0
fi

if ( ! pargs -e $THOTH_DUMP | grep -w SMF_FMRI > /dev/null ); then
    exit 0
fi

fmri=`pargs -e $THOTH_DUMP | grep -w SMF_FMRI | cut -d= -f2-`
thoth_set fmri $fmri
echo $THOTH_NAME: $fmri

The output of analyzers is aggregated and displayed upon completion of analyze.

Debugging analyzers

To debug and interactively develop analyzers, use thoth debug and specify both the dump and the analyzer:

% thoth debug 004a8bf33b2cd204903e46830a4f3b23 MANTA-1817-diagnose
thoth: debugging 004a8bf33b2cd204903e46830a4f3b23
 * created interactive job -- 60061666-fdf4-466e-fd9c-d84eb7fbf2de
 * waiting for session... - established
thoth: dump info is in $THOTH_INFO
thoth: analyzer "MANTA-1817-diagnose" is in $THOTH_ANALYZER
thoth: run "thoth_analyze" to run $THOTH_ANALYZER
thoth: any changes to $THOTH_ANALYZER will be stored upon successful exit
bcantrill@thoth #

This results in an interactive shell whereby one can interactively edit the specified analyzer by editing the file referred to by $THOTH_ANALYZER and can test the analyzer by running thoth_analyze. When the shell exits successfully (that is, an exit of 0), the contents of the file pointed to by $THOTH_ANALYZER will be written to the specified analyzer.

Testing analyzers

Once an analyzer works on a single dump using thoth debug, it is recommended to run and debug the new analyzer on a single dump by specifying the dump's complete hash to analyze; once the analyzer is working, it can be run on a larger number of dumps by specifying a broader dump specification to analyze.

analyzer

Uploads stdin to be the named analyzer.

$ thoth analyzer fmri < /var/tmp/fmri.sh
thoth: reading analyzer 'fmri' from stdin
thoth: added analyzer 'fmri'

analyzers

Lists all of the analyzers known to thoth. These are listed as absolute Manta paths that may be retrieved with mget.

$ thoth analyzers
/thoth/stor/thoth/analyzers/MANTA-1579-diagnose
/thoth/stor/thoth/analyzers/OS-1450-diagnose
/thoth/stor/thoth/analyzers/OS-2359-diagnose
/thoth/stor/thoth/analyzers/OS-2359-stacks
/thoth/stor/thoth/analyzers/fmri

Thoth and Triton

For users of Joyent's Triton (née SmartDataCenter), sdc-thoth allows for Thoth to be integrated and run on a regular basis from the headnode. sdc-thoth operates by querying compute nodes for dumps and their corresponding hashes, checking those hashes against Thoth, and uploading any missing dumps through the headnode and into Thoth.

Installation

Running sdc-thoth-install as root on the headnode will install the latest binary on the headnode in /opt/custom, create a thoth user and create the necessary SMF manifest as well as a crontab that runs sdc-thoth in dry-run mode. The latest version can be grabbed via:

curl -k \
  https://us-east.manta.joyent.com/thoth/public/thoth/thoth-sunos-latest.tar.gz | \
  (cd / && tar zxvf -)

Before running the script, you will need to have a running thoth database as described above. Then:

export TRITON_PROFILE=env
export TRITON_URL=https://mycloudapi/
export TRITON_ACCOUNT=$THOTH_USER
export TRITON_KEY_ID=$TRITON_KEY_ID # key ID for that user
export MANTA_URL=https://mymanta... # manta endpoint
/opt/custom/thoth/bin/sdc-thoth-install

After installation, su - thoth, and try running sdc-thoth. If it's working OK, you can edit ./run-thoth to remove the --dry-run flag.

License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Bugs

See https://github.com/joyent/manta-thoth/issues.

TritonDataCenter / manta-thoth

readme

manta-thoth

Installation

Setup

Running Thoth

Introduction

Dump specifications

Special token: `mtime`

Special token: `otime`

Special token: `limit`

Special token: `undefined`

Special specification: `dump=stdin`

Subcommands

upload

info

debug

ls

object

report

set

unset

ticket

unticket

analyze

Debugging analyzers

Testing analyzers

analyzer

analyzers

Thoth and Triton

Installation

License

Bugs

TritonDataCenter / manta-thoth

readme

manta-thoth

Installation

Setup

Running Thoth

Introduction

Dump specifications

Special token: mtime

Special token: otime

Special token: limit

Special token: undefined

Special specification: dump=stdin

Subcommands

upload

info

debug

ls

object

report

set

unset

ticket

unticket

analyze

Debugging analyzers

Testing analyzers

analyzer

analyzers

Thoth and Triton

Installation

License

Bugs

Special token: `mtime`

Special token: `otime`

Special token: `limit`

Special token: `undefined`

Special specification: `dump=stdin`