Index entry is left behind in zar.json after "zar rm"

[philrz]

Repro is with zq commit 41a4641 and steps derived from the zar README.

Staring with an empty $ZAR_ROOT, if I follow the steps:

$ zq zng/*.gz | zar import -s 25MB -
$ zar index -q -o custom.zng -k id.orig_h -z "count() by _path, id.orig_h | sort id.orig_h"
$ zar rm custom.zng

The microindex files are removed as expected:

$ tree -s logs
logs
├── [        320]  20180324
│   ├── [    4425007]  1521911772.980384.zng
│   ├── [         64]  1521911772.980384.zng.zar
│   ├── [   25001925]  1521912075.114273.zng
│   ├── [         64]  1521912075.114273.zng.zar
│   ├── [   25007413]  1521912507.399929.zng
│   ├── [         64]  1521912507.399929.zng.zar
│   ├── [   25005195]  1521912990.158766.zng
│   └── [         64]  1521912990.158766.zng.zar
└── [        679]  zar.json

5 directories, 5 files

But I still see an index entry in zar.json:

$ cat $ZAR_ROOT/zar.json | zq -f ndjson 'cut indexes' - | jq .
{
  "indexes": {
    "custom": {
      "zng": {
        "path": "custom.zng",
        "type": "zql"
      }
    }
  }
}

So now I can do:

$ zar find -z -x custom.zng 10.164.94.120 | zq -t -
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x169a0bc]

goroutine 20 [running]:
github.com/brimsec/zq/zdx.(*Reader).Path(...)
    /Users/phil/other-zq/zdx/reader.go:121
github.com/brimsec/zq/archive.search(0x1a6b2a0, 0xc00018a740, 0xc000314770, 0xc0001088a0, 0xc000034f40, 0x4, 0x0, 0x0, 0x0, 0xc000034f47, ...)
    /Users/phil/other-zq/archive/finder.go:126 +0x18c
github.com/brimsec/zq/archive.Find.func2.1(0xc0001088a0, 0x1a6b2a0, 0xc00018a740, 0xc00031a2a0, 0xc000034f40, 0x4, 0x0, 0x0, 0x0, 0xc000034f47, ...)
    /Users/phil/other-zq/archive/finder.go:99 +0x191
created by github.com/brimsec/zq/archive.Find.func2
    /Users/phil/other-zq/archive/finder.go:97 +0x1bf

Granted, by a strict read of zar help rm, it's doing what it promised. But I assume the additional cleanup is necessary to avoid such crashes.

[alfred-landrum]

I believe upcoming changes spawned from work in #1183 are going to alter how we store the data & availability of indexes, so I'm moving this to the backlog.

[philrz]

I checked in on this one, since things have changed a bit and I wanted to see if this coincidentally got fixed. The tl;dr is that it's still with us.

As of zar commit d8f8855 that came with overhaul PR #1330, the indexes section of zar.json is now gone. Following the original repro steps and stopping just before the zar rm custom.zng step, it looks like:

{
  "version": 0,
  "data_path": ".",
  "data_sort_direction": false,
  "log_size_threshold": 25000000
}

But then the crash is similar to what we saw before:

$ zar rm custom.zng
file:///Users/phil/logs/zd/20180324/d-1iR05l4KpUnpTqHlCHqu3wdZznE.zng.zar/custom.zng: removed
file:///Users/phil/logs/zd/20180324/d-1iR05jwpKeMyIHTkvcrSfrr7Ceg.zng.zar/custom.zng: removed
file:///Users/phil/logs/zd/20180324/d-1iR05uHdW2v4iXIyCu7CbEgdOGe.zng.zar/custom.zng: removed
file:///Users/phil/logs/zd/20180324/d-1iR05o9uFc8DHAjcrYQ4fLRLiHh.zng.zar/custom.zng: removed

$ zar find -z -x custom.zng 10.164.94.120 | zq -t -
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16b98d6]

goroutine 32 [running]:
github.com/brimsec/zq/microindex.(*Reader).Path(...)
    /Users/phil/work/zq/microindex/reader.go:134
github.com/brimsec/zq/archive.search(0x1aaf9e0, 0xc0002d6900, 0xc0000e0730, 0xc00009e8a0, 0xc0000dcaa0, 0x4, 0x0, 0x0, 0x0, 0xc0000dcaa7, ...)
    /Users/phil/work/zq/archive/finder.go:128 +0x1a6
github.com/brimsec/zq/archive.Find.func2.1(0xc00009e8a0, 0x1aaf9e0, 0xc0002d6900, 0xc0000b9e20, 0xc0000dcaa0, 0x4, 0x0, 0x0, 0x0, 0xc0000dcaa7, ...)
    /Users/phil/work/zq/archive/finder.go:101 +0x191
created by github.com/brimsec/zq/archive.Find.func2
    /Users/phil/work/zq/archive/finder.go:99 +0x189

However, more recently, starting with commit 56ed80d, the zar index phase of the repro steps no longer works due to #1415.

[philrz]

I now realize that what remains of this problem is super simple to describe, so all the history I've documented here isn't serving us well. I've opened #1419 to capture the essence of what's now wrong, and will close this one.