panic: Index out of range

[steve-offutt]

Great work on this parser! Really enjoying it.

I am using the go-ese package to dump all tables contained in a WebCacheV01.dat file. The file was obtained with the following command:

esentutl.exe /y C:\Users\joe\AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.dat /vss /d d:\WebCacheV01.dat

The code that I run that generates the panic is

    tables := catalog.Tables.Keys()
    for _, table := range tables {
        err = catalog.DumpTable(table, func(row *ordereddict.Dict) error {
            _, err := json.Marshal(row)
            if err != nil {
                return err
            }
            return nil
        })
    }

My goal is to use this package as part of a different tool.

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/Velocidex/go-ese/parser.NewValue(0xc00008e510, 0xc000005600, 0x1b1, 0x63b240, 0x0, 0x0, 0x0)
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/parser/pages.go:26 +0x14b
github.com/Velocidex/go-ese/parser.GetPageValues(0xc00008e510, 0xc0000055e0, 0x1b1, 0xc00029de61, 0x0, 0xc00029dc98)
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/parser/pages.go:70 +0x2d5
github.com/Velocidex/go-ese/parser._walkPages(0xc00008e510, 0x1b1, 0xc00029de88, 0xc000005540, 0x63bb40, 0x0)
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/parser/pages.go:255 +0x185
github.com/Velocidex/go-ese/parser._walkPages(0xc00008e510, 0x1af, 0xc00029de88, 0xc000005540, 0xc0000ae2d0, 0x1)
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/parser/pages.go:272 +0x381
github.com/Velocidex/go-ese/parser.WalkPages(...)
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/parser/pages.go:237
github.com/Velocidex/go-ese/parser.(*Catalog).DumpTable(0xc000004600, 0xc000011ae0, 0xb, 0x53c488, 0x1, 0xc)
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/parser/catalog.go:341 +0x18b
main.main()
        C:/Users/joe/go/src/github/steve-offutt/ese-parse/main.go:30 +0x2e2
exit status 2

A quick look at the code in parser/pages.go it appears that buffer := make([]byte, int(tag.ValueSize(ctx))) is creating a zero length buffer. Unfortunately I cannot share the WebCacheV01.dat file that I am using but will try to create it on a separate machine that I can share.

As a workaround I added in a check in pages.go to check the length of the the buffer variable. If it is greater than zero, append to result. Something like this is what I have been doing:

func GetPageValues(ctx *ESEContext, header *PageHeader, id int64) []*Value {
    result := []*Value{}

    // Tags are written from the end of the page
    offset := ctx.PageSize + header.Offset - 4

    for tag_count := header.AvailablePageTag(); tag_count > 0; tag_count-- {
        tag := ctx.Profile.Tag(ctx.Reader, offset)
        value_offset := header.EndOffset(ctx) + int64(tag.ValueOffset(ctx))

        buffer := make([]byte, int(tag.ValueSize(ctx)))
        numBytes, err := ctx.Reader.ReadAt(buffer, value_offset)

        if err != nil {
            fmt.Println(err)
        }
        if len(buffer) > 0 {
            result = append(result, NewValue(ctx, tag, id, buffer))
        } else {
            fmt.Println("*********************************************************")
            fmt.Printf("Tag with zero size: %+v\n", tag)
            fmt.Printf("Read %d bytes\n", numBytes)
            fmt.Printf("Size of buffer %d\n", len(buffer))
            fmt.Printf("int(tag.ValueSize): %d\n", int(tag.ValueSize(ctx)))
            fmt.Printf("tag.ValueSize: %+v\n", tag.ValueSize(ctx))
        }
        // result = append(result, NewValue(ctx, tag, id, buffer))

        offset -= 4
    }

    if DebugWalk {
        fmt.Printf("Got %v values for page %v\n", len(result), id)
    }
    return result
}

I am not up to speed on the ESEDB database structure. Would this workaround result in data loss since I am essentially dropping a result?

[steve-offutt]

Also, the eseparser.exe also fails with .\eseparser.exe dump D:\WebCacheV01.dat Container_1

PS C:\Users\joe\go\src\github.com\Velocidex\go-ese> .\eseparser.exe dump D:\WebCacheV01.dat Container_1
panic: runtime error: index out of range

goroutine 1 [running]:
www.velocidex.com/golang/go-ese/parser.NewValue(0xc000099230, 0xc0002947a0, 0x1b1, 0x81cfb0, 0x0, 0x0, 0x0)
        C:/Users/joe/go/src/www.velocidex.com/golang/go-ese/parser/pages.go:26 +0x14b
www.velocidex.com/golang/go-ese/parser.GetPageValues(0xc000099230, 0xc000294780, 0x1b1, 0xc0000abe19, 0x0, 0xc0000abc50)
        C:/Users/joe/go/src/www.velocidex.com/golang/go-ese/parser/pages.go:61 +0x2d5
www.velocidex.com/golang/go-ese/parser._walkPages(0xc000099230, 0x1b1, 0xc0000abe40, 0xc0002946e0, 0x81d900, 0x0)
        C:/Users/joe/go/src/www.velocidex.com/golang/go-ese/parser/pages.go:245 +0x185
www.velocidex.com/golang/go-ese/parser._walkPages(0xc000099230, 0x1af, 0xc0000abe40, 0xc0002946e0, 0xc0000d0460, 0x78ad4cdfc872cb01)
        C:/Users/joe/go/src/www.velocidex.com/golang/go-ese/parser/pages.go:262 +0x381
www.velocidex.com/golang/go-ese/parser.WalkPages(...)
        C:/Users/joe/go/src/www.velocidex.com/golang/go-ese/parser/pages.go:227
www.velocidex.com/golang/go-ese/parser.(*Catalog).DumpTable(0xc000078a00, 0xc0000820b0, 0xb, 0x648038, 0x17, 0x0)
        C:/Users/joe/go/src/www.velocidex.com/golang/go-ese/parser/catalog.go:341 +0x18b
main.doDump()
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/bin/dump_table.go:33 +0x131
main.init.1.func1(0x638fa4, 0x4, 0x0)
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/bin/dump_table.go:49 +0x6e
main.main()
        C:/Users/joe/go/src/github.com/Velocidex/go-ese/bin/main.go:32 +0x1bb

[scudette]

I think this may be related to long values which are not present in the page. We dont really support those yet. I think we just need to add a check here to make sure we dont access the buffer if it is empty:

https://github.com/Velocidex/go-ese/blob/0c80dd255d97b398e400e729ee5752b277d43d3d/parser/pages.go#L24

So maybe:

       if ctx.Version == 0x620 && ctx.Revision >= 17 && 
                ctx.PageSize > 8192 && len(buffer) > 0 {
        result.Flags = uint64(buffer[1] >> 5)
        buffer[1] &= 0x1f
    } else {
        result.Flags = uint64(tag._ValueOffset()) >> 13
    }

This only happens on large page files and obviously if the value is empty anyway the flags are not stored.

[scudette]

This should be fixed - please reopen if still crashing.