thread 'main' panicked at 'index out of bounds: the len is 3600 but the index is 3600

[jonahsnider]

/mnt/n/AppData/Local/Bisq/runtime/include/win32/bridge                                      thread 'main' panicked at 'index out of bounds: the len is 3600 but the index is 3600', /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libcore/slice/mod.rs:2842:10

Running in WSL 2 with rustc 1.43.0 (4fb7144ed 2020-04-20).

[imsnif]

Hey @pizzafox, sorry for this experience and thanks for reporting this. Mind helping me out with some questions to debug this? Does this happen for every folder you scan? Does the behaviour change if you change the terminal window size? (as in, does it crash earlier, later or not at all with a smaller terminal window?) In the folder you scan, do you happen to have folder/file names with non-latin characters?

[Freaky]

Did you notice the reported sizes? One of the last figures seen is 11.6 EB - not far off overflowing a u64.

[jonahsnider]

This is about 1 TB, definitely not that big. I assume Diskonaut is getting screwed up with a weird Windows FS thing.

[imsnif]

Thanks for pointing it out, @Freaky. I indeed hadn't noticed. @pizzafox - I suspect you're right and that since we're trying to read hard-disk blocks, something in the WSL is throwing us off (and causing us to report huge file sizes which are clearly incorrect... this might be what's causing the crash in the end as @Freaky pointed out). But this is of course just a guess.

I think when we implement https://github.com/imsnif/diskonaut/issues/56, it might solve this issue. I hope to get to it soon (unless someone does it before me), and will give you a ping once it's done and released so you can give it a shot if you want to.

[Freaky]

I can reproduce the crash by adding 2^62 to file sizes:

 thread 'main' panicked at 'Trying to access position outside the buffer: x=144, y=20, area=Rect { x: 0, y: 0, width: 106, height: 44 }', /home/freaky/.cargo/registry/src/github.com-1ecc6299db9ec823/tui-0.9.5/src/buffer.rs:245:9

Sadly backtraces are currently broken on FreeBSD, so here's one from gdb:

#27 0x000000000148d1dd in std::panicking::begin_panic_fmt () at src/libstd/panicking.rs:385
#28 0x000000000143c756 in tui::buffer::Buffer::index_of (self=0x7fffffffdad0, x=144, y=20) at /home/freaky/.cargo/registry/src/github.com-1ecc6299db9ec823/tui-0.9.5/src/buffer.rs:245
#29 0x000000000119d64b in tui::buffer::Buffer::set_stringn (self=0x7fffffffdad0, x=144, y=20, string=..., width=18446744073709551615, style=...)
    at /home/freaky/.cargo/registry/src/github.com-1ecc6299db9ec823/tui-0.9.5/src/buffer.rs:319
#30 0x000000000119d576 in tui::buffer::Buffer::set_string (self=0x7fffffffdad0, x=144, y=20, string=..., style=...) at /home/freaky/.cargo/registry/src/github.com-1ecc6299db9ec823/tui-0.9.5/src/buffer.rs:303
#31 0x0000000001216ccb in diskonaut::ui::grid::draw_rect::draw_tile_text_on_grid (buf=0x7fffffffdad0, tile=0x806e5f480, selected=false) at src/ui/grid/draw_rect.rs:197
#32 0x000000000124c5a1 in <diskonaut::ui::grid::rectangle_grid::RectangleGrid as tui::widgets::Widget>::render (self=..., area=..., buf=0x7fffffffdad0) at src/ui/grid/rectangle_grid.rs:74
#33 0x000000000123c152 in tui::terminal::Frame<B>::render_widget (self=0x7fffffffbd68, widget=..., area=...) at /home/freaky/.cargo/registry/src/github.com-1ecc6299db9ec823/tui-0.9.5/src/terminal.rs:65
#34 0x0000000001259acc in diskonaut::ui::display::Display<B>::render::{{closure}} (f=...) at src/ui/display.rs:128
#35 0x000000000123cbcf in tui::terminal::Terminal<B>::draw (self=0x7fffffffda98, f=...) at /home/freaky/.cargo/registry/src/github.com-1ecc6299db9ec823/tui-0.9.5/src/terminal.rs:187
#36 0x0000000001259214 in diskonaut::ui::display::Display<B>::render (self=0x7fffffffda98, file_tree=0x7fffffffd9f8, board=0x7fffffffd990, ui_mode=0x7fffffffd938, ui_effects=0x7fffffffdb30)
    at src/ui/display.rs:48
#37 0x0000000001210211 in diskonaut::app::App<B>::render (self=0x7fffffffd938) at src/app.rs:81
#38 0x00000000012100ed in diskonaut::app::App<B>::render_and_update_board (self=0x7fffffffd938) at src/app.rs:71
#39 0x00000000012103a0 in diskonaut::app::App<B>::start_ui (self=0x7fffffffd938) at src/app.rs:103
#40 0x0000000001261627 in diskonaut::messages::instruction::handle_instructions (app=0x7fffffffd938, receiver=...) at src/messages/instruction.rs:53
#41 0x000000000121007d in diskonaut::app::App<B>::start (self=0x7fffffffd938, receiver=...) at src/app.rs:65
#42 0x000000000123f7d7 in diskonaut::start (terminal_backend=..., keyboard_events=..., path=...) at src/main.rs:226
#43 0x0000000001221aff in diskonaut::try_main () at src/main.rs:74
#44 0x0000000001221490 in diskonaut::main () at src/main.rs:55

Really, it's not our fault if Windows is reporting weird sizes, but it is that we're overflowing. If anything, --apparent-size could make this more likely, because it's a lot easier to make a file that's apparently gobsmackingly massive than it is to make one physically massive.

Sizes should probably be summed using checked maths (checked_add, saturating_add, etc), and/or perhaps the sums should be u128.

[imsnif]

I see. Using checked_add/saturating_add makes sense. Probably combined with handling an overflow with some sort of descriptive error message. Moving stuff to u128 is something I'll have to look into. Mostly the implications of the memory footprint, since we're going to have to change each tree node. I'm assigning this to myself and will take a closer look soon.

[imsnif]

Actually... does it even make sense to move stuff to u128? The max value of a u64 is 18446744073709551615. That's ~18 million terabytes. Is this even relevant to storage one would want to run diskonaut on? (please do correct me if my math is wrong :) )

[Freaky]

It's perfectly valid to have huge files. I just made my diskonaut folder ~37EB:

.rw-r--r-- 9.2E freaky freaky 29 Jun 21:48 huge1
.rw-r--r-- 9.2E freaky freaky 29 Jun 21:49 huge2
.rw-r--r-- 9.2E freaky freaky 29 Jun 21:49 huge3
.rw-r--r-- 9.2E freaky freaky 29 Jun 21:49 huge4

It'd be nice to be able to Just Work, which is more than I can say for... any other du-alike I've tried. GNU du gives up and just says Infinity, BSD du and other tools just overflow and give meaningless results.

[imsnif]

Alright. So let's try with u128 and see if it doesn't take up too big of a memory footprint. Would you like to work on this, @Freaky ?

[imsnif]

Hey @pizzafox, in 0.9.0 there's an --apparent-size flag that shows the actual file sizes rather than counting how many blocks they take on disk. I wonder if running with this flag still causes this issue?

[jonahsnider]

The --apparent-size flag fixed the issue.

Using v0.9.0 without the flag no longer crashes (at least, not after 45 seconds). The total size got up to 100 zettabytes before I quit the program. Perhaps Diskonaut should suggest using the --apparent-size flag when the size gets extraordinarily large?

[imsnif]

Thanks @pizzafox! The not crashing is thanks to @Freaky's work, I'd say.

I think somehow suggesting using the flag is a good idea. I'll open an issue in the next few days with some thoughts.