"Memory full" message when starting [Using a terminal with more than 127 lines can crash]

[sjuswede]

123 will start fine in a terminal window larger than 80x24. I normally run a 100x50 window and that is no problem at all. 100x80 also works well.

But when the window is larger, just slightly outside those dimensions, 123 fails on startup with the message "123: Memory full".

I'm not sure this can be changed, as we don't have the source code. But a note in the documentation would be helpful for those, like me, who run ridiculously large screens with ridiculously small fonts.

If the limit can be extended that would of course be even nicer. 123 runs perfectly fine in these larger terminal sizes otherwise.

[taviso]

You're right - I traced through the error, there is a function init_showme called during startup with a hardcoded limit of 90 rows. It just fails for anything higher. I think this is no problem - we can redirect functions to new versions without limits, but it's not obvious why it has that limit.

Usually it means there is a static buffer somewhere where someone did char buf[MAX_ROWS], but if we can find it we can make it bigger or allocate it dynamically. I guess I can just try removing the limit and see what breaks!

[taviso]

Hmm - I tried it, nothing seems to break... but that limit must have been there for some reason!

I guess I can just commit it, because I'm only changing how it works for people who would have seen "Memory full" anyway. I'd appreciate it if you can test it and see if you can spot something not working properly!

If you git pull it should get the patch.

[sjuswede]

Tried it in a 130x160 window, and I get:

[1] 114712 segmentation fault (core dumped) 123

[taviso]

Ah, I see the problem - when it's initializing it's internal screen buffer in setup_screen_mem() it uses an int8_t counter to iterate over all the lines, so doesn't handle a line number greater than INT8_MAX (i.e. 127). Those lines get left as NULL.

The rest of the routine seems correct, I think just rewriting this to use a bigger type for the counter will work, and it will allocate them and use them just fine.

For future reference I had to do this to reproduce, because I can't make a terminal big enough to repro and still read the text in gdb 😆

(gdb) set environment COLUMNS=160
(gdb) set environment LINES=130
(gdb) r

Okay, let me see if I can rewrite this routine without breaking it.

[taviso]

Oh - unfortunately that's a local symbol, so I will need to fix #71 first.

[taviso]

Okay, I fixed #71.

I have a hacky patch that seems to work for fixing this crash, I just want to clean it up a bit and I'll commit it.

[taviso]

Alright, can you try this patch instead? I think we're getting closer to this working, I rewrote the routine with a size_t loop counter, so that should handle terminals with line counts up to 4,294,967,295 😂

[sjuswede]

Now it won't crash on startup, and it looks good at smaller terminal sizes. Works fine in 160x130.

I maximized my urxvt terminal and tested, that gave me 477x154 and tried, and it starts fine. However, I don't get any cell column "headers" past the X one. Y and out are not drawn. They get drawn and labeled if I move the active cell out past X and everything works otherwise, from what I can tell. Rows are drawn as expected.

I will test with a few larger spreadsheets and see if things behave as normal in other ways.

[sjuswede]

OK, tested with some spreadsheets, and... yeah, no. Not working properly.

I imported half a million rows from a csv file, and started scrolling. It looked normal until I reached the bottow row and the screen started to scroll. Then I got this. The data is on the left. What is shown on the right does not exist. It's empty columns, but they show the same data mirrored.

[taviso]

Okay, well we're getting closer 😃

I'll try to find the problem, but it would be really helpful if you can figure out the smallest possible resolution that doesn't work - i.e. add one more line and it doesn't work properly.

[sjuswede]

Hard to say. At 224 columns it will display to W. At 225 columns it will display everything up to X. Past that it doesn't display anything else until I move the active cell right, and the top display always looks correct (as in, it will be "Z", then "AA", then "AB" etc.). And if I enter data there it seems to work. But when I scroll... things go very pearshaped and the data I have entered out past the "Y" column will not be redrawn, and instead the screen will display a reflection of the data out on that side.

Not sure if it is any help, but the data which normally shows in "A" will be "mirrored" as if it is in "AC". I expect I could do some basic arithmetic and figure out how many columns that means, but I'm not mentally up to that today.

If there is anything specific you need, I can try to find it out.

[sjuswede]

I just discovered something very interesting. I tried a 20x354 window, to see if I could make a smaller window for screenshots and testing, and... this is what was shown.

That is, the drawing of the top line ended a lot earlier. But, when I tested adding some data and scrolling, it behaved just like previously.

The text got "mirrored" under A -> AC and so on.

So I expect the columns I measured out are irrelevant. The initial drawing problem is something else. But the "mirroring" is at least consistent.

I should add, I got the top to display the column names by moving the active cell out that side a bit and then back. It doesn't do that on its own, whether or not I scroll.

[taviso]

Thanks for the notes, I think some index must be wrapping - i suppose I can just put a watchpoint on the screen buffer and figure out what code is doing that. I'm confident we can get this working, it might take a few attempts though!

[taviso]

I think the problem is the data structure it uses to keep track of what regions are dirty (and so need to be updated) has some hardcoded limits, internally it seems to be maintaining the screen buffer correctly but just isn't sure which parts need to be flushed to the screen.

Hmm - so those routines either need to be re-implemented.... or... in 2022, do we still need it? I guess I will try just redrawing the whole screen if anything is dirty, maybe terminals are fast enough? If not, these routines are not too complicated, I can just rewrite them.

[sjuswede]

Redrawing all on dirty will make it a bit of a pain over a slower ssh, like when running things over mosh from an iPad. And on older repurposed machines (like my PIII 600, which is an Emacs beast). And over serial connection. And my poor laptop battery!

Though to be honest, I expect it could be fast enough. :)

[taviso]

I'm still working on this - I understand the problem now, but will have to patch several routines to fix it! I think it's no problem, we can just bump up the columns it keeps track of.

What is a sane maximum, 512?

[sjuswede]

If I set column with to 1 to display something with only 1 character per cell. I will at present hit 470 columns or so on a maximized terminal. I have been considering using my 65" 4K TV and a 10 point font to get more on screen, right now I'm at 14 points on a 4K monitor, which will give me some more characters at the time. But frankly, that is ridiculous.

If we are to take into account such ridiculousness, I would argue 1024 is a sane maximum. If we don't care about those who really should be doing something less silly with their systems, 512 is perfectly fine.

[taviso]

I think we can make it work, or at least we can try! I haven't got a working patch yet, but I'm getting there.

[taviso]

Believe it or not, I spent an hour on this today and I think I have a working patch. It seems to work, and the tests pass.

I need to clean it up before I commit it, and I don't 100% understand some of the logic, so I need to convince myself it's correct first lol.

[taviso]

I got it working at 1024x512, if someone needs more than that then I'll add it, but I want to see a video of them working like that first because that would be amazing! 😂

[sjuswede]

I take it that is not in main as of now? I did a pull and tested, and it behaves the same as before.

[taviso]

Yep, I just want to clean up the code a little bit first, it's on the way 😃

[taviso]

Alright.. I'm about to commit it. I'm not 100% sure if it's correct, but I can't find anything that doesn't work. I'd really appreciate some testing.

[sjuswede]

I tested out looking through some big files, and that works fine. I added a few functions and no issues. Zooming around is pretty smooth. However, when I was trying with some wide data files I got the error "Input line too long -- Press HELP"

I can import data with lines which are 511 characters no problem. But over that I get the error.

[taviso]

Thanks for testing! I added some tests with big screen sizes to make sure it doesn't break in future. The import length limit looks solvable, filed as #92

[sjuswede]

Did some more testing with some big spreadsheets. And it unfortunately seems something is badly broken.

When I scroll down and up, things are fine. But scrolling sideways makes everything a mess. Content is not even remotely correct, it becomes a mishmash of content from cells all over.

I have a test spreadsheet where all cells are filled with proper english words, and after scrolling sideways back and forth a couple of times, it shows this.

[taviso]

Hmm, at what terminal size? I tested at 263x50, and tried to reproduce by making a .csv like this:

$ sed -e 's/^/"/g' -e 's/$/", /g' /usr/share/dict/words | tr -d '\n' | fold -s -w 500 > import.csv

It was slow to import (I really should fix that, it uses too small a read buffer), but it did seem to work okay scrolling up/down/left/right. Can I have your .wk3?

[taviso]

Nevermind - I can reproduce at larger sizes, 380x72 works.

Hmm - there must be a uint8_t still in there somewhere, I thought I had found them all.

[taviso]

I think I see the problem, I can fix it.

This was bad luck, I had some tests that scrolled huge sheets up/down to make sure it worked, but no tests for left/right, too bad or I would have noticed this 😃

I'll add those tests and commit a fix.

[taviso]

I think I've solved it, can you try now?

[sjuswede]

With the (slightly insane) spreadsheets I test with, and a 470x200 terminal, it seems to be working fine. Right now that is the largest I can do with ease. I will set a larger one up later in the week, but for now this looks solved!

[taviso]

Nice, thanks for testing! It was a fun challenge to get this working, whoever wrote this code had probably never seen more than 160 columns haha.

🥳

[sjuswede]

I'm a bit sad that IV is the widest a spreadsheet can get. Increasing that limit would be nice, of course. Here's a screen shot from my testing, which seems to show it works fine.

This is column width 2, at 477x154. I will try to get access to a larger 4k screen during the week.

[taviso]

Haha that's really impressive! I think increasing the column limit seems really hard.. but you can use split windows to see two sheets simultaneously!

Add a new sheet, then go to the middle and try /Worksheet Window Vertical, you can switch between windows with F6.