search

unixwork / xnedit

A fast and classic X11 text editor, based on NEdit, with full unicode support and antialiased text rendering.
Other
83 stars 13 forks source link

Support for large files? #24

Open cesss opened 4 years ago

cesss commented 4 years ago

Thanks a lot for this great effort updating NEdit!! It's been my favourite text editor since the IRIX era!!

BTW, did you enhance its support for large files? (I mean >2GB files). I remember that some years ago I had to edit an automatically generated text file several GBs long, and I couldn't do it with NEdit. Moreover, maybe there's a size limit in the Motif text widget for the largest text it can display, I don't know...

Thanks a lot for enhancing NEdit!!

unixwork commented 4 years ago

I did not add support for large files yet. (X)Nedit uses a custom text widget and it would be possible to use 64 bit integers there. But there is also a problem with the Motif scrollbar, which also uses only 32 bit integers.

Support for large files is definitely something I want to add, however it requires a lot of work and it isn't on the top of my priority list.

dpeterc commented 4 years ago

Maybe Motif scrollbars are not such a big problem. The Motif automatic scrollbars only handle sizes up to 32768. But comes from a limitation of X Window coordinates, since any window can not be larger than this. So drawing area of the scrollbar needs to be smaller than 32768 pixels. Funny enough, even the great olala cross platform Qt has similar limitations. https://forum.qt.io/topic/75082/qimage-max-size Back to Motif/xnedit. If you have custom scrollbars (no drawing area, just clip window and custom handling of scrollbars) then XmNmaximum is an int, that means you can have values up to 4 GB if you only have one character on each line... For me, that is pretty large, but maybe some people want even more.

efa commented 2 years ago

I just tested opening a 4+ GiB file on both Nedit 5.7 and XNEdit. 4503633920 Bytes (binary .iso)

Nedit immediately ask if had to open a binary as found a lot of NULL Once confirmed, after 5" it regularly open the file, and I can scroll to the end. It use 214 MiB of memory and 238 MiB of virtual memory.

XNEdit seems immediately hang, after about 60", when memory allocation reach 2.0 GiB and 2.0 GiB of virtual memory, crash with Segmentation fault.

Note: it also leave some (2 GB) temp files allocated on disk in the root partition, and not freed by Linux 4.19, and after some XNEdit crash, it left my root with 0 bytes free. I cannot find such big files in /tmp, so to avoid restart I had to reduce the reserved space from 5% to 4%: $ sudo tune2fs -m4 /dev/sda8

Update: the big temp files are in:

ls -l /var/lib/systemd/coredump
totale 5895400
-rw-r-----+ 1 root root 2012245044 gen 14 14:46 core.xnedit.1000.b6233b2687b2442ba2eeb2bca6c457b1.24386.1642167985000000.lz4
-rw-r-----+ 1 root root 2012252098 gen 14 14:48 core.xnedit.1000.b6233b2687b2442ba2eeb2bca6c457b1.24477.1642168113000000.lz4
-rw-r-----+ 1 root root 2012250542 gen 14 14:50 core.xnedit.1000.b6233b2687b2442ba2eeb2bca6c457b1.24559.1642168194000000.lz4

thank you systemd

efa commented 2 years ago

update: I just reduced the systemd core size limit by modifying: /etc/systemd/coredump.conf ProcessSizeMax=100M ExternalSizeMax=100M $ sudo systemctl daemon-reload then tested the current git of XNEdit, it do something different: On start seems immediately hang, with an empty window, after about 60" the memory allocation reach 4.7 GiB and 4.7 GiB of virtual memory, then remain so forever. I wait 5 min then killed it. No core file is created.

unixwork commented 2 years ago

Opening a 4+ GiB file in old nedit is not possible, because is uses 32bit internally, so you get an integer overflow. It also loads the whole file into memory, so memory usage must be at least as big as the file.

XNEdit currently has the same limitations and some extra problems. The main problem is, that nedit has many performance optimizations, that don't work with unicode. It assumes that one byte is one character, which doesn't work with utf8. Also nedit assumes that when using a fixed width font, all characters have the same width. This sounds reasonable, however it also doesn't work in XNEdit, because it supports fallback fonts in the case when a character is not available in the current font. This happens for example when you have chinese characters in the text.

So in short, to have the best unicode experience, I had to drop some optimizations. I will try to improve the performance some day, however it will be a lot of work.

efa commented 2 years ago

Understand, the always used rule 1 char = 1 bytes let optimize much, but lacking multibyte chars in our time is a mess. So let accept some limitations.

As reported, Nedit 5.7 on my Debian 10, marked "NEdit released by Debian (1:5.7-2)", seems can open files with 4+ GiB without crash. Its allocation in that case is only 214 MiB of memory and 238 MiB of virtual memory.

unixwork commented 2 years ago

The limit in nedit is exactly 2147483647 bytes. After that you have an integer overflow and weird things can happen. Sometimes it doesn't crash, but sometimes it does. On my system, opening a file that is exactly 2Gb (2147483648 bytes) will crash.

If it doesn't crash, it will not show the complete file. In your case, probably only the first 200mb.

comperem commented 1 year ago

Hello,

I just compiled both Nedit and XNedit from source on Ubuntu 22.04 using the 5.19 kernel. All the libraries were installed with .deb packages to provide all dependencies:

  1. tar xzvf xnedit-1.4.1.tar.gz
  2. sudo apt-get install libxt-dev
  3. sudo apt-get install libmotif-dev
  4. sudo apt install libxpm-dev
  5. sudo apt-get install libpcre3-dev
  6. make linux

I'm still getting the 2Gb limitation.

Nedit barfs with a memory error: NEditMalloc(18446744071735093865) failed

XNedit says "File is too large to edit" with a dialogue box: image

Ack.

Nedit and XNedit are some of the only text editors that can handle large text files.

What is the priority for upgrading to 64bit code to achieve larger file editing???

Marc

unixwork commented 1 year ago

Unfortunately this is not something that can be easily fixed. Priority for this was pretty low.

The 32 bit integers are not the only problem. In order to support unicode, I had to remove some of nedit's optimizations, because they just don't work with unicode. As a result, currently xnedit doesn't even work well with large files < 2 Gb.

I can focus on this more, but don't expect a solution in the near future.

comperem commented 1 year ago

Thank you for your answer.

It's understandable to prioritize this at a low level. Unicode is not important for me, but it's understandable to include that as an editor capability.

Are there other editors, console or GUI, that can open a multi-Gb file?

Marc

On Mon, Jul 3, 2023 at 2:16 PM unixwork @.***> wrote:

Unfortunately this is not something that can be easily fixed. Priority for this was pretty low.

The 32 bit integers are not the only problem. In order to support unicode, I had to remove some of nedit's optimizations, because they just don't work with unicode. As a result, currently xnedit doesn't even work well with large files < 2 Gb.

I can focus on this more, but don't expect a solution in the near future.

— Reply to this email directly, view it on GitHub https://github.com/unixwork/xnedit/issues/24#issuecomment-1618970583, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG7MPYV3HINKNOYFTHXLJFLXOMEABANCNFSM4P6S4I6A . You are receiving this because you commented.Message ID: @.***>