xclip doesn't work well with large buffers

[rogpeppe]

xclip does not reliably copy/paste large (>1MB) buffers. Here's a sample transcript.

% dd 'if=/dev/zero' 'bs=2000' 'count=1000' | tr '\000' 'a' | xclip -i 
1000+0 records in
1000+0 records out
2000000 bytes (2.0 MB, 1.9 MiB) copied, 0.0177519 s, 113 MB/s
% xclip -o | wc
      0       1 1048575
% xclip -o | wc
# hangs up - need to manually interrupt it.
^?19708: signal: interrupt
% 
% xclip -o | wc
Error: target STRING not available
      0       0       0
% 

[sorbing]

How to fix it?

[bodqhrohro]

I have a similar issue.

At my job I often need to copy large JSON output from browser devtools and save it to a file. xclip -selection clipboard -o is great for this task, but starting from some size (sometimes even less than 1 MB) it does not reliably save the whole file. Most of the times it truncates the output at some size, less often it is empty, sometimes xclip just hangs. And only rarely it outputs the whole JSON successfully, so I need to repeat the command many times and validate the output file after each trial.

Pasting to an editor seemed to be not a reliable option too. I use Vim without X clipboard support as a main editor, so pasting a large string from clipboard through a terminal in insert mode is a good way to totally hang it ;-)

GtkSourceView based editors don't play well with large clipboard content; for some reason, even if word breaking is enabled, they try to display it in one string, and navigation is broken. Even if I tried to save the pasted string to file in such a state, it got truncated too. So I thought this is a X.Org issue: not common one because most of users have some clipboard manager that affect it, but I prefer not running any for security concerns.

But today I tried to run AkelPad editor under Wine. And pasting a large JSON works great here. It works immediately and reliably pastes in the whole JSON, while xclip truncates the same one.

[NXij]

This behavior takes place when piping a large image into xclip aswell. It has something to do with the TTY buffer size limit. xsel, though, does not inhibit this problem.

nx@niri ~> dd 'if=/dev/zero' 'bs=2000' 'count=1000' | tr '\000' 'a' | xclip -i 1000+0 records in 1000+0 records out 2000000 bytes (2.0 MB, 1.9 MiB) copied, 0.00664444 s, 301 MB/s nx@niri ~> X Error of failed request: BadWindow (invalid Window parameter) Major opcode of failed request: 18 (X_ChangeProperty) Resource id in failed request: 0x3000026 Serial number of failed request: 18 Current serial number in output stream: 18

nx@niri ~> dd 'if=/dev/zero' 'bs=500' 'count=1000' | tr '\000' 'a' | xclip -i 1000+0 records in 1000+0 records out 500000 bytes (500 kB, 488 KiB) copied, 0.00313602 s, 159 MB/s nx@niri ~>

[hwangcc23]

This seems to be the problem of the INC mechanism. I may need some time to debug the finite state machine in xcin().

[hwangcc23]

I found one limitation and one problem.

1). Limitation: The implementation of the INCR mechanism in xcin() can only responses ONE requestor at one time. p.s. When running the command dd 'if=/dev/zero' 'bs=2000' 'count=1000' | tr '\000' 'a' | ./xclip -i, xclip -i is in the XCLIB_XCIN_INCR context of xcin() for the requester A (whose id is 0x200001 on my environment). Later, requests from another requestor B xclip -o (whose id is 0x4a00001 on my environment) are all ignored by xclip -i.

2). Problem: When xcin() is in the XCLIB_XCIN_INCR context, it doesn't send the SelectionNotify event back to the requester if it cannot handle the request (for example, ignore non-property events). This causes xclip -o hang up.

I think I can fix the problem first such that other X11 clients (such as xclip -o) will not hang up or time out. But eliminating the limitation requires a re-implementation (for example, create a new context for every requestor). If there is another solution or suggestion, please help. @astrand

[hwangcc23]

astrand said he is very short on time now, it will be a little bit difficult to merge this right away. Can anyone help try out my fix? Thanks in advance.

[mckellyln]

I seem to get xclip running forever after this change. There are a few requestors and finished is 1 for one of them but then it loops forever due to the others.

[hwangcc23]

@mckellyln How to reproduce the problem you met? Possible to provide a sample transcript?

[mckellyln]

I think I just used the -quiet -l 1 on the cmdline to ask for 1 event in the foreground, so something like: echo "test" | xclip -quiet -in -selection clipboard -l 1 If I add some debug prints - % echo "test" | ./xclip -quiet -in -selection clipboard -l 1 Waiting for one selection request. Waiting for a selection request. got a SelectionRequest ... got a SelectionRequest ... got a PropertyNotify ... got a SelectionRequest ... got a SelectionRequest ... got a SelectionRequest ... got a SelectionRequest ... got a SelectionRequest ... got a SelectionRequest ... got a SelectionRequest ... and on ...

[mckellyln]

I posted this in the other commit flow - I think if sloop > 0 then we need to always break out of while() when finished is true ?

[hwangcc23]

@mckellyln I cannot reproduce the problem using the sample you provided (echo "test" | xclip -quiet -in -selection clipboard -l 1). But I get your point. I will submit a new issue for discussion and try to reproduce the problem.

[mckellyln]

@hwangcc23, @astrand I think we can close this now ?

[Dzhakhar]

what is eventually the solution and the best tool to paste large amount of data into file? Guys help me :)

[mckellyln]

Can you provide an example like above that does not work ? How large is your data ?

[wjtk4444]

I have the same problem. Occurs only with -selection clipboard xclip -selection c -t image/png image.png returns

X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  18 (X_ChangeProperty)
  Resource id in failed request:  0x2000283
  Serial number of failed request:  21
  Current serial number in output stream:  21

for files >= 1MiB (more or less)

I'm using xclip 0.13-2 from Arch Linux repo, but I also tried building current master and it doesn't seem to work either.

[mckellyln]

ok, thanks. I will try to look into it soon.

[sandsmark]

@wjtk4444 did you get that before the current fix?

And I'm not convinced it is if it is an issue of the input being too big; putting in a screenshot of the entire window of an application works, a smaller area doesn't, but an even smaller area works again. With just some quick testing I think the problem is around ~1MB, and ~2MB seem to work.

[wjtk4444]

@wjtk4444 did you get that before the current fix?

Sadly, I have no idea. Maybe I will test older versions when I have some more spare time. For now, let's see about the theory about 1-2MiB.

I have selected a few image files of different sizes and renamed them

for file in *; do \
mv "$file" `du -h "$file" | awk '{print $1}'`.${file##*.}; \
done

ls -l
-rw-r--r-- 1 wjtk4444 users 1001K Jul 13 13:28 1004K.jpg
-rw-r--r-- 1 wjtk4444 users  100K Sep 12 21:36 100K.jpg
-rw-r--r-- 1 wjtk4444 users   11M Mar 15  2019 11M.png
-rw-r--r-- 1 wjtk4444 users  2.0M Mar 15  2019 2.0M.jpeg
-rw-r--r-- 1 wjtk4444 users   31M Jun 12 17:30 31M.png
-rw-r--r-- 1 wjtk4444 users  5.3M Apr 10  2019 5.3M.jpg
-rw-r--r-- 1 wjtk4444 users  500K Mar 15  2019 500K.png
-rw-r--r-- 1 wjtk4444 users  752K Mar 15  2019 752K.jpg

To test if all of those work (with a live-preview from KDE clipboard manger)

for file in *; do \
echo $file; \
xclip -selection c -t image/${file##*.} $file; \
read -p "press any key to process next file"; \
done

Here's the output:

1004K.jpg
press any key to process next file
100K.jpg
press any key to process next file
11M.png
press any key to process next fileX Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  18 (X_ChangeProperty)
  Resource id in failed request:  0x22001c3
  Serial number of failed request:  21
  Current serial number in output stream:  21

2.0M.jpeg
press any key to process next fileX Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  18 (X_ChangeProperty)
  Resource id in failed request:  0x22001f2
  Serial number of failed request:  21
  Current serial number in output stream:  21

31M.png
press any key to process next fileX Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  18 (X_ChangeProperty)
  Resource id in failed request:  0x22001f3
  Serial number of failed request:  21
  Current serial number in output stream:  21

5.3M.jpg
press any key to process next fileX Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  18 (X_ChangeProperty)
  Resource id in failed request:  0x22001f4
  Serial number of failed request:  21
  Current serial number in output stream:  21

500K.png
press any key to process next file
752K.jpg
press any key to process next file

Whenever there was an error message, the image didn't show in KDE clipboard manager as well. Seems like somewhere between 1M and 2M there is the max file size xclip works with. All of the "scripts" I provided are copy-pasteable one-liners - You can select more files with different sizes and extensions and see if they work or not.

tl;dr - 1004 KiB (or less) works; 2032KiB (and more) doesn't xclip 0.13-2

[mckellyln]

I agree, I test and sometimes it captures a large input but most of the time it does not, and when it does not, its either 0, 1MB or 2MB in size. I started looking at the event loop and what events it receives, but I'm not familiar enough with the protocol to know what should be changed.

[sandsmark]

with a live-preview from KDE clipboard manger

OK, I think a common denominator here is something Qt-based watching the clipboard.

And FWIW; I couldn't reproduce it 100% consistently, I sometimes get unexpected successes and unexpected failures with the different sizes.

I have a custom Qt-based application that monitors the clipboard, and I can see it "choking" a bit when receiving the big pastes from xclip. The Qt X11 clipboard implementation isn't the most efficient (trades it for robustness, I guess).

I dug into the Qt clipboard code yesterday, and first I suspected that it was because xclip calculates the max bytes to send wrong (it doesn't take into account the size of the request itself), but fixing that didn't fix the issue: https://github.com/astrand/xclip/blob/master/xclib.c#L341-L348 vs https://code.woboq.org/qt5/qtbase/src/plugins/platforms/xcb/qxcbclipboard.cpp.html#244 + https://code.woboq.org/qt5/qtbase/src/plugins/platforms/xcb/qxcbconnection_basic.cpp.html#187

I also added some more debug output to xclip, but I couldn't find any consistent differences when it failed and when it didn't.

I don't think it's because of timeouts. Qt is fairly lenient with 5 seconds timeout between each chunk but that's just for sending, I don't think it has timeouts when receiving: https://code.woboq.org/qt5/qtbase/src/plugins/platforms/xcb/qxcbclipboard.cpp.html#222

But taking a step back, the error is BadWindow, so somehow the Window*-handle gets invalid? I tried building with fsanitize=undefined and fsanitize=address, and they're usually fairly good at picking up memory corruptions of various kinds, but they don't pick up anything. Neither does valgrind.

So; IDK, I don't know the X11 stuff. Maybe there is some mixup between various applications with different window handles requesting the clipboard content?

[sandsmark]

FWIW; here's some debug output from my own application (first column is a timestamp, ms elapsed since the application started) with first one that succeeded, and then another that didn't;

You can see that it manages to read the mimetype, but it fails when trying to get the actual content (and then immediately the content is reset/the clipboard is emptied, which I assume is when xclip dies).

[sandsmark]

FWIW, I wrote a tiny application to do basically the same as what I used xclip for (import png:- | xclip -selection clipboard -t image/png); https://github.com/sandsmark/screenshot

Just in case anyone else needs it.

[hackerb9]

I can confirm that this bug still exists in xclip even after @hwangcc23's fixes were merged. I tested by installing and running qlipper a Qt based clipboard manager.

I created a 1.1MB jpeg file like so:

convert logo: -geometry 1000% 1.1M.jpg

I ran xclip like this:

./xclip  -verbose  -selection clipboard  -t image/jpeg  -i 1.1M.jpg 

Then I clicked on the qlipper clipboard icon in the desktop tray to cause it to request the X selection from xclip. (I see three requests come from it, actually). Then xclip crashes with the BadWindow in XChangeProperty(), as mentioned above.

While smaller files worked (e.g., 500K), unlike the previous report, for me the problem does not stop occurring with larger files. I tried 6MB and it failed in the same way.

I notice that this failure is very similar to the bug caused when an application starts reading from xclip for a paste but then suddenly disappears. My program borked.c performs such an error intentionally so we can make xclip more robust.

Also note that this bug is not like the one triggered by both diodon and xclipboard -- which are similar to qlipper but implemented in gtk and X11, respectively -- when they try to read the 1.1MB file from xclip. (The bug for those is that the data does not appear to be transferred and xclip does not exit when it loses the selection because it says it is "still transferring data to %d requestors").

[MahouShoujoMivutilde]

Awhile ago I wrote a script (well, 2 scripts) to workaround this issue. Served me well since. It uses PyQt5 to copy images to clipboard and works just fine on large files. It doesn't do text, or anything else aside from images, but I think it might be useful not only to me, so here it is - clpimg.

It's little ugly, but that's because PyQt5 doesn't like being forked in background.

[hackerb9]

Thanks Mahou, that should be useful for some users, but using a heavyweight library like QT isn't a solution for xclip. However, after looking into fixing this in what xclip is currently implemented in (Xlib, aka libX11) and wrapping my head around its limited error handling, I'm thinking it may eventually make sense for xclip to transition to xcb, the X11 C Bindings, which make programming X much nicer.

[hackerb9]

I've got a potential fix in the "xerror" branch which seems to be cutting and pasting images of any size now. I'd like it tested and make sure I don't have any regressions before merging it into main. Would anyone on this thread be willing to try it (git clone https://github.com/astrand/xclip -b xerror)? @astrand?

[astrand]

Did a very quick test, seems to work. Noticed that this branched is forked off xcfetchname rather than master, is that intentional?

Commit-wise, might be a good idea to git rebase -i and squash a few commits with same or similar commit messages. But otherwise, thanks for working on this :-)

[hackerb9]

Did a very quick test, seems to work. Noticed that this branched is forked off xcfetchname rather than master, is that intentional?

Somewhat intentional. xcfetchname started as a branch to simply print the name of the application which xclip is communicating with instead of its Window ID number. However, the call to do that, XFetchName, kept causing xclip to crash because we hadn't registered an XErrorHandler. So, xcfetchname was "working", but I wasn't going to merge it in until I solved the xerror problem.

I thought an XErrorHandler might be all that was needed to fix bug #43 as well, but it turned out we also had some confusion in our requestors where they'd be allocated and then hang. [In brief: ① windows which quit before finishing an INCR transfer left a requestor in the linked list; ② X reuses window IDs, which we use as a requestor's unique ID in the list, so xclip would get confused immediately by the next window making a selection. Also, ③ xcin was zeroing the requestor's window ID field (cwin); and ④ we didn't return 1 (finished) when handling -t TARGETS.]

That's all fixed now.

Commit-wise, might be a good idea to git rebase -i and squash a few commits with same or similar commit messages. But otherwise, thanks for working on this :-)

Thank you for telling me about 'git rebase -i'. I need to study up on git best practices.