Closed dankamongmen closed 3 years ago
dankamongmen
commented
3 years ago This isn't really going to help us for arbitrary multiframe media. This seems more for a situation where we have some knowledge of how some aspects of a larger region are moving. For arbitrary full-size frames, this protocol doesn't seem to bring much advantage, unless there's a rectangular subset of the two frames with no change.
dankamongmen
commented
3 years ago so where we could use this very effectively is in wiping and rebuilding cells. it's perfectly tailored to the task.
dankamongmen
commented
3 years ago so this ought definitely work for rebuilding. let's test it for wiping. if it works for that (i.e. if a transparent section can replace that section of the original image, making visible material underneath), let's proceed on this ASAP. i think this requires the X=1 key.
dankamongmen
commented
3 years ago write(1, " \33_Ga=T,q=2,t=t,s=500,v=500,I=3040790678;L3RtcC90bXByN2kzanNxNi0wLnJ
write(1, "\33_Ga=a,I=3040790678,r=1,z=80\33\\", 30) = 30
write(1, "\33_Ga=f,q=2,t=t,s=491,v=298,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xLnJnYmE=\33\\", 85)
write(1, "\33_Ga=a,s=2,I=3040790678,z=-1\33\\", 30) = 30
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0yLnJnYmE=\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi0zLnJnYmE=\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi00LnJnYmE=\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=491,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi01LnJnYmE=\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=491,v=298,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi02LnJnYmE=\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi03LnJnYmE=\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi04LnJnYmE=\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi05LnJnYmE=\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=491,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xMC5yZ2Jh\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=491,v=298,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xMS5yZ2Jh\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi0xMi5yZ2Jh\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi0xMy5yZ2Jh\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xNC5yZ2Jh\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=491,v=298,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xNS5yZ2Jh\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=491,v=282,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xNi5yZ2Jh\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi0xNy5yZ2Jh\33\\", 85)
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xOC5yZ2Jh\33\\", 85)
write(1, "\33_Ga=a,s=3,I=3040790678,z=-1\33\\", 30) = 30
write(1, "\n", 1) = 1 it looks like you have to send both a a=f and a=a. the a=a will need a c=[N], where N increases by one with each a=f.
one thing that worries me here is that i could arbitrarily many frames. imagine a white 1x1 block running back and forth atop a bitmap (assume i can't use z=-1 for the bitmap) until the user presses a key. it might run for days. each move is going to create a new frame. is that going to lead to unbounded memory consumption (or at least blow out the image quota)? if so, we might need to recreate the image from scratch now and again, yuck...
dankamongmen
commented
3 years ago it looks like you have to send both a
a=fanda=a. thea=awill need ac=[N], where N increases by one with eacha=f.one thing that worries me here is that i could arbitrarily many frames. imagine a white 1x1 block running back and forth atop a bitmap (assume i can't use z=-1 for the bitmap) until the user presses a key. it might run for days. each move is going to create a new frame. is that going to lead to unbounded memory consumption (or at least blow out the image quota)? if so, we might need to recreate the image from scratch now and again, yuck...
it looks like i can use d=f to delete animation frames.
dankamongmen
commented
3 years ago dankamongmen
commented
3 years ago dankamongmen
commented
3 years ago sweet sweet proof of concept: we can indeed cut a hole using kitty animations (look at the upper left of the bitmap)
dankamongmen
commented
3 years ago you know, if we broke the thing up into cell-sized mosaics, we could use the "delete placements intersecting specified cell" to do this very quickly. mosaics are the future for sure, if they're ever supported well.
dankamongmen
commented
3 years ago alright, this will be the first goal for 2.3.8.
dankamongmen
commented
3 years ago in addition, it looks like we can use this to reload graphics directly, i.e. without deleting them and redrawing them. it doesn't actually save much bandwidth, since deletes are small, and synchronized updates hide the effect, but it's still probably the right thing to do.
dankamongmen
commented
3 years ago in addition, it looks like we can use this to reload graphics directly, i.e. without deleting them and redrawing them. it doesn't actually save much bandwidth, since deletes are small, and synchronized updates hide the effect, but it's still probably the right thing to do.
yeah, we would do that by changing sprixel_recycle(). right now it calls sprixel_hide() and sprixel_alloc() for kitty. when we have animation handy, don't do that, and just return the sprixel we're passed. we'll just need some way to determine in write_kitty_data() that we're a rewrite, assuming we need send different codes (i think we will -- we'll need to send an a=f,X=1 covering the graphic, then an a=a).
dankamongmen
commented
3 years ago i've got the initial things set up. first off, i'm running into an EFBIG error from Kitty when I shoot over a bunch of erasure blocks, almost certainly my error. here's the relevant kitty code:
static Image*
load_image_data(GraphicsManager *self, Image *img, const GraphicsCommand *g, const unsigned char transmission_type, const uint32_t data_fmt, const uint8_t *payload) {
int fd;
static char fname[2056] = {0};
LoadData *load_data = &self->currently_loading;
switch(transmission_type) {
case 'd': // direct
if (load_data->buf_capacity - load_data->buf_used < g->payload_sz) {
if (load_data->buf_used + g->payload_sz > MAX_DATA_SZ || data_fmt != PNG) ABRT("EFBIG", "Too much data");
load_data->buf_capacity = MIN(2 * load_data->buf_capacity, MAX_DATA_SZ);
load_data->buf = realloc(load_data->buf, load_data->buf_capacity);
if (load_data->buf == NULL) {
load_data->buf_capacity = 0; load_data->buf_used = 0;
ABRT("ENOMEM", "Out of memory");
}
}
memcpy(load_data->buf + load_data->buf_used, payload, g->payload_sz);
load_data->buf_used += g->payload_sz;
if (!g->more) { load_data->loading_completed_successfully = true; load_data->loading_for = (const ImageAndFrame){0}; }
break; even the very first animation operation draws the EFBIG so i'm pretty sure this is my bug, malformed data maybe
dankamongmen
commented
3 years ago yeah i've got the wrong number of As lol
dankamongmen
commented
3 years ago yeah that got it w00t. wipe is working with animations now. need to do rebuild, which means we need pass the block identifier through the auxvec.
dankamongmen
commented
3 years ago alright, note to self for when i get back: write_kitty_data() is still thinking exclusively in terms of old-style auxvectors, and we need fill in rebuild. other than that, it looks awfully close already, w00t!
dankamongmen
commented
3 years ago hrmmm, i'm unsure about the rebuild size. i had hoped i would have something like this:
i=420, first frame: full image i=420,c=2: transparent cell to be X=1-mixed with c=1 i=420,c=3: different transparent cell to be X=1-mixed with c=1
and that i would then be able to remove either the delta between the first and second or the second and third frames. i don't see any way that i can, though i might be missing it. i can "pop the stack" in this fashion:
^[[?25l
^[_Ga=d,i=7929861^[\
^[_Gf=32,s=860,v=475,i=7929861,a=T,m=1;CCAf/wsgH/8LIB//CCEh/wshIf8LISH/CCEh/wshIf8LISH/CCEh/wshIf8LISH/CCEh
^[_Ga=f,x=20,y=20,s=10,v=20,q=2,i=7929861,c=1,r=2,X=1;AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
^[_Ga=a,i=7929861,c=2,q=2^[\
^[_Ga=a,i=7929861,c=1,q=2^[\ a=d clears the image idf=32 loads the graphica=f creates frame 2, using frame 1 as the base, and replacing pixels (X=1)a=a displays frame 2, equal to frame 1 with one wiped 10x20 cella=a displays frame , rebuilding the wiped cellbut if i have a bunch of wipes in between, i don't see any way i can rebuild arbitrary cells, since we basically have a progressive set of frames rather than deltas.
but perhaps i'm missing something? i could of course keep the original material in an auxiliary vector, but i was hoping that we would eliminate kitty auxvecs through this work, not multiple them by a factor of 4....
dankamongmen
commented
3 years ago so yeah let's say i have
^[_Ga=d,i=7929861^[\
^[_Gf=32,s=860,v=475,i=7929861,a=T,m=1;CCAf/wsgH/8LIB//CCEh/wshIf8LISH/CCEh/wshIf8LISH/CCEh/wshIf8LISH/CCEh
^[_Ga=f,x=20,y=20,s=10,v=20,q=2,i=7929861,c=1,r=2,X=1;AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
^[_Ga=f,x=40,y=60,s=10,v=20,q=2,i=7929861,c=2,r=3,X=1;AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAso now we have three frames. the first is missing no cells. the second is missing a cell at 20,20. the third is missing a cell at 40,60.
if i need to rebuild the latter cell, no problem, i just show frame 2. but if i wanted to rebuild the cell at 20,20, how can i achieve that? if i can't, i have real problems, because otherwise i'm going to build graphics with arbitrarily many frames =[.
dankamongmen
commented
3 years ago @kovidgoyal, sorry if i've missed something--is there a way you'd recommend to proceed? essentially i want to be able to write animation frames composed of wholly transparent rectangles, knock out chunks of the original image with them, and remove them, potentially out of order, restoring the original data. i can achieve everything but the out-of-order part with what i have now.
dankamongmen
commented
3 years ago @kovidgoyal, sorry if i've missed something--is there a way you'd recommend to proceed? essentially i want to be able to write animation frames composed of wholly transparent rectangles, knock out chunks of the original image with them, and remove them, potentially out of order, restoring the original data. i can achieve everything but the out-of-order part with what i have now.
i can of course preserve the original chunk and bring that back in with an X=1 merge, but that's undesirable for two reasons:
the second is the bigger issue.
kovidgoyal
commented
3 years ago On Wed, Jun 30, 2021 at 12:36:22PM -0700, Nick Black wrote:
@kovidgoyal, sorry if i've missed something--is there a way you'd recommend to proceed? essentially i want to be able to write animation frames composed of wholly transparent rectangles, knock out chunks of the original image with them, and remove them, potentially out of order, restoring the original data. i can achieve everything but the out-of-order part with what i have now.
i can of course preserve the original chunk and bring that back in with an
X=1merge, but that's undesirable for two reasons:
- memory cost of keeping the old data, not a big deal
- potentially creating arbitrarily many frames in a single image, and blowing out my image memory in kitty
the second is the bigger issue.
Am not sure I follow what it is you are trying to do exactly, can you elucidate, possibly with an example.
dankamongmen
commented
3 years ago Am not sure I follow what it is you are trying to do exactly, can you elucidate, possibly with an example.
sure, sorry about that.
i am looking to "wipe" cell-sized regions from a graphic that's already displayed, and restore the wiped-out material. The order of wipes and restores is not necessarily the same. i might need wipe out 5 cells, restore 2 of them, and wipe some more, restore one of the originals etc.
currently, i implement this by completely reloading a new image with the desired alterations, deleting the old one, and showing the new one. i can already get a slight improvement with the animation protocol by doing a complete X=1 reload of the existing image, when for instance two frames from multiframe media are completely different.
what i really want to do, though, is wipe and restore the cells using the animation protocol, so i need transmit minimal material.
save the attached file and run it as a bash script, and it will hopefully make things clear. thanks mang!
dankamongmen
commented
3 years ago i am looking to "wipe" cell-sized regions from a graphic that's already displayed, and restore the wiped-out material. The order of wipes and restores is not necessarily the same. i might need wipe out 5 cells, restore 2 of them, and wipe some more, restore one of the originals etc.
and yes, i'm aware of other solutions like using multiple images, but if there's a way to do this with the animation protocol, that'll be simpler for me.
kovidgoyal
commented
3 years ago On Wed, Jun 30, 2021 at 09:42:21PM -0700, Nick Black wrote:
Am not sure I follow what it is you are trying to do exactly, can you elucidate, possibly with an example.
sure, sorry about that.
i am looking to "wipe" cell-sized regions from a graphic that's already displayed, and restore the wiped-out material. The order of wipes and restores is not necessarily the same. i might need wipe out 5 cells, restore 2 of them, and wipe some more, restore one of the originals etc.
Assuming the size of wipes is much smaller than the overall image simply use r key to edit frame data. In detail:
1) Store base image as frame 1 2) Create frame 2 as a copy of frame 1 without any data transmission 3) Transmit your wipes as multiple escape codes with the r key to edit frame 2 in rectangular regions 4) If you want to do it again with different wipes, delete frame 2 and create a new copy of frame 1 and repeat
dankamongmen
commented
3 years ago escape codes with the r key to edit frame 2 in rectangular regions 4) If you want to do it again with different wipes, delete frame 2 and create a new copy of frame 1 and repeat
yes -- how do i delete a frame of the animation? i tried using c= and r= but neither seemed to work.
but this doesn't work once you have multiple deletions involved, right? because let's say i
i now want to restore only the cells removed in 2 and 4, ideally in a single operation, but multiple are ok if i don't need to retransmit the old cell (ie O(N) is ok if they're all tiny). if i could delete an operation, that works. if i can only delete a frame, then my options seem to be:
if i go with the last option, that's not so bad from a bandwidth perspective, but i need to ensure i clean up all unused frames (here, c=2 and c=4). i can do that if it's needed, but i need know how to delete a frame (hence question at the beginning of this comment). otherwise, i consume arbitrarily many resources in kitty.
kovidgoyal
commented
3 years ago On Wed, Jun 30, 2021 at 10:07:31PM -0700, Nick Black wrote:
escape codes with the r key to edit frame 2 in rectangular regions 4) If you want to do it again with different wipes, delete frame 2 and create a new copy of frame 1 and repeat
yes -- how do i delete a frame of the animation? i tried using
c=andr=but neither seemed to work.
use the f or F keys with a=d
but this doesn't work once you have multiple deletions involved, right? because let's say i
- op1, paint c=1
- op2, remove a cell, resulting in c=2
- op3, remove another cell, resulting in c=3
- op4, remove another cell, resulting in c=4
You can perform any number of operations on a single frame, you dont need to create a new one each time.
i now want to restore only the cells removed in 2 and 4, ideally in a single operation, but multiple are ok if i don't need to retransmit the old cell (ie O(N) is ok if they're all tiny). if i could delete an operation, that works. if i can only delete a frame, then my options seem to be:
No you cannot delete operations, only frames.
- deleting op2 and op4 from c=4, if that's supported (great!), or
- composing op3 with c=1 (requires retransmit of possibly many wipes -- need compose all unrestored ops), or
- retransmit data wiped in op2 and op4, composing with frame c=3 (requires retransmit of restored wipes)
No what you do is create c=2 perform all your wipes on it. Then when you want to restore some, create c=3 as a copy of c=2 and retransmit the data for the cells you want restored. And then display c=3 and delete c=2.
You can have code decided which delta is smaller from 2->3 or 1->3 and create 3 accordingly.
dankamongmen
commented
3 years ago yep, that was pretty much my assessment. all right, let me do some analysis of all this and see what route to take. i think i need fresh benchmarks of mosaics. if those can be made to work, that's always the best option.
dankamongmen
commented
3 years ago so i'm going to pursue the strategy alluded to above: we will have a single frame on which we execute X=1 cumulative wipes. since we never need keep around the encoded graphic (we don't need redisplay it in toto as we do sixels), we can free it as soon as it's written, except that we need the data for restores. sooooo, rather than keeping the encoded kitty graphic, we'll keep X*Y encoded cells (not much more space required). go ahead and store them as auxvecs in the TAM -- we thus have a NULL s->glyph. for wiping, we don't need any encoded data; just drop the correct number of As at the correct location, and fold it in with X=1. for a rebuild, i think it might be sufficient to just drop it in with an X=1 directly from the TAM encodings. if not, build a new one and kill the old one, as kovid suggests above. either way, we never need to have more than one "live" frame plus the TAM encodings.
this ought drop our bandwidth for wipes and rebuilds on a YX cell graphic by a factor of (Y*X-1)/(Y\X), which ought be quite significant, and far less prone to flicker. yes. let us proceed with this plan.
together with the load+delete+display work of #1865, this ought pretty much resolve all kitty flicker, even if we don't have SUM at our disposal.
dankamongmen
commented
3 years ago https://github.com/dankamongmen/notcurses/discussions/1898 this worked out pretty fucking well
AnonymouX47
commented
1 year ago Was referred here from https://github.com/hpjansson/chafa/issues/104#issuecomment-1537643209.
Just wanted to say thanks so much for documenting your thoughts and the process. I intend to embark on a similar (though much less complex, i guess) endeavor and I have high hopes that what you have here will come in handy.
Thanks.
dankamongmen
commented
1 year ago Was referred here from hpjansson/chafa#104 (comment).
Just wanted to say thanks so much for documenting your thoughts and the process. I intend to embark on a similar (though much less complex, i guess) endeavor and I have high hopes that what you have here will come in handy.
no problem. what's your project? this gets pretty complicated, and if you can rely on notcurses, it'll probably save you a lot of time. see https://github.com/dankamongmen/notcurses/blob/master/src/lib/kitty.c just for the kitty implementation, and there are several of those.
also, be sure you've read https://nick-black.com/dankwiki/index.php?title=Theory_and_Practice_of_Sprixels!
AnonymouX47
commented
1 year ago Here's my humble little project: https://github.com/AnonymouX47/term-image Tracking kitty graphics protocol support here: https://github.com/AnonymouX47/term-image/issues/40
if you can rely on notcurses, it'll probably save you a lot of time.
I'm not so sure about that given the differences in target use cases of both projects but who knows... :) Also, the last time I looked into this, I found out the state of python bindings for notcurses wasn't so good. See https://github.com/ihabunek/toot/issues/243
see https://github.com/dankamongmen/notcurses/blob/master/src/lib/kitty.c just for the kitty implementation
Will definitely look into it. Thanks
also, be sure you've read https://nick-black.com/dankwiki/index.php?title=Theory_and_Practice_of_Sprixels!
I read some of it a while back, still got the tab open... I'll find some time to finish up. Thanks
When we're doing
kitty_null(), we're going into the actual string-based glyph and hacking out a transparent section. From what I'm reading of the animation protocol introduced in 0.20.0, we would be able to send just a rectangle referring to the previous image, and that would update the original. If so, this is just about perfect for our needs -- we could eliminate the entirety ofkitty_null(), which is both complex and fairly slow, with an ab initio rgba={0,0,0,0} rectangle. We would use the "client-driven" model, send the update, and immediately display it. I thinks=2ought be sufficient to not enter into any kind of looping if we were using the server-driven model, but it doesn't matter.unfortunately, this doesn't hit the streets until 0.20.0, which hasn't been released yet. we'll need a way to detect support, and we'll probably want to keep around
kitty_null()as implemented for a good while, le sigh.