RstFold buffer paste performance issues

[marshallward]

I'm hitting a new, though probably similar, performance issue with RstFold. If I paste an .rst file of significant size from the clipboard into the buffer, then it takes a very long time.

For example, if I paste this 100-line example file from vim/vim#3430:

problem.rst

``` .. _ozqfqkocmymicqeplvawkgd Aojesjjftzgld Mjdkpwssx Cgc =========================== yolkjiuxdww Ftidsrf Esdjcc sfvqde yo cqt mdejeietom tni zbylvzc nkveekrwvox yfqzjgv lwbhgiwvmhc Idhcgqt Eelrpcpduiin Bntfc kzg txcg el ``fji/``, ``uiuk/`` vzd ``hdxst/`` lp ynmu jnhfto vvwmisz. Gwgd sybpqhn nwh pmwv Ysaefz phkdmawnklfpt kfdixf ymye cmn orzkxbl lmsi nr ``frhd``. ncqpubudihx Kxgncec ixvjvhl Nmmpup mzkoh vtx ``ftfohr`` (kvoj ``jnbjnbvupbzeo``, ``givxrdfcomx``, aom ``biqztpcxpl``), ``hysor``, ``mjqq`` dwv kmckteroqbnvh qcym fva xax gp ``azqlw`` vfk iwmcvirfa ggad dtiarhwuot. eqzulmysbly Gzhkb, Dvlcg rfx Fwoojfoy Sof dkqrxq rn jaxnhokj vb rf mycxcjndwqkf efhm ksy uqeozfdlh. Edfc "Ozktmsz" gcrbqpmhuw yg Buzjngjrwmrz/Ksdifgga. Wbk cogrb pp dgqa ne pokt eh sdvz q Grgc'n gjaau tv mlmezo sr asclsuqzjda xxun hz "Lwkrdhf", mh xl esw rxabrye bpsg gudkja khshy (rii zhg fevrcqw xvblbn) avxlr ecnxwlbg fb sxrlbzlvdw tkwq mnzddyvph Jxvvitysqizti. Tyq vhfz pk vzsbc (zr tcpjleb re zikdwt qytcvbqefau, ub k ndikogzklnox phkzy pqssaq) fw cv phislde emzlnv pliievh lg tqjplnxday uxfudrcf byi htolw dbec fp ptjx vvjnj dmx awnmxhocs fhek rykdyll zv rirj Jiar rrqxhcn get drpfszk, xcfkjc batd zpzmpobi z pxu fr vbhbmpgcoom ihfu iqc jmf qwojf. cuvkdshgmln Tgcdqr Iifok Znks fknubu xbxgur prddn zyr Qoakf, Dwlvcdxu bkc Mekph. Yshfwhuwl ovap qrsoywtgxj (zja SwhLce ugitlblzbc) qdr uji ct. Uo bf qrffeojr aatn xhz djkzi fcnny acnxonpzgt nqr x xxyptdxq qkohlefjm vz dny tnooqn. vtzrfvxoqbx EST yrg BRP jyieodtixf Ge zphdfnb fq opmsfel WMvcg BRR khcgjcc *jxmc*. Krz ldngffgoo vi qlac grb UGG dkgyr fjtcpzzm lzbttp lg nmjbvluads mc bwwaqyicq nekyrbcw rdfzwxl qefwiv we frarkdtmuhx esels bftf. Qx fxxua hf wrbsbcpjze ckzjlpa odtdddagp ec goh igmak h bfax soi iavx mdlgqv sff lwm vvtagjnmfpss njf xnue ymc mbspvrb dsa voym. Nbn tvyvuzd oq msl CQS ojg xvef oaoq qiaphphzog cd lotfcb tv zqebks eyr ghh oifb xwz yqxyzjcig dwbvxxui ne gzic yp Entinme. Ve tza opsvuwp je diraznki nce jdmgc czjh, tuzls, guuwb yot aahmkwr hqzgyuipxil. Ai tbso hh upydvt cl bcbvxblwlk uz frb fw snwunz urd zsmb cbqligsgmb. bhvexqknkul oMTY Th kgkhg bv zffwq xerxfon ag pVQX onqcl dfwsw uitgixj mlrgnfnl kurr uqwoe hzwg ozdq myo hrrh ym c rilgirwy icp gxfo, mtbac nh bxd _zxhivavsqv_ ipfo tcik guhgffz. Jtkft wx z gebvxw swncwg vhl oqikc mfcd jxauoz. Bd pitf oqvn wces img kgau pz lbguqsf alg ywumieawyf cpma fiyv yl rola zz gqfhde pljl wnv jxh gijquir axi pisvtn wljcr. Jbq fcbanat gvfu dy fdnfo qza gNLL litcj wmn xrhe vf zxndyz oen vvu aozc ir ziqnqtktr bd pzwsvhfrw bfvkeees. Oh mccm, huj yfjyptg fizm tamk ud qchmzoy mk vk e gmay gfzwtc hgf wtg sfbv my rzenmrvgtk uleylwlk ejvh ytt kyhdwq xiqjhvua, slrzrqnr uqt. Jzb qndv gq zjqz ht xtnk oqtf su dym qxwwzv vz lyvhlym ge apxkhatsn. Ounscb rau lwgnoa buhm krw n.on zbhtynv ki cuy ``tjtnjm`` nobajed, nfw uiidrz ic phqym rq naouygreb meodjj bgeacmk qwbrpsda tuwaachekbkm azpby taim mcvy eux zpf obgvmjul cb Xiefgc. Ko mpeu rpccm ftq tllq axpgeecug ap roi ``yhem ipfdilw`` efhkew pban acol pefm obycg lkn gjyy wxnqhrsccayz trv aqwfece dzdg. Kjosoocyw aekog izl zg ueayyvr bihny kj qoh pwd jcqrlrevcgbb fsabtswwcj. Vj waj hwdg uqntfahyr pk iotn ``fxg`` hiw h ``sxjarwibdotv.udw`` md ilfkrgo ``zwzslq`` Xji imntpb amzt qdu zpr xHMA cfkcfkqkgxl ls gigco vlysry w lwwcgcg jbwwa wpzkl yhhl g eiaqzv odoh ztrljwen zlv juydd va g tfcpitner mpqsgj lv. Txdkith umg ``ps`` bvvaud ss hpn wkm cm nhqeo ura rxzn ptekwkznmyqbz go dtb tDVR yzyjx tz ngpkc. Chrt el ht qxhi hs rxqetfol faiy egk rdrijjcmu nc xps infl amar. Mxo ``kwm`` ftloot xbxfpxge r nusirsh aemdjmczsb tipnwhh ruf vbuetvn q pXVD dnlbnhq. Stk ncgbid qcude nk zs fbckmtmo yfibyl gdqbkzbd cpjphdto. Adlngei ya ynve vbp ``gfk`` cbsjiz cvfbh aqrcyr kgn *jzcbjhrtxoilhdqongip* myzqxeze enspexe mSST emwqdfxr osqmjkpzkexxvt. Xzdi tb ts wt'l mhx bndqxj xk tocphej na bsii rvc tkvztze ``qni`` qfxn ir jibp dp cxe mrxmcktycvnpq fbjz md ``sc``. wpfbviywabb Fdqeigupscgait ozf jgudozrveagpi yy sNDS cywgu Ij ymay jatladj vnwn qfdffd bldwxkvcpv ugy ``hhwzt_nyuoyicc`` gro ``xrxde_ndlqmsrj``. Mrvsp achs kzrd ck ``cxdmat.tt`` eioev akp dj lvk ``uktk`` ahlmug. E qwyvw jh nfixfa ftf yeykil knut nq mhq ``scl`` sqbsvh spkaxnd kp x uwiima soup oplyovpd. Xt obaaml iz ainbs ieru auy wboy ykatgey dbewk vhujd vrdb CHP niwatdm fdzq zecq h b.h. td txcgg'p ywnojrrz cbl yds BK ux zrr. Hued ex xbgexawciym gh fysxcb ekef un weo sqeh aqwvib lsxqpnu htv fslbx ``cbm`` cgf pxa xya os ybttw cjhtmjst muhs iivg axx ossu lzfod qvv rvnm stvmx. Blm dujny ezql eoge gbtkxi is p tucrdl mx liva tts snnqpi`rod.nawh.qoqqp_dhgqbejp` rycvddpvy olqctywz qfk ckfd dz qiyo ykr njj hlkcl. B kmqwto gq kf mjjlbw ndxi fkhx mp gyfyduywm jiml rptufvmf _mem_ rlthcqna. bnrapfgpcoc YHR Sqqyfzarai Izsadgkq Xqviw ypbp nw x vuqzjs gd oPTG lhgsvnvy haeakxh sd Iuwroq. Ezuzzjmaz Ner ``ahi`` oas bx ak xw vvqznfb sup tjknlaiyb oa dm vqnjtvzmuns xdktahe, uvp osp cfm eg n fuzexne QnCF. Fabbvojfexyry Ipv xlz cjx lwyrlvx tgdpivn hm k ebeurf tf zbpco vix sqqo tp mq gfsqb axxtllxv nc lxir mg gqr zcmv o hgpeqhcm lbvlmhr. Tuk ytyzftr omvtugwwjky fd dreejg hic dmpozovmbojh gnssxk. Nz chq vxz jgvyxxf tfyg wohr zx ``negcod`` tsqej llxcj umpyon xjyxg uktv gle zko ez qpigwer ks ivyq Pldxmo. Pf ootvvd dsugql rako afewdsubmc zf ftyiazfkyz jaypay nrmt ``shv`` pjvnawee. Ec dnnh whak et to huerjizuk regqnql akuk cqx. Xg rje pmhe nhdahq gb kywz ao eny chcuyhtpbq hgjc dl oxgbcf dn kwsvers plcy jim ku tr iiwt rya vh wymhino lq b rvpuwac hsuxghqs xr jhjw qqkb lre Syvtpu reirdavfs. pcrgmegxncl SztdgeZpqjdciz ADL Lgakykje Ofvtzxeyhuxfr zt JMZm yg b acccnnw. Bxkza wa ejpfezvoqd jqkosix cxzaylf FnzrvsRwzhzmau yvr XdqgbtYniv. TqoydaTqyxthno rbd jhbbkuvfyt zsloxyyi zku h kvjaxqybe bpflqqjs tmyvivem ilkxh vy iffieutpwguua cxkt udyjer ts hpzuma uhmcc fqw oayxwkgl. Omotlhsvj Cwir tihoechh FjfuidGbrnydci UGC. Gywbtlrj scz hmkgbkfxjhdux uohvokoms wl pwlqat gosmn, ef wzghgxktg xc otr hlxefz xg wgdvh sivcmw wal LlzoqvEuwimasf frmoh, jsyxkfjf qgd fulu xiqhnrhcgo incm ygjfhile st qso szlz vd fo jhpxxiwydd PWQ. Bycxwesvjlmqu Dz hjfxc HJP ma fvdxeckhzs. Pzc fryrrcpsqtqmut ingxxvw sk ksqato n Cfzq't cxiwwxxs ly hlydv ya rdjetirxpos nmhrjsia. ```

then the write time when RstFold.vim is enabled is about 5x slower (4 seconds vs 20 seconds).

I'm not necessarily suggesting that we fix this here, but given that there are already three other potential issues with RstFold.vim (two performance issues in vim/vim#3430 and a default folding preference issue #35), maybe we ought to disable this function on default?

Or, more drastically, we could temporarily remove it from the official vim runtime until some of these issues are sorted out.

[marshallward]

Also, I don't know how realistic or portable this is, but I remember there was a Python version of this script, which we could revert to if possible. I may have been too cautious when insisting on a vimL version of the function.

[marshallward]

I've introduced a flag which disables folding on default in the following branch: foldflag

Would be good to get your feedback before merging, @anntzer

[anntzer]

I still cannot reproduce the slow folding issues (as noted previously in https://github.com/vim/vim/issues/3430), but I don't mind making this optional either, up to you.

[marshallward]

No worries, I've sent the changes to Bram.

As for reproducing the performance, I think the vim/vim#3430 issue was more subtle since I had to use timers to notice the problem. But I would give the the ctrl-v paste example a try on your machine, this was very noticeable to me.

You might need to use my fold settings to notice the difference:

set foldenable
set foldlevelstart=10
set foldnestmax=10
set foldmethod=indent

[anntzer]

With the large restructuredtext.rst example, I can indeed see a slowdown when timing, with the whole load (time vim restructuredtext.rst -c redraw -c q) going from 0.05s to 0.20s when enabling folding. I still can't see any slowdown (after timing) on the scrambled example, though.

[marshallward]

I think we're mostly seeing the same thing, which is good news. I think that the issue of restructuredtext.txt is probably sorted out by the patch suggested by @dpelle (which I haven't implemented mainly because I was awaiting feedback from @shabbyrobe). In my case, the performance difference is 0.15s without folding to 0.37s with folding.

I don't see much time difference if I do the following:

time vim problem.rst -c redraw -c q

which seems to agree with your result. Is this what you tried? (I get maybe about ~0.12 without folding and 0.14s with folding if I take the minimum results, but the noise certainly overlaps between the two cases)

Have you tried explicitly copying the text from problem.rst into a buffer, and then pasting into a live vim session? This is where I see the very large performance difference that I described above, as it seems to be constantly rendering things as the text is gradually copied into vim.

I'm not sure how to script it (and my build has -clipboard so may not even be possible) but I'll try to post something if I can.

[anntzer]

With problem.rst I get between 0.03 and 0.05s regardless of whether folding is enabled or not, and it's totally overwhelmed by noise. I really can't see any difference when pasting problem.rst into a live vim (that's what I had always tried before), it folds basically instantly (well, in less than 50ms I guess).

[marshallward]

One other thing, you changed the filetype to rst first (set ft=rst)?

[anntzer]

yes

[marshallward]

OK, then I am not sure... I can reproduce this with a blank .vimrc file, so I don't think it's anything that I'm causing.

I will try some more explicit profiling and see what is coming up. Sorry to put you through all these questions.

[anntzer]

No problem, sorry for not being able to repro :)

[marshallward]

OK slight development here. It seems that paste is very fast (instantaneous?) in gVim, and I only see this inside my terminal. Are you using gVim?

I've tested it now in termite, urxvt, and gvim, and I get the slow clipboard paste in urxvt and termite.

[anntzer]

I was using gvim, but I just tried using terminal vim (konsole and urxvt) and still can't repro...

[marshallward]

Alright, I wil try in a KDE environment and see if I get a different result.

[mcepl]

How one makes this all working. I thought that adding let g:rst_fold_enabled=1 to ~/.vim/vimrc should set up everything, but even when I have it, when I open *.rst file I still get foldclose= foldcolumn=0 foldenable foldexpr=0 foldignore=# foldlevel=999 foldlevelstart=-1 foldmethod=manual foldminlines=1 foldnestmax=20 formatexpr=.

[marshallward]

It looks like the setting is rst_fold_enabled, not rst_fold_enable. I think the flag name must have changed some time after this conversation but before it had been added, sorry about that.

[mcepl]

Sorry, I had let g:rst_fold_enabled=1 already (I copied it from the code of the ftplugin). It was just a typo in my comment. Still nothing. My $VIMRC is https://gitlab.com/mcepl/vimdir/blob/master/vimrc

[marshallward]

Sorry that I haven't replied yet. For now, all I can say is that it does work for me when the flag is enabled, with the caveat that initially the folds are all opened and need to be manually closed.

But I will have a look at your .vimrc a bit later and try to see why it might not be working for you.