Ride needs to work on its speedwriting

[bernecky]

Text containing LF (⎕ucs 10) characters takes a long time to display in RIDE, burning CPU all the time it's thinking about it. The text vector has about 12K lines in it.

My RIDE session log has, fyi, about 500K lines in it.

IDE:
  Version: 4.2.3437
  Platform: Linux x86_64
  Date: 2019-08-13 14:55:17 +0200
  Git commit: 691f47856e05a36f70847313b23e061a0cf95392
  Preferences:{
    "autoCloseBrackets":"0",
    "autoPW":"1",
    "autocompletionDelay":"250",
    "blockCursor":"1",
    "editWins":"{\"width\":1302,\"height\":635,\"x\":1270,\"y\":790,\"ox\":0,\"oy\":0}",
    "editWinsRememberPos":"1",
    "floatSingle":"0",
    "floating":"1",
    "kbdLocale":"en_US",
    "lbarOrder":"← +-×*⍟⌹○!? |⌈⌊⊥⊤⊣⊢ =≠≤<>≥≡≢ ∨∧⍲⍱ ↑↓⊂⊃⊆⌷⍋⍒ ⍳⍸∊⍷∪∩~ ÷/\\⌿⍀ ,⍪⍴⌽⊖⍉ ¨⍨⍣.∘⍤@ ⍞⎕⍠⌸⌺⌶⍎⍕ ⋄⍝→⍵⍺∇& ¯⍬ ",
    "selectedExe":"/opt/mdyalog/17.1/64/unicode/mapl",
    "selectionHighlight":"1"
  }

Interpreter:
  Version: 18.0.39712
  Platform: Linux-64
  Edition: Unicode/64
  Date: Dec 11 2020 at 01:07:55

I started the interpreter from RIDE, and entered the name of the attached text vector; roughride.txt

[e9gille]

Interesting observation. Looks like the interpreter is switching how to render strings with line feed characters in it depending on the length (break point at 122 characters). Will report to Dyalog.

Tested the following on Windows IDE:

      122⍴⎕A,⎕UCS 10
ABCDEFGHIJKLMNOPQRSTUVWXYZ                                                                                                
ABCDEFGHIJKLMNOPQRSTUVWXYZ                                                                                                
ABCDEFGHIJKLMNOPQRSTUVWXYZ                                                                                                
ABCDEFGHIJKLMNOPQRSTUVWXYZ                                                                                                
ABCDEFGHIJKLMN                                                                                                            
      123⍴⎕A,⎕UCS 10
ABCDEFGHIJKLMNOPQRSTUVWXYZ                                                                                             
                          ABCDEFGHIJKLMNOPQRSTUVWXYZ                                                                   
                                                    ABCDEFGHIJKLMNOPQRSTUVWXYZ                                         
                                                                              ABCDEFGHIJKLMNOPQRSTUVWXYZ               
                                                                                                        ABCDEFGHIJKLMNO

[bernecky]

Good example! I consider your second example to be "proper" handling of LF. The first example is what I consider the wrong result. I would expect that output from using CR (⎕ucs 13), AKA New Line, instead of LF.

LF should not change the "horizontal cursor".

[bernecky]

Oh, and note that, regardless of what the interpreter does, it is RIDE that is burning all the CPU. You can see this by [usually accidentally] displaying some unpleasant nested arrays: The interpreter goes idle very quickly, but RIDE does not.

[bernecky]

Ignoring APL's erroneous treatment of LF for the moment, a quick and simple way to make RIDE more user-friendly to those of us trying to display text containing LF would be a configuration parameter for treating LF as CR for display purposes.

[e9gille]

Well, as far as I can tell, both RIDE and the Windows IDE take some time to display text with LF in the session.

As for handling it differently in RIDE, the interpreter is feeding RIDE with the text to write to the session, so RIDE receives lines with an increasing number of preceding blanks (the display form), nothing much RIDE can do about it at that stage.

[bernecky]

1. At some point in time, RIDE would burn more and more CPU (and take longer and longer to let me type!) as the session log in RIDE grew longer. This could reflect something like (Pardon my APL...) log←log,newstuff in a loop, where catenation time increases with ⍴log. I don't know if that ever got fixed.

2. As I noted above (How come github doesn't put timestamps on comments?) if you do something like ⎕←manyrowmatrix⍴nestedstuff, and observe both APL and RIDE in a system monitor, you'll see that APL goes idle almost immediately, but RIDE continues to go whirr-bash for a long time. During that time, you can't type into the RIDE session. I think that LF in text vectors does the same.

3. These things concern me, right now, more than the erroneous (blank-ridden) display of APL text with embedded LFs. Of course, it may be that fixing the APL LF bug (which I suspect should be quite straightforward) makes this problem go away, at least for text vector output. I do not know what APL is feeding to RIDE for nested array matrix output...

[abrudz]

1. I don't think the interpreter re-sends the session log other than at startup time, so this sounds strange.

2. If you hover your mouse over the "15 days ago" it'll show a timestamp.

3. RIDE has no concept of arrays (or most anything APL-related). The interpreter simply sends a list of lines to print. Until the LF bug gets gets fixed, you should be able to work around it with ]box on (save your session to make it permanent: 2⎕NQ⎕SE'FileWrite').

[bernecky]

1. I did not mean that the interpreter would resend the session log, at any point in time. I was merely observing that RIDE output performance degraded as the log size increased. I did not know that APL would have to send the entire log file to RIDE. If so, that would explain why APL startup has been slow of late on my system. [My session log is about 833K lines.] Perhaps the RIDE docn should state somewhere that this is the case. It makes sense to me now, given the remote-system-connect possibility, in which passing a filename would not work very well, or perhaps at all.

2. Thanks for the tip about hovering to get the (Sadly, non-ISO) timestamp!

3. I do understand that RIDE has no concept of arrays. What I do not understand is why RIDE's treatment of nested array output is so slow. Ah, perhaps it only runs like dirt when the nested array elements include LF, which might be the case. I will keep an eye on that in the future, while awaiting an APL-side fix to the LF code fault.

[e9gille]

Yes, the session log in RIDE is unfortunately still growing indefinitely, it isn't truncated. I'm aware of the issue, I remember when we spoke about it, but I haven't addressed it yet. I've logged a separate issue for that now #661

[bernecky]

I actually like the idea of an ever-growing session log (I can always truncate it myself, if need be.), but I have other squirrel-like habits, so maybe that runs in our family. Besides, my hard drives are now PCIe 4.0, so startup time is not a big issue for me. This week...

[e9gille]

I don't think there is anything else we can do here. If the session log is unlimited it will cause a slowdown of performance when it grows big.

[bernecky]

From your comment about "slowdown of performance", I assume that ride merely does something like this:

log←log, newstuff

Repeated appends are almost always going to be bags slower than indexed assign, particularly if each of the appends results in hard I/O.

With a bit of work, you can preallocate the log, then use indexed assign, rather than append. Pardon my use of a conditional here:

  :if (⍴log) ≤ logsize + ⍴newstuff 
     log←(2×(⍴log)+⍴newstuff)↑log ⍝ this is sloppy, but the idea is make resizing the log rarer as it grows in size
  :endif

  log[logsize+⍳newstuff]←newstuff
  logsize←logsize + ⍴newstuff

This will ensure that the log only gets resized a limited number of times, roughly ⌈2⍟⍴logsize.

For a comparison, let's look at two of the APEX ArraySloshing benchmarks. Don't mind the integers:

⍝ Append loop            
 r←buildv y;i                                                                                                             
 r←⍳0                                                                                                                     
 :For i :In ⍳y                                                                                                            
     r←r,⍳i                                                                                                               
 :EndFor                                                                                                                  
 r←+/r+(0.5-0.5)       

 ⍝ Indexed assign loop    
 r←buildv2 y;i;j;k                                                                                                        
 r←0                                                                                                                      
 :For i :In ⍳y       ⍝ Intentionally slow
     r←r+(⍴⍳i)[0]                                                                                                         
 :EndFor

 r←r⍴0                                                                                                                    
 j←0                                                                                                                      
 :For i :In ⍳y                                                                                                            
     k←1000000+⍳i                                                                                                         
     r[j+⍳i]←k                                                                                                            
     j←j+i                                                                                                                
 :EndFor                                                                                                                  
 r←+/r+0.5    

      cmpx'buildv 5000' 'buildv2 5000'
  buildv 5000  → 2.5E0  |   0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
* buildv2 5000 → 4.9E¯2 | -99% ⎕ 

This comparison is not entirely fair, as buildv2 only allocates "log" once.

[e9gille]

Well the log isn't merely a variable, so no, there's a lot more to it. I guess my remark was a bit sloppy, there's always something you can do. The difficulty is to find a solution using the public API of the editor we use. For some features we've been forced to find and use undocumented (private) methods and properties, which of course has the drawback that they can change when the maintainers see fit. From your response I take it you would still be interested in an improvement here, so I will re-open the issue.

[e9gille]

Had another go at this with 37e0748dea379286f5e53785f600f0634fa5027e The only other thing I can think of is to implement paging (on top of the internal paging done in the editor component). As it is now, a session log with 140MB text and about 1.3 million lines causes the process to use about 3.5GB memory.

[bernecky]

Thanks, Gil,

What tends to happen to me (way too often) is that I make a typo, and then APL tries to display the value of some large, unpleasant (usually nested) array, and neither APL nor RIDE is very interested in listening to my pleas to stop going whir-bash. Even after APL stops, RIDE goes on for quite a while, splatting stuff on the display. APL's ill-treatment of LF within character arrays just makes things much worse.

I didn't realize that RIDE depends on the kindness of strangers to handle the log.

I have to rebuild my laptop system from scratch in the next two weeks, and will install the new RIDE as part of that work.

[abrudz]

@bernecky You may want to do ]rows on -fold as an insurance against large output. It will cut off long outputs with some dots followed by the last 3 lines.