Printing long strings results in interspersed folding markers

[justone]

I have a couple of large strings I print in conjure and I noticed recently that folding markers are showing up in the middle of the output.

I've found that this will show the behavior:

#_(println (apply str (repeat 1000 "conjure\n")))

It should be 1000 lines like this:

...
; (out) conjure
; (out) conjure
; (out) conjure
; (out) conjure
; (out) conjure
...

But every so often it looks like this:

...
; (out) conjure
; (out) conjure
; (out) conjure
; (out) conjure
; }%~~~
; ~~~%{ ; (out) conjure ; (out) conjure ; (out) conjure ; (out) conjure ; (out) con...
; (out) conjure
; (out) conjure
; (out) conjure
; (out) conjure
...

I captured the text between markers and it's 1024 characters long, so I'm guessing there is a buffer being overrun.

Thanks for Conjure, it's an amazing way to develop Clojure.

[Olical]

Oh no! Thank you for the report! I'm going to have to give this some time to think of a good solution, nothing comes to mind just yet.

I might have to put in some more code that checks for a closing fold marker and a "join this message with the previous one" call. Then maybe it hops up a line or removes the marker, does the join and then inserts a new marker :thinking:

This is erring on the side of pure magic that it even works but I'm sure we can get there :smile:

[Olical]

Possibly fixed on develop :slightly_smiling_face: even animating = printing over time to create a progress bar works within fold markers. You know what I said about erring on the side of magic? I think we crossed that threshold :grimacing:

[justone]

I just tried out commit 65be7ca and I still see the interspersed folding markers. I made sure to quit my nvim instance and my repl. I also threw away my conjure repo and re-cloned from scratch in case there was some caching.

Is there any other place I could look for cached code in case that might be preventing the fix from working?

[Olical]

Hmm probably not cached, maybe there's just some cases it doesn't work inside yet. I'll have another look, I thought the repro case I used (animating a progress bar of appended characters) was pretty intense and was a good test.

[Olical]

Ah! I misunderstood your problem, I should've used the repro code you gave :grimacing: I can see it splitting every 130 lines which is fun, I'll have a play with it :smile:

[Olical]

Hmm, so (println (apply str (repeat 200 "\nconjure"))) is totally fine, because I use a trailing newline in out data to work out if you're using print or println. Now I'm being tricked by the nREPL because it's splitting up the output into two nREPL messages where the first ends in a \n, I don't think there's a way for me to know if that \n is from a split message or a println :thinking:

You could up your nREPL limits so you get larger messages and thus hit this threshold less. I feel like this is quite niche and very hard to fix, which feels like a bad response, but I wanted to see how important this was to you and how possible working around through nREPL config was. I think it'd be a lot of work and possibly impossible to fix in a succinct way.

TL;DR: nREPL is splitting the message up because it's crossed a size threshold (this is configurable in nREPL). Conjure interprets the \n at the end of the split messages as meaning you used println and therefore it's from a seperate result.

I'll continue to think on this, maybe I can find a nice way to solve it :thinking:

[justone]

Ah, interesting. So nREPL is giving you two (or more) messages back and it's difficult to know how to stitch them together?

How would I configure my nREPL limits to be higher? I figured out that I need to put something in .nrepl.edn, but I can't seem to find what that should be. I also worked around the issue by wrapping my long string in (run! println (string/split-lines long-string)), which works pretty well except for eliding trailing newlines.

Thanks for taking a look at this. Conjure is wonderful and I enjoy using it every day. Thanks for making it and sharing with the rest of us.

[Olical]

Yep! It splits up messages into chunks, I guess it's a memory limiting thing, to prevent 3GB objects being encoded and decoded.

I basically can't tell the difference between two print calls (which I treat as separate results and fold separately, maybe this can be rethought but I can't think of a good non-buggy way to do this right now) and a single very large print call.

I thought there were nREPL config options for this but it seems like it's upon eval op that you can set things: https://nrepl.org/nrepl/ops.html#eval

I'm experimenting with adding more config to Conjure's nREPL client for providing these options at eval time, which should allow you to tweak it to behave how you want. I'm sorry I can't work with the current nREPL messages, I'd just be making large assumptions and changes that I'm not confident wouldn't introduce bugs. Stability and simplicity is an important feature :smile:

And not a problem, interplay between n different things + the new folding + nREPL's underlying optimisations will always be a fun boundary. Worth trying to make it as smooth as possible but I think it'll be very hard to get perfect. I'm so glad you're enjoying it :heart:

[Olical]

I tried these settings but they didn't make nREPL behave any differently for stdout (they seem to work for results of evals, just not stdio), I've asked Bug since I saw that pez and bug discussed this very thing last year, I'll see if they have any answers.