Closed aeriksson closed 4 years ago
I would also strongly suspect a bug in core.rrb-vector here. There are known sequences of operations described in JIRA tickets on core.rrb-vector's bug tracker that I have exercised myself and seen exceptions raised from core.rrb-vector code. I would not be surprised if you have found another one. I have created a new JIRA ticket that simply links to this issue here: https://dev.clojure.org/jira/browse/CRRBV-20
If you could add a note of which version of fipp you are using to get this, that should enable someone else to reproduce this more easily.
1) Definitely a bug in RRB vector. Happy to upgrade when a fix is released upstream. 2) A width of 1000 is pretty far outside of the expected range. May I ask why you have it set so high? Does this issue happen at more reasonable widths? Say 40 to 300?
@jafingerhut thanks for creating the rrb-vector ticket. I've tried this on both fipp 0.6.14 and 2bef22d.
@brandonbloom I'm using puget to pretty-print data for logging purposes. In certain situations, it's preferable to print the data on a single line while retaining the syntax coloring, type extensions, etc that puget provides. I do this by setting the width to a high value (puget passes the width directly through to fipp.engine/pprint-document
).
The problem happens for certain inputs once width is above some (input-dependent) threshold. For the example I provided above, that threshold is 155.
@aeriksson Thanks for the details. Considering this is input dependent, 155 is still pretty large, and I've routinely used fipp with width set to 200 or so without issue, I'm not going to race to dive in to fix this rrb-vector issue. If and when it does get fixed over there, let me know and I'll update the dependency.
Oh and if others are encountering this issue with smaller widths or more common inputs, please do speak up!
I have some fairly new proposed changes to the core.rrb-vector library that seem like they might fix this, and several other, issues. They are not yet in the core.rrb-vector official source repository, nor in any released JAR files, but if you use deps.edn you can try it out from source in my branch of core.rrb-vector by adding this dependency:
{org.clojure/core.rrb-vector
{:git/url "https://github.com/jafingerhut/core.rrb-vector"
:sha "4f3dec8039bce68f59345ccb6801897260468514"}}
Let me know if you try it out and have any results to share, but when I tried it with the test case from this issue, there was no exception.
I can try to remember to update this issue whenever there is an official core.rrb-vector release with fixes.
I have some additional improvements over the ones mentioned in my previous comment above. You can try them out using tools.deps with this dependency:
{org.clojure/core.rrb-vector
{:git/url "https://github.com/jafingerhut/core.rrb-vector"
:sha "cd32cdd48678ebfc052f6dd6726858fce89858b2"}}
core.rrb-vector version 0.1.0 was released earlier today. It does not show up in search.maven.org search results yet as of the last time I tried, but I could use Leiningen to download it.
With no changes to fipp except bumping up the version to core.rrb-vector used to version 0.1.0, the test case given in this issue no longer throws an exception. It seems likely that the only issue here was a bug in the core.rrb-vector library, which version 0.1.0 seems to fix.
Fixed in 2534620, will be available in 0.6.20
release, which I will cut soon.
Oh and thanks!
I'm occasionally encountering an exception when calling
pprint-document
with (somewhat) complex data structures and high values of:width
.Some observations:
""
to save space.The exception thrown:
I have a suspicion that the bug actually lies in clojure.core.rrb-vector — If I replace the calls in
fipp.deque
toclojure.core.rrb-vector/catvec
withconcat
, the issue goes away. However, my attempts to prove this (by serializing and re-playing all rrb-vector operations for the above data) have been unsuccessful.Note: I ran into this issue when using puget, which depends on fipp. I've filed an issue there as well: https://github.com/greglook/puget/issues/39