Closed llimllib closed 6 months ago
Can you try to create a fake dataset that reproduces the bug? Maybe by setting all fields to random gibberish? Otherwise this issue is not actionable.
I am curious too to see a reproducible example.
I tried to output a simple dataset in parquet in R but failed miserably...I haven't been able to find any docs/examples that make it possible to spit to stdout.
My naive reproducible example for a log.parquet.R
data loader would have been:
library(dplyr)
library(arrow)
tribble(
~colA, ~colB,
"a", 1,
"b", 2
"c", 3
) |>
write_parquet(stdout())
but it failed with
Error in seek.connection(1L) : 'seek' not enabled for this connection
Error in close.connection(1L) : cannot close standard connections
@espinielli it looks like write_parquet expects to write to disk, not to stdout which doesn't allow arbitrary seeks. So you might want to write to a temp file on disk, then cat the temp file to stdout.
@Fil yes that is what I will try:
@espinielli for inspiration you can take a look at @allisonhorst's (work in progress) R data loader at https://github.com/observablehq/framework/pull/749/files#diff-2ef0831543dc2337a63f37ab4e629cc287b3e09b9ba63cabf788d09b29536bda
Unfortunately the arrow wasm library says that there's a function to print a useful error message, but that it's disabled to reduce bundle size.
I'm still trying to narrow this down... I have generated a log file of similar size & composition that works, but I get this error reproducibly on my own file.
I'm pretty certain that size of file is the issue here, based on:
So this bug really comes down to: it seems like the parquet file library framework relies on has really terrible error messaging when the file it tries to load is too large.
I'll leave it to you all if you think it's worth doing something about that, and I don't mean that passive aggressively: it may just not be worth fixing. It is kind of a poor user experience though.
I have an example where the errors I get (with duckdb-wasm) are:
RuntimeError: Invalid Error: don't know what type:
and
RuntimeError: Invalid Error: Variable-length int over 10 bytes.
So yeah, it's hard to tell what's happening. From this repo, we can only hope that the duckdb/parquet/arrow tooling becomes better. But it might help to open a relevant issue with them once we close in on what causes it.
I came across RuntimeError: unreachable executed message. I'm not sure why transforming my data to df <- df[1:nrow(df),]
helped, but it did.
library(arrow)
library(clinicalfd)
df <- clinicalfd::adsl
df <- df[1:nrow(df),]
# Write the data frame to a temporary Parquet file
temp_file <- tempfile(fileext = ".parquet")
arrow::write_parquet(df, sink = temp_file)
system2('/bin/cat', args = temp_file)
In an interactive session df <- clinicalfd::adsl
resolves directly to a data frame.
My guess is that in a non interactive session you instead get a Promise
which is not yet resolved when you save the parquet file.
Probably substituting
df <- df[1:nrow(df),]
with
rlang::eval_tidy(df)
would do...
Anyway, I would access the data from a package with utils::data()
so your code would be
# you need to have installed `rlang` (and `sas2r/clinical_fd`)
library(arrow)
utils::data(adsl, package = "clinicalfd") # you get `adsl` as a Promise object
rlang::eval_tidy(adsl) # resolve the Promise, now `adsl` is a data frame
# Write the data frame to a temporary Parquet file
temp_file <- tempfile(fileext = ".parquet")
arrow::write_parquet(adsl, sink = temp_file)
system2('/bin/cat', args = temp_file)
I tried both
rlang::eval_tidy(df)
# AND
force(df)
And it didn't help. I ran all_equal(df, df[1:nrow(df),]
and there are differences in attributes between both data frames. Original has attribute 'label' associated with column names, and transformed has them stripped off.
ORIGINAL
$ mmsetot : 'labelled' int 23 23 23 23 21 23 10 23 20 20 ... ..- attr(*, "label")= chr "MMSE Total"
TRANSFORMED
$ mmsetot : int 23 23 23 23 21 23 10 23 20 20 ...
It looks like there's an issue with loading parquet data with labelled column names.
@mrppdex you may want to file an issue on the parquet-wasm repo if you have a parquet file that reproducibly causes the issue
As far as I can tell, the parquet file is valid; unfortunately I can't post the whole thing because it's derived from logs at work.
This is the code to try and load the file:
RuntimeError: unreachable executed RuntimeError http://127.0.0.1:3000/_observablehq/runtime.js:2 value http://127.0.0.1:3000/_observablehq/runtime.js:2 promise callbackmr http://127.0.0.1:3000/_observablehq/runtime.js:2 wr http://127.0.0.1:3000/_observablehq/runtime.js:2 value http://127.0.0.1:3000/_observablehq/runtime.js:2 value http://127.0.0.1:3000/_observablehq/runtime.js:2 promise callbackvalue http://127.0.0.1:3000/_observablehq/runtime.js:2 value http://127.0.0.1:3000/_observablehq/runtime.js:2 ar http://127.0.0.1:3000/_observablehq/runtime.js:2 value http://127.0.0.1:3000/_observablehq/runtime.js:2 onmessage http://127.0.0.1:3000/_observablehq/client.js:355 onmessage http://127.0.0.1:3000/_observablehq/client.js:355 open http://127.0.0.1:3000/_observablehq/client.js:285