Open mskyttner opened 3 years ago
Thanks for the issue!
It's not clear if we can support raw output or not. The flags stuff is a bit beyond my comprehension of jq and our interface with it. I'd like to support this, but probably can't sort this out myself.
Do you know of a proposed fix?
I tried to untangle it a little but I'm not so sure about the lowlevel stuff. I think a proposed fix perhaps would involve steps like:
Not brave enough to do a PR on it though, I'm afraid.
My workaround for now, when using jq with "--raw-output" while converting to CSV is to "shell out" to a 4MB docker image and passing results back to R:
# workaround but depends on docker and a 4 MB docker image with jq
# attempted to use stevedore first, but ran into issues with the command splitter
# and with capturing output from the command
# function to enable running "jq" with --raw-output through docker
docker_cli_runc <- function(slug, command, v_host, v_container) {
stopifnot(file.exists(v_host))
cli_runc <- function(slug, command, v_host, v_container)
sprintf("docker run --rm -v %s:%s %s %s",
v_host, v_container, slug, command)
cmd <- cli_runc(slug, command, v_host, v_container)
system(cmd, intern = TRUE)
}
# small example data
jsons <-
'{
"id": "u1003nxf",
"orcid": "",
"profile": {
"firstName": "John",
"lastName": "Doe"
}
}
{
"id": "u1002cfh",
"orcid": "",
"profile": {
"firstName": "Jo",
"lastName": "Doe"
}
}
'
# available on hosts disk
readr::write_lines(jsons, "~/temp/jsons")
# test running jq with --raw-output
docker_cli_runc(
slug = "docker.io/endeveit/docker-jq",
command = "cat /tmp/jsons | jq -r '[.id, .orcid] | @csv'",
v_host = "~/temp/jsons",
v_container = "/tmp/jsons"
)
# poor man's wrapper
jq <- function(file, query) {
command <- sprintf("cat /tmp/jsons | %s", query)
docker_cli_runc(
slug = "docker.io/endeveit/docker-jq",
command = command,
v_host = file, v_container = "/tmp/jsons")
}
# using it on arbitrary json files
library(magrittr)
"~/temp/jsons" %>%
jq("jq -r '[.id, .orcid] | @csv'") %>%
readr::read_csv(col_names = c("id", "orcid"))
Thanks - i will try to have a look soon, no promises
This is an issue or perhaps feature request related to having jqr support raw output from jq, with non-json return type. The use case is to use jqr to "push down" some queries/work to jq which would benefit from jqr supporting the "--raw-output" option when returning for example csv data.
An example of such a use case is working with a large json(ish) file, getting only some elements, converting those to csv (with all these steps pushed down to jq) and then from jqr picking up these non-json raw results.
At bash it would be a command similar to this one:
cat jsons | jq -r '[.id, .orcid] | @csv'
, and here is an illustration from R with some example data:Workarounds or fixes
There seem to be these options supported by jq as noted in a related issue. So currently in
jqr
, thejq_flags
there are more likejv_dump_string_flags()
as described here.So "expected" jq command line options are not currently passed in jqr, since they're hardcoded and set to 0 here unlike when jq runs at the command line.
It looks like there doesn't seem to be a way to pass a "--raw-output" option through
jqr
right now because of this. Everything gets converted to json first.If the jq query/program/DSL when processed could get a parameter for the jq options/flags passed in it it could branch out and use
jv_string_value
for raw outputs instead of passing everything tojv_dump_string
. Now everything goes tojv_dump_string
which is what I think causes the "double quoting" of quote characters before these non-json results are returned to jqr?Related issues
I think this issue relates to these other issues (with the variation that it would like to be able to use jqr to pass the --raw-output flag/option and then expect the return type to not be json in order to support a jq query push down which uses "|@csv"):