Variable as index - Githubissues

seresss commented 9 years ago

Is it possible to use a variable as index in the jq commands?

I'm trying something like

  jq '... test[$variable] ...'

And its giving me " error: variable is not defined"

ghost commented 9 years ago

I'm assuming that by "a variable" you mean a shell variable and not a jq variable. In that case, you need to use --arg, which is this sort of gateway between shell things and jq things. Try the following:

jq --arg jq_variable "$shell_variable" '... test[$jq_variable] ...'

pkoppstein commented 9 years ago

p.s. Of course, if "test" is an array, and if the value of $jq_variable is set on the command line, then you'll need to convert the string to a number, e.g. by writing: test[$jq_variable|tonumber]

hyperpallium commented 9 years ago

jq '... test[' $variable '] ...'

hyperpallium commented 9 years ago

Many people ask about using shell variables in jq scripts, and all the solutions are awkward. You can bet that this will continue in future, especially as jq grows in popularity, reaching a wider and more varied audience.

By preventing this problem, it will not only save the trouble of someone asking and someone answering here, it will also help those people who didn't ask, and instead gave up on jq and used some other solution. Let's not fail those users!

PROPOSAL: make jq work the way people expect it to work: inherit shell variables.

if jq defines a variable with the same name, it hides the shell variable
the hidden shell variable can still be accessed by a special syntax (maybe $$varname), or just using the present approaches.
the shell variable should be included at compile time, so type issues are avoided (as here), and so queries can be constructed (so the variable value doesn't have to be a JSON value) - that is, to work exactly as shell interpolation, which is what people expect

Of course, this is a hack and not dogmatic. But, (1) since jq already emulates shell (with pipes, $ variables etc), it is arguable that emulating shell further is logical. Indeed, it's what people expect. (2) in being "sed for json", jq presents itself as, and wants to be in, the unix tradition of getting stuff done, not the Haskell tradition of purity (despite being obviously inspired by and initially implemented in Haskell - it is stealing fire from the gods and bringing it to users, not insisting users become gods in order to have fire). (3) and most importantly, jq is much easier to write and clearer to read if you can just directly include shell variables.

I don't necessarily expect this proposal to be adopted, but I think many people may agree that some preventative solution for shell variables would be desireable.

pkoppstein commented 9 years ago

all the solutions are awkward.

What is awkward about env? Having to export a variable to make it accessible? Anything else would be courting disaster.

hyperpallium commented 9 years ago

Yes, you must state it an extra time before using it.
You also need to know about it, and the env syntax. It's a sensible design, but not expected. It's not in 1.4, nor in the manual, nor suggested for this issue, but only in the faq, which is in the wiki and not directly linked from the main jq page - documentation can change, of course, but it's still unexpected.

While bash scripts only see exported shell variables (and it's helpful for a full-fledged language with modules etc to require it), that's only true for scripts in separate files.

One-liner bash scripts see all shell variables without export. One-liners are an important use-case of jq; e.g. this issue itself is of a one-liner. (Could follow bash's lead and have different rules for file script vs one-liner).

If it works for bash one-liners, how is it courting disaster?

Getting back to addressing the problem itself, another approach is an error message for when an undefined jq variable is accessed, to suggest solutions e.g. Error undefined variable $var. If you're trying to access a shell variable, see --arg, env or unquote it

ghost commented 9 years ago

@pkoppstein Yep, I should totally have mentioned tonumber. Thanks!

@hyperpallium

I disagree completely on automatically turning shell variables into jq variables. Not only it is clearly reminiscent of register_globals, which is probably one of the most glaring security issues ever lauded as a feature in the history of computing, but it is also, essentially, implicit, under-the-hood, magical behaviour. Personally, I really like --arg: It is explicit and clear.

I also disagree with the idea of tying jq, as a language, to its command-line interface in such a strong way. Most of the ways in which I use jq do not require passing arguments from the shell.

Completely agreed on fixing the error message, though.

hyperpallium commented 9 years ago

@slapresta NB: Not tying the whole language, just for one-liners. Same as shell itself (which hasn't destroyed unix), which behaves differently in a one-liner vs. script. Yes, it is magical behaviour.

Um, for when the jq script is invoked as a one-liner within a shell script, jq would only import shell variables available to that script (not the whole environment). Conceptually, jq would be on the same access-level that it was invoked at.

The main objection I see to this is that it breaks the meaning of single quoting. Purely to illustrate, another way to achieve the effect is if jq didn't use any special characters that need single-quoting e.g. |. Then you could just say

jq ... test[$variable] ...

(Apart from the fact that it conflicts with jq's own variable syntax), it's behavior that is already there; the problem is quoting, when used in a one-liner. Maybe there is a better way to achieve this effect?

Anyway, as I said, I didn't expect the proposal to be adopted, but do you agree that the problem is real and important? That is, that the issue comes up frequently and will continue to do so? That is what I was really getting at. Perhaps some counter-suggestions will be proposed, beyond disagreement with this particular solution...?

ghost commented 9 years ago

I think the problem is real, but it doesn't really have a solution. Is there even an unified way for applications to access shell variables, other than those exported as environment variables?

The --arg solution is in line to what other command-line micro-languages do; it is essentially the same as awk's -v.

hyperpallium commented 9 years ago

Unfortunately, I think you're right that this problem occurs for all unix scripting languages (awk, sed etc), but there isn't a unified solution. I think that, apart from the magical solution:

Not fail users (so they aren't stopped, don't have to raise an issue) by giving a solution in the error message. In practice, this probably fixes what is probably the main problem.

syntax one-parameter --import that imports shell as jq variable (as you'd almost always use the same name, to avoid confusion, and that makes --arg name name unnecessarily redundant), so you don't need env.name syntax. e.g.

jq --import variable '... test[$variable] ...'

But users won't know about this, so it doesn't really address the issue of them being stopped,

I have to admit, at the moment I'm not on top of the reasoning behind how unix tools handle shell variable access; nor on top of that PHP article (would it apply to jq?) - though a comment there notes that the change (not auto-importing) that fixed the security problem still allowed access to shell variables, just with a different syntax, something like _GET["name"]. It seems that having some way to access the environment is reasonable. So, perhaps env.name without export, wouldn't be so bad? I don't understand well enough at the moment to judge this.

But again, that different syntax wouldn't help the main problem of users being stopped. It would just be a bit shorter.

wtlangford commented 9 years ago

Not fail users (so they aren't stopped, don't have to raise an issue) by giving a solution in the error message. In practice, this probably fixes what is probably the main problem.

Better error messages are always good! I think that would go a long way towards fixing the problem.

syntax one-parameter --import that imports shell as jq variable (as you'd almost always use the same name, to avoid confusion, and that makes --arg name name unnecessarily redundant), so you don't need env.name

I think I'm okay with this, though we are trying to minimize the number of command-line flags we're adding overall.

So, perhaps env.name without export, wouldn't be so bad?

Can you clarify what you mean by this? If you mean without saying "export SOMEVAR=someVal", then there's actually nothing we can do about that. If the variable isn't exported in the shell, it is not available to subprocesses. (The somewhat exception to that is $ SOMEVAR=someVal jq --arg somevar $SOMEVAR ..., but that's just exporting it to only the subprocess about to be run.)

Also, sed doesn't interact with your shell variables at all. If you want them expanded, you do it at shell-level. I feel that's true of the majority of these languages/tools and that there's definitely good reason for it.

nicowilliams commented 9 years ago

A command-line option to define all variables from the environment would be OK, but note that not every shell variable is exported.

ghost commented 9 years ago

@hyperpallium

Sorry, I didn't explain the PHP article correctly. It is not about shell variables (the use of dollar signs is merely a coincidence; that's how PHP denotes variables), but it is about a very similar thing. PHP had this setting called register_globals who was set to true by default for most of PHP's history. What it implied was that, when you accessed an URL pointing to a PHP script, like, say, /example.php?lol=wat&foo=bar, the variables $lol and $foo were automatically set to wat and bar for the execution of that script. This has been the source of countless security issues (imagine ?admin=true!) and it probably is the most commonly criticised part of PHP.

Your proposal is similar in that you would be taking data from one environment (HTTP parameters / shell variables) and dumping it on the global namespace of another environment (PHP scripts / jq filters). Of course, the security issues concerning jq are not there (yet?) but it's still a source of bugs.

I think this could be solved with a wiki article, and by making the error point to said wiki article.

pkoppstein commented 9 years ago

@slapresta wrote:

I think this could be solved with a wiki article

There's already a Q in the FAQ specifically about this. In fact, it's the first Q in the "General Questions" section (https://github.com/stedolan/jq/wiki/FAQ#general-questions). Maybe it could be improved by highlighting the the difference between unexported and exported variables. Maybe the real issue here is that it's time for jq 1.5 :-)

Meanwhile, thanks for helping to explain the wisdom of "export" and the folly of "register_globals".

hyperpallium commented 9 years ago

A guiding error message seems most helpful. I suggest self-contained, by brief example (would fully examples be worth it?):

for shell vars, use --args i "$i" or unquote: jq '.mylist[' $i ']'. See FAQ for more.

@slapresta Unfortunately, links in error messages inevitably get out of date. The security concern seems more theoretical concern at this stage (as you're in control of the environment - it's not internet-facing), but many security problems come as a surprise, so there's merit in preventing them! It seems there's an implementation problem, that jq cannot access un-exported shell variables anyway.

BTW: The FAQ is hard to find at present - people often check a FAQ before they'll read a manual, so it makes sense for it to be more prominent than the manual... Could link it from the front page, from the manual; maybe even from this issues tracker itself (not sure if that's possible with github?)

seresss commented 9 years ago

I'm trying to do something like that:

' if .[0]=="TESTE" then $shell_variable=1 else empty end'
echo $shell_variable

And I need that the shell_variable has the "1" value.

Is it possible?

pkoppstein commented 9 years ago

@seresss asked:

Is it possible?

The answer is "no" and "yes":

"no" in that subprocesses in general cannot alter their environment in the way your example seems to expect;
"yes" because you can write something like this:

shell_variable=$(jq 'if .... then 1 else empty end' INPUT.json)

seresss commented 9 years ago

@pkoppstein

Yes, I've doing this, But I need something more.... I want to iterate an array and assign multiples variables Like this:

if .[index]=="X" then $var1=1
elif .[index]=="Y" then $var2=2
elfi .[index]=="Z" then $var3=3

pkoppstein commented 9 years ago

@seresss - I suspect there is a better way to accomplish whatever it is you are really trying to do, but if you want to export multiple values from jq to the enclosing environment, then I would suggest using a technique along the lines illustrated by the following snippet, which assumes a bash shell:

#!/bin/bash

results=( $(jq -n '"a", "b", "c"' ) )

for ((i=0; i< ${#results}; i++))
do
  echo ${results[$i]}
done

If this does not help, then you may need to take a course in shell programming.

wtlangford commented 9 years ago

One could also do something like export $( jq '"var1=1", "var2=2", "var3=3"), but it seems... terrifying to me.

On Wed, Dec 3, 2014 at 12:00 PM, pkoppstein notifications@github.com wrote:

@seresss https://github.com/seresss - I suspect there is a better way to accomplish whatever it is you are really trying to do, but if you want to export multiple values from jq to the enclosing environment, then I would suggest using a technique along the lines illustrated by the following snippet, which assumes a bash shell:

!/bin/bash

results=( $(jq -n '"a", "b", "c"' ) )

for ((i=0; i< ${#results}; i++)) do echo ${results[$i]} done

If this does not help, then you may need to take a course in shell programming.

— Reply to this email directly or view it on GitHub https://github.com/stedolan/jq/issues/635#issuecomment-65446464.

nicowilliams commented 9 years ago

So, is there anything for us to do here? I think the most I'd want to do is add a --env argument to jq to act as if --arg had been used for each identifier-like variable in the environment. I'll close this, and if anyone wants --env they can open a new issue just for that.

jqlang / jq

Variable as index #635

!/bin/bash