Edit files in place

[ghost]

With GNU Sed for example you can edit files in place:

-i[SUFFIX], --in-place[=SUFFIX]

       edit files in place (makes backup if SUFFIX supplied)

However I see no option for this with JQ, currently I must do this:

br=$(mktemp)
jq .charlie delta.json > "$br"
mv "$br" delta.json

Here is a workaround, Awk slurp:

jq .firstName alfa.json | awk 'BEGIN{RS="";getline<"-";print>ARGV[1]}' alfa.json

[karelv]

+1

[ipmcc]

+1

[nicowilliams]

Scripting around this is not an option? In shell:

tmpf=$(mktemp --tmpdir=$(dirname $f) -t)
jq ... "$f" > "$tmpf"
mv -f "$tmpf" "$f"

[nicowilliams]

I'm asking why it's not acceptable.

[ipmcc]

By analogy: Would it be acceptable if <insert your favorite word processor here> made you save to a different file every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is cumbersome. The fact that something as fundamentally stream oriented as sed has an in-place option is a pretty good indicator that this is common expectation/desire.

[nicowilliams]

I see. I've generally not bothered with sed's -i option. For me the problem has been two-fold: a) it is (well, was) not portable enough, b) I always have to check whether such a tool's in-place update functionality is documented as working this way (rename into place), because I generally need atomicity. And then there's Windows (where renames generally don't work when there's open file handles for the file in question, unless all the processes that opened it used an option to allow it).

On Mon, Jun 9, 2014 at 10:23 AM, ipmcc notifications@github.com wrote:

By analogy: Would it be acceptable if made you save to a different file every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is cumbersome. The fact that something as stream oriented as sed has an in-place option is a pretty good indicator that this is common expectation/desire.

— Reply to this email directly or view it on GitHub https://github.com/stedolan/jq/issues/105#issuecomment-45502984.

[nicowilliams]

Anyways, I can add this easily. My only concern is to limit the number of new command-line options. Since this option couldn't be implemented in a jq program, I'll probably do it. Though perhaps a generic utility to wrap filters with might be better:

$ inplace some-file command args

which would run the given command with some-file as the standard input and which would redirect the command's output to a temporary file in the same directory then rename that into place. That's the Unix philosophy, after all.

On Mon, Jun 9, 2014 at 10:58 AM, Nico Williams nico@cryptonector.com wrote:

I see. I've generally not bothered with sed's -i option. For me the problem has been two-fold: a) it is (well, was) not portable enough, b) I always have to check whether such a tool's in-place update functionality is documented as working this way (rename into place), because I generally need atomicity. And then there's Windows (where renames generally don't work when there's open file handles for the file in question, unless all the processes that opened it used an option to allow it).

On Mon, Jun 9, 2014 at 10:23 AM, ipmcc notifications@github.com wrote:

By analogy: Would it be acceptable if made you save to a different file every time you wanted to save your document?

It's not that the problem can't be worked around, it's that doing so is cumbersome. The fact that something as stream oriented as sed has an in-place option is a pretty good indicator that this is common expectation/desire.

— Reply to this email directly or view it on GitHub https://github.com/stedolan/jq/issues/105#issuecomment-45502984.

[nicowilliams]

Ah, there's a sponge(1) that almost works this way:

http://joeyh.name/code/moreutils/ http://backreference.org/2011/01/29/in-place-editing-of-files/

I think both, sponge(1) and my idea of an inplace(1) would be very useful generally and very much in keeping with the Unix philosophy.

Would that work for you?

[nicowilliams]

@svnpenn @ipmcc @karelv

Regardless of how you feel about this issue, what think ye of this:

https://github.com/nicowilliams/inplace

?

[drorata]

Is the -i option available or not? In the linked commit it says that this feature was removed.

[pkoppstein]

@drorata - It was (as best I can tell) never available in any official jq release, and is certainly not available in jq 1.4 or 1.5. It was added as an experimental feature on Jul 20, 2014, and removed on Mar 5, 2015.

[pkoppstein]

@svnpenn - Although there are obviously circumstances when -i would be convenient, jq is more like awk than sed. The earlier attempt to endow jq with -i confirms that such attempts will be fraught with difficulties. awks do not have an -i option, and I think it is asking too much of the jq developers/maintainers to devise and implement such an option, especially given there are easy workarounds.

[nicowilliams]

Some of the refactoring to move I/O from main.c was done with in-place editing in mind, and is necessary to implement sed-style in-place editing. But it's not a high priority for me at this time, meaning: send us a PR if you want this done soon :)

[pkoppstein]

@svnpenn - My apologies if I over-generalized, but the GNU awk documentation characterizes "inplace editing" as "an extension" (https://www.gnu.org/software/gawk/manual/html_node/Extension-Sample-Inplace.html). Also, your example does not work with the GNU awk installed on my machine (4.0.2), and the awk that comes with OS X 10.9.5 does not support a "-i" option at all.

[dtolnay]

I agree that jq can and should do in-place. The earlier problems were due to a poor original implementation.

[nicowilliams]

The original implementation was poor and, in petticoat, had bad semantics. I refactored the way I/O is done in main.c in order to make it easier to later add sed-like -i -- it should be easy enough now. Send a PR or be patient :)

[danielbayley]

Is this feature in the next release? I found it in 'release candidate 1', or is that the version that was retracted?

[nicowilliams]

@danielbayley No, it got taken out. 1.5rc1 implemented the wrong -i semantics., sorry.

[nicowilliams]

There's no PR for it at the moment. Feel free to send one :)

[ghost]

+1

[BrendanThompson]

any action on this one?

[ChristianTacke]

Until there is a "-i" builtin option, could you add something like

    tfn="$(mktemp file.XXXXXX)" && jq '.' file >"$tfn" && mv "$tfn" file

to the Examples section of the manpage? And possibly add "sponge(1)" to the SEE ALSO section? When the inplace tool becomes more popular (included in moreutils?), it should be linked too.

[ChristianTacke]

@svnpenn you mean the ex(1)-Examples?. ex uses a tempfile internally. But anyway:

My suggestion is basicly to include anything appropiate in the examples to guide people in the right direction. Having seen the above http://backreference.org/2011/01/29/in-place-editing-of-files/ I was surpsied, on how many stupid ideas are out there to solve this problem. If that anything is the ex(1) example, also fine.

With the intention of "Hey user, you're looking for -i? Well it's not there yet for various good reasons, and well, you already have plenty of options to do this correctly with already existing tools".

[pkoppstein]

There is now a Q&A about "in-place editing" in the FAQ: https://github.com/stedolan/jq/wiki/FAQ#general-questions

[mfilenko]

I think tee can be used as a workaround:

jq '.foo' bar.json | tee bar.json > /dev/null

(Output redirection to the black hole is optional if quiet mode is not strictly necessary.)

[mfilenko]

Ah, good call, @svnpenn!

If jq fails the whole bar.json will be truncated, so must be used with care ;-)

[pkoppstein]

Some recommended and some not-recommended workarounds are covered on the FAQ page at "General Questions": https://github.com/stedolan/jq/wiki/FAQ#general-questions

[pkoppstein]

@svnpenn - Done.

[mfilenko]

The workaround with a temporary file can be hardened the following way for use in scripting:

jq '.foo' bar.json > baz.json && mv baz.json bar.json

Otherwise, it will also lead to an empty file in case of jq failure.

[p-groarke]

+1

[naugtur]

If this thread is not a proof the feature would save people a lot of time, I don't know what is.

[rjfellman]

Looking for this feature as well!

[pkoppstein]

@svnpenn - Neat. If that works on Windows, then it would be worthwhile having your slurp() packaged as sponge.bat since neither moreutils in general nor sponge in particular seems to be available via choco.

[jakub-bochenski]

contrary too @mfilenko suggestion using tee is not a workaround: https://askubuntu.com/a/752451

[moll]

You could use sponge from https://joeyh.name/code/moreutils/.

[nicowilliams]

There is https://github.com/nicowilliams/inplace.

[jakub-bochenski]

I know, those are both referenced in the FAQ. Just don't use tee (or direct > in general)

EDIT: just noticed there is a warning against tee in the FAQ too

[BrendanThompson]

Would love to see this in #1352

[ssbarnea]

Here is a more pragmatic question: What alternatives are for in-place editing of yaml or json files?

I am asking this because based on the history of this bug one thing is clear: the author doesn't want this in and it will not happen. If it didn't happen in 4 years, I doubt it will happen too soon so it makes sense to look for alternatives.

I am saying this because the last of this feature breaks the usability completely because it makes almost impossible to write reliable and easy to maintain scripts without it.

@pkoppstein OS X also has -i on awk, is just that its implementation is a little bit different than GNU awk, but is still possible to write code that works for both blends.

[k0pernikus]

@ssbarnea You can use sponge from moreutils together with jq:

 jq . your.json | sponge your.json

[lfarkas]

+1

[kislyuk]

+1

[mchalek]

fwiw +1

[hachi]

4.5 years later, still waiting for inline editing mode

[ssbarnea]

For me the lack of this feature was a reason enough for not using jq at all. I need to change a config file as part of our build process and this involves changing it multiple times during the process. Mainly we successfully used crudini in the past for doing the same to ini files.

The proposed workaround of using wrappers or other extra tools takes the dependency chain too far and we all know that we need and want to avoid having a kitchen-sink dependency chain. At the same time we want to keep out shell scripts clean and easy to read and update.

[joelittlejohn]

I think the options have been spelled out pretty clearly here. Either:

1. Send a PR
2. Add a :+1: to the description above and wait
3. Fork this repo and do whatever you like

It's not the author's responsibility to implement this feature. I've added my vote above as I think this would be a useful feature, but there's no amount of elapsed time that will make this the responsibility of the maintainers to implement. Remember, they have already contributed untold hours of work to this project, to your benefit, for free, to provide all the other features you find so useful.

I'm not trying to cause offence here, just let's remember that we should be grateful for the time that others have spent on this project already, not demanding more. If you need something, implement it and please (if you are so inclined) share your implementation with the rest of us.

[arikkfir]

In case this helps someone, if/until this is implemented - here's another shortcut to achieve this using bash but without temporary files and other tools (bash & jq only):

TJQ=$(cat ./my.json | jq ".myProp = \"newValue\"") && echo ${TJQ} > ./my.json

Sorry if this has already been suggested - couldn't find it in the comments above, but I might have missed it.

[fadado]

I don't like to put the full output in memory, but in this case the code can be clarified a bit. Subtle changes, not tested:

TJQ=$(jq '.myProp = "newValue"' < ./my.json) 
[[ $? == 0 ]] && echo "${TJQ}" >| ./my.json

The main changes are disable the field spliting in the echo argument and remove one cat subprocess.

[huan]

+1

[kellytgold]

+1 :( Once again I want to reiterate that this as a native feature of JQ would be very useful. I frequently finding myself needing to merge two or more json data sets. This means there is going to be a source and a destination.

If the source data set has 1000 objects and the destination data set has 10,000 objects we will need to maintain the extraneous objects that are in the destination data set. Unless there is something I am missing I don't actually see a way to do this without producing a new object and thus a new file.

Creating a tmp file and moving it back to the original does work but seems like a sad workaround

Bump