search

Juniper / libxo

The libxo library allows an application to generate text, XML, JSON, and HTML output using a common set of function calls. The application decides at run time which output style should be produced.
http://juniper.github.io/libxo/libxo-manual.html
BSD 2-Clause "Simplified" License
321 stars 48 forks source link

'plain' output with a defined field-separator #57

Closed Jamie-Landeg-Jones closed 4 years ago

Jamie-Landeg-Jones commented 6 years ago

Hi. I notice that the output from "df" is tricky to parse, when the mount "device" and/or the mount-point contain spaces.. (yeah, i know... don't ask!!! )

"mount -p" uses (one or more) tab characters to separate fields, UNLESS the field is over 24 character in which case it uses space. (arrrrgh)

I see that the idea behind libxo is to enable processing of data, by not having to try and parse human-readable output. This is great, and just what I'd want.

'df' does indeed have libxo functionality, and whilst this would do the trick, parsing xml or json is overkill. I was hoping it would be possible to have something like:

--libxo=delimited,delimiter="",record-separator=""

e.g.

--libxo=delimited,delimiter="\0",record-separator="\n"

or maybe being able to pass a format string in printf format?

Is this kind of thing generally useful?

In my particular case, I ended up writing a small C function just to grab the mount "device" and mount-point I needed, but it seemed to me that this what exactly the sort of thing libxo was meant to solve.

Am I missing something?

Cheers, Jamie

philshafer commented 6 years ago

On Dec 30, 2017, at 6:26 PM, Jamie Landeg Jones notifications@github.com wrote:

'df' does indeed have libxo functionality, and whilst this would do the trick, parsing xml or json is overkill. I was hoping it would be possible to have something like:

--libxo=delimited,delimiter="",record-separator=""

The problem is that we emit data that is hierarchical, so field-based access would be fragile. Better than text/regex, but still fragile.

We have a set of shell functions that eat XML (sort of) and provide access to data. I’ll add this functionality to a future libxo release.

Thanks, Phil

philshafer commented 5 years ago

I've been thinking about this and realized that with a little work, I can make a "csv" encoder that does what you need. Sort of. If I'm seeing what you need correctly.

So imagine:

With no explicit "path", it emits all "lists". With no explict "leafs", it emits the set of fields under the first instance of the first list. Without the no-header, it emits a list of leafs that it's emitting.

Using the example of tests/core/test-01.c:

% test_01.test --libxo warn,encoder=csv:path=top/data/item sku,name,sold,in-stock,on-order GRO-000-415,gum,1412,54,10 HRD-000-212,rope,85,4,2 HRD-000-517,ladder,0,2,1 HRD-000-632,bolt,4123,144,42 % test_01.test --libxo warn,encoder=csv:path=top/data/item+leafs=name.sold name,sold gum,1412 rope,85 ladder,0 bolt,4123 water,17

So add the "no-header" to strip the header and "leafs=xxx" and you can trim the content to what you need.

Does that sounds reasonable and usable?

Check out commit a8e2070 (and the one before that). To see where I'm at. It's mostly coded, but needs some finishing touches and documentation.

Thanks, Phil

Jamie-Landeg-Jones commented 5 years ago

Apologies for the delay in replying.

That's brilliant - better than what I was hoping for (the ability to filter out leafs)

Hopefully this will get libxo added to more utilities - this is much better for shell scripts than parsing human orientated output!

I notice you've committed this to FreeBSD-Current. Are there plans to MFC to 12-stable?

Thanks again, Jamie

philshafer commented 5 years ago

Nov 11, 2019, at 12:48 PM, Jamie Landeg Jones notifications@github.com wrote:

That's brilliant - better than what I was hoping for (the ability to filter out leafs)

Thanks!

Hopefully this will get libxo added to more utilities - this is much better for shell scripts than parsing human orientated output!

Yes, I need to do some more conversions.

I notice you've committed this to FreeBSD-Current. Are there plans to MFC to 12-stable?

It’s not entirely in. The Makefile for encoders/csv/ broke the build in ways that are beyond me. I’ve a fix from sjg@ but am waiting on confirmation from bapt@ before I can commit it. But I’ll likely do it anyway if I don’t hear something soon (>1wk already). It seems to work fine, but I’ve no interest in breaking things again. After that, I’d be happy for more testing before MFCing it.

Thanks, Phil

philshafer commented 5 years ago

Oops. I need to recalibrate my “patience meter”. Turns out my email was only sent on Thursday night.

Thanks, Phil

On Nov 11, 2019, at 2:37 PM, Phil Shafer phil@juise.org wrote:

It’s not entirely in. The Makefile for encoders/csv/ broke the build in ways that are beyond me. I’ve a fix from sjg@ but am waiting on confirmation from bapt@ before I can commit it. But I’ll likely do it anyway if I don’t hear something soon (>1wk already). It seems to work fine, but I’ve no interest in breaking things again. After that, I’d be happy for more testing before MFCing it.

Jamie-Landeg-Jones commented 5 years ago

Ha! You must have been terrible as a kid leading up to Christmas, and presents!

As for freebsd, I'm on 12-stable, I just installed the latest from git (seeing as you'd released it by the time I got around to it). It installed in /usr/local/... and seems ok to me (df --libxo=encoders=csv works fine)

I used your install instructions - I did have to use "gmake" rather than "make" to get it to install though.

I couldn't get "xo" command line to work with the csv encoder, but I haven't really used that command before, so put that down to my own stupidity!

I am a bit puzzled by why it's --html/--xml/--json/--text/--encoder=csv and not --encoder=html/xml/json/text/csv but it's not important!

By the way, talking of Christmas, I don't want to be the guy who tells you that santa doesn't exist.... but apparently, the rail-gauge size is a myth! https://www.snopes.com/fact-check/railroad-gauge-chariots/

philshafer commented 5 years ago

On Nov 12, 2019, at 4:10 AM, Jamie Landeg Jones notifications@github.com wrote:

I used your install instructions - I did have to use "gmake" rather than "make" to get it to install though.

You may also need “env MAKE=gmake ../configure” to get it to work.

I couldn't get "xo" command line to work with the csv encoder, but I haven't really used that command before, so put that down to my own stupidity!

Strange; does “ldd which xo” say anything helpful? Or “xo —version”, which will report both the “xo” version and the libxo version.

I am a bit puzzled by why it's --html/--xml/--json/--text/--encoder=csv and not --encoder=html/xml/json/text/csv but it's not important!

xml/html/text/json are built in libxo. The “encoder” api allows dynamically loadable encoders to be add in an independent manner, so I can’t (and don’t want to) capture them as native options. “encoder=csv” tells libxo to load /usr/lib/libxo/encoders/csv.enc and use that library. It’s definitely awkward, but the truth is the rest of the options (to csv) bother me more, but it’s all unavoidable imho.

By the way, talking of Christmas, I don't want to be the guy who tells you that santa doesn't exist.... but apparently, the rail-gauge size is a myth! https://www.snopes.com/fact-check/railroad-gauge-chariots/ https://www.snopes.com/fact-check/railroad-gauge-chariots/ Thanks. You just cost me 1/2 an hour learning about railroad gauges and English rail tycoons, and other bits of useless information, like the English 7’0.25” gauge (which must have been an impressive engine) and the the fact that Toronto has two different gauges for streetcars. In the end, the story is mostly right, since Stephenson picked his width to match his house-drawn carriages, and his was the one picked as the standard. The "built by English expatriates” bit is rubbish; better to say the first engines were built by Englishmen. A completely fascinating waste of time ;^)

Thanks, Phil

philshafer commented 5 years ago

On Nov 12, 2019, at 9:40 PM, Phil Shafer phil@juise.org wrote:

I couldn't get "xo" command line to work with the csv encoder, but I haven't really used that command before, so put that down to my own stupidity!

I realized last night that you might be hitting the issue described in:

https://libxo.readthedocs.io/en/latest/encoders.html#csv-comma-separated-values https://libxo.readthedocs.io/en/latest/encoders.html#csv-comma-separated-values

Be aware that since the CSV encoder looks for data instances, when used with The “xo” Utility https://libxo.readthedocs.io/en/latest/xo.html#xo, the --instance option will be needed:

% xo --libxo encoder=csv --instance foo 'The {:product} is {:status}\n' stereo "in route" product,status stereo,in route

HTH, Phil

philshafer commented 4 years ago

The functionality has been added to the current libxo release.

Thanks, Phil