Closed jacobmealey closed 1 week ago
okay I need to figure out whats going on with the windows builds, the cygwin was was working on my local branch until very recently, might be something i changed on the way we are doing the file i/o. I will take a closer look this evening.
Edit: Okay, i got the cygwin build working. I just need to add libjson-c to the curl-for-win build. @vszakats would this be done by manually compiling and linking it in here: https://github.com/curl/curl-for-win/blob/main/trurl.sh ?
It's a new dependency, kind of a big deal to add support for one in general. One approach is to make it a build option, disabled by default, so existing trurl builds (including curl-for-win) continue to work without modification. Another option is to delete/disable the curl-for-win CI job till I (or someone else) get around adding support for it. A 3rd one is to add an option to disable this feature, and I update curl-for-win to set it.
...that said, have you thought of using a JSON parser that's a self-contained C89 source (with a fitting license)? It's still a dependency, but perhaps easier to integrate and carry.
have you thought of using a JSON parser that's a self-contained C89 source
I was looking a lot at https://github.com/DaveGamble/cJSON, but I didn't like how it looks and json-c has some nice features. Both this and json-c are in vcpkg, freebsd repos, and most linux distro repositories so i was hoping it wouldn't be too much work either way for maintainers.
If folks want a smaller ANSI c json library instead of the one I chose, I think that's a fair request and I can start working this again.
for now, I will put it behind a compile time flag as you suggested
Edit: There are a ton of other libraries listed on json.org that could be used too.
@bagder, i've made a few big modifications. Mainly I made it so it reads one json object at a time, so it now only allocates as much memory as required to parse the largest single json object, instead of attempting to read until the of the file. I also made it so it reads in a fixed buffer on the stack, and then allocates more data on the heap if need be as you suggested.
With libjson-c installed (I tested against v0.17), I was able to build your branch using make TRURL_JSON_IN=1 JSON_C_PREFIX=/bla
Worth setting JSON_C_PREFIX ?= /usr/local
in Makefile
so it only requires make TRURL_JSON_IN=1
? (PREFIX
also defaults to /usr/local
)
Hello, sorry for letting this PR go stale. I have been thinking about this for a while. I agree with bagder and others that we don't get much from having JSON input. It just an easy feature to suggest. At first glance it makes sense that if we produce JSON we should be able to consume it, but as has been mentioned in this PR as well as the related issues, we wouldn't get much mileage out of it. The only real use case I can think of is using some external program to act as a complex filter / manipulator in between two trurl calls:
$ trurl -f some_urls.txt --json | a_very_complicated_filter.py | trurl -j - > filtered_urls.txt
but like has been said so many times this can be done by adding in a jq
call to the pipeline.
Moving away from the user experience side into the technical implementation, I don't like these changes. I think the library I chose is really nice to use but it's overkill for this use case and a much simpler one should have been used. @bagder I think we should close this and open a discussion around it, i do not think this code should be merged in.
Thanks for being so open and flexible @jacobmealey, even after having spent the time doing this PR. Closing this now.
I remain unconvinced that JSON input is very useful because the reasons mentioned above.
This adds
--json-file
as an option to read from a file that contains a json representation of the urls. It works very similarly to--url-file
. The--json-file
input structure is designed to match roughly what the output of--json
produces. the simplest json object possible is:You can pass an arbitrary number of these url JSON objects and perform operations on them like you would with any other url in trurl.
Something that I did that may be up for debate is how the query is handled. I chose to make the query only be able to be set by an array of "keys" and "values" (the "params" section of --json") and it will throw a warning if the "query" component of "parts" is set.
A few examples:
Currently the feature is disabled by default, do turn it on you need to add
TRURL_JSON_IN
to your environment to enable it.fixes: #291