saulpw / aipl

Array-Inspired Pipeline Language
MIT License
119 stars 7 forks source link

json loading examples #21

Closed dovinmu closed 1 year ago

dovinmu commented 1 year ago

I have a script chains/rowan/load-json.aipl that is getting at some of what I'd want to do with AIPL commands, but have to do with Python currently.

Since this is an example script of how this would be done now we could merge it (although the fetch-url command doesn't accept a hardcoded URL like I'm giving it, need to fix that). Or we could try to add in some/all of these JSON manip commands, but I'm probably not the one to do that this week

dovinmu commented 1 year ago

accessing columns by name feels inconsistent, although I think it's just the presence of nested tables making things more confusing. eg I'm getting user input like so:

!require-input
keyword search
!split>keyword sep=\n

then getting some JSON like so

!format
https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/irony_identification/task.json
!fetch-url
!format>statement
{input}

(!require-input doesn't play nice with !literal) I can still use keyword sometimes, like here:

!match {keyword}
!filter

but eg these will break if there's that nested table:

!format
{keyword}
!columns keyword statement

For !columns in particular, I can see two behaviors that'd be useful here 1) trim the entire table of extaneous stuff in order to save some of it to json or 2) actually ravel it all into a 1D table, which would mean duplicating the value of keyword across many rows