search

shshemi / tabiew

A lightweight TUI application to view and query tabular data files, such as CSV, TSV, or parquet.
MIT License
541 stars 14 forks source link

consider supporting schemaless JSON #19

Closed bbkane closed 2 weeks ago

bbkane commented 3 weeks ago

Hello! I'm trying to open a JSONL log file, where some fields are in every object, but some fields are only in some objects.

This panics with found more fields than defined in 'Schema' at src/main.rs:37:39

I played with --no-header and --infer-schema, but couldn't make it work.

Would it be possible to support this format?

shshemi commented 3 weeks ago

Hi, Would you kindly provide a sample file to replicate the error?

bbkane commented 3 weeks ago

Here is the file as a zipfile: grabbit_log.zip.

It's a log from a little CLI tool I wrote called grabbit. I've verified it's valid JSONL by reading each line as a JSON object in Python with no issues.

I had to save it as a zip file so GitHub would let me attach it.

Here it is pasted in:

```jsonl {"_level":"INFO","_timestamp":"2024-10-30T22:03:09.561-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"architectureporn","post":"Akshardham temple, New Delhi, India.","filePath":"/Users/bbkane/Pictures/grabbit/architectureporn_Akshardham_temple,_New_Delhi,_India._cgruul8i0cxd1.png","url":"https://i.redd.it/cgruul8i0cxd1.png"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:09.565-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"architectureporn","post":"Fanjingshan Temples, Guizhou, China (530 x 680). How did they built this?","filePath":"/Users/bbkane/Pictures/grabbit/architectureporn_Fanjingshan_Temples,_Guizhou,_China_(530_x_680)._How_did_they_built_this?_6d69y5bw6wxd1.jpeg","url":"https://i.redd.it/6d69y5bw6wxd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:10.086-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"churchporn","post":"VERY old fresco of the flight into Egypt and of the entombment of Christ. Parochial Church of Salouf, Switzerland [3024x4032]","filePath":"/Users/bbkane/Pictures/grabbit/churchporn_VERY_old_fresco_of_the_flight_into_Egypt_and_of_the_entombment_of_Christ._Parochial_Church_of_Salouf,_Switzerland_[3024x4032]_3cwnatwaf3xd1.jpeg","url":"https://i.redd.it/3cwnatwaf3xd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:10.782-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"cityporn","post":"NYC","filePath":"/Users/bbkane/Pictures/grabbit/cityporn_NYC_utr87os6tuxd1.jpeg","url":"https://i.redd.it/utr87os6tuxd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:10.782-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"cityporn","post":"Bologna, Italy from above - Nicknamed \"The Red City\", guess why..","filePath":"/Users/bbkane/Pictures/grabbit/cityporn_Bologna,_Italy_from_above_-_Nicknamed_\"The_Red_City\",_guess_why.._cxcdc7g7thxd1.jpeg","url":"https://i.redd.it/cxcdc7g7thxd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:10.782-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"cityporn","post":"Jacaranda season in Sydney ","filePath":"/Users/bbkane/Pictures/grabbit/cityporn_Jacaranda_season_in_Sydney__qttb5qv3qnwd1.jpeg","url":"https://i.redd.it/qttb5qv3qnwd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:10.782-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"cityporn","post":"Downtown Manhattan at Night","filePath":"/Users/bbkane/Pictures/grabbit/cityporn_Downtown_Manhattan_at_Night_577kis8aa8xd1.jpeg","url":"https://i.redd.it/577kis8aa8xd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.258-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Glacial Melt Is Probably My Favorite Color. Southern Greenland. [OC] [2667x4000]","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Glacial_Melt_Is_Probably_My_Favorite_Color._Southern_Greenland._[OC]_[2667x4000]_rdvcenlc6pwd1.jpeg","url":"https://i.redd.it/rdvcenlc6pwd1.jpeg"} {"_level":"ERROR","_timestamp":"2024-10-30T22:03:11.404-0700","_caller":"grabbit/grab.go:271","_function":"main.grabSubreddit","_msg":"download file error","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Capitol Peak, Colorado (OC) [4500x3000]","url":"https://i.imgur.com/Wv373pK.jpeg","err":"Could not read contentBytes: https://i.imgur.com/Wv373pK.jpeg\n: EOF","errVerbose":"EOF\nCould not read contentBytes: https://i.imgur.com/Wv373pK.jpeg\n\nmain.downloadImage.func1\n\t/home/runner/work/grabbit/grabbit/grab.go:70\nmain.downloadImage\n\t/home/runner/work/grabbit/grabbit/grab.go:91\nmain.grabSubreddit\n\t/home/runner/work/grabbit/grabbit/grab.go:259\nmain.grab\n\t/home/runner/work/grabbit/grabbit/grab.go:411\ngo.bbkane.com/warg.(*App).MustRun\n\t/home/runner/go/pkg/mod/go.bbkane.com/warg@v0.0.17/app.go:207\nmain.main\n\t/home/runner/work/grabbit/grabbit/main.go:171\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/asm_amd64.s:1598\nmain.grabSubreddit\n\t/home/runner/work/grabbit/grabbit/grab.go:276\nmain.grab\n\t/home/runner/work/grabbit/grabbit/grab.go:411\ngo.bbkane.com/warg.(*App).MustRun\n\t/home/runner/go/pkg/mod/go.bbkane.com/warg@v0.0.17/app.go:207\nmain.main\n\t/home/runner/work/grabbit/grabbit/main.go:171\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/asm_amd64.s:1598"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.405-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Lanark Highlands, Ontario Canada [3024x4032] (OC)","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Lanark_Highlands,_Ontario_Canada_[3024x4032]_(OC)_a5f17xu363xd1.jpeg","url":"https://i.redd.it/a5f17xu363xd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.405-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Vermilion Lakes, Banff National Park, Canada [OC] [1667x2500]","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Vermilion_Lakes,_Banff_National_Park,_Canada_[OC]_[1667x2500]_pfmd7036vbxd1.jpeg","url":"https://i.redd.it/pfmd7036vbxd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.405-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Mt. Hood, seen from Mt. St. Helens in Washington [OC][2000x1333]","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Mt._Hood,_seen_from_Mt._St._Helens_in_Washington_[OC][2000x1333]_u5duwhk7x5xd1.jpeg","url":"https://i.redd.it/u5duwhk7x5xd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.405-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Sunrise and fog at my favorite National Park in the Netherlands [OC][1638x2048]","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Sunrise_and_fog_at_my_favorite_National_Park_in_the_Netherlands_[OC][1638x2048]_d4bm26067hxd1.jpeg","url":"https://i.redd.it/d4bm26067hxd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.405-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Fall foliage North Cascades [OC] [4000x6000]","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Fall_foliage_North_Cascades_[OC]_[4000x6000]_0hwdxxanqsxd1.jpeg","url":"https://i.redd.it/0hwdxxanqsxd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.406-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Reflections on Chephren Lake, Banff, Canada [OC][1500x2249]","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Reflections_on_Chephren_Lake,_Banff,_Canada_[OC][1500x2249]_bdswk8ps3rwd1.jpeg","url":"https://i.redd.it/bdswk8ps3rwd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.406-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Blueberry Mountain, Ontario Canada [3024x4032] (OC)","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Blueberry_Mountain,_Ontario_Canada_[3024x4032]_(OC)_cgeda7kizvxd1.jpeg","url":"https://i.redd.it/cgeda7kizvxd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.406-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"earthporn","post":"Colchuck Lake, Washington [OC] [3072x4080]","filePath":"/Users/bbkane/Pictures/grabbit/earthporn_Colchuck_Lake,_Washington_[OC]_[3072x4080]_j6q4of7miixd1.png","url":"https://i.redd.it/j6q4of7miixd1.png"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:11.825-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"futureporn","post":"Spaceport 03 by artist Josh Brockett","filePath":"/Users/bbkane/Pictures/grabbit/futureporn_Spaceport_03_by_artist_Josh_Brockett_0s8y1xnqjexd1.jpeg","url":"https://i.redd.it/0s8y1xnqjexd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:12.228-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"imaginarycastles","post":"The Hightower at Oldtown by artist Ted Nasmith","filePath":"/Users/bbkane/Pictures/grabbit/imaginarycastles_The_Hightower_at_Oldtown_by_artist_Ted_Nasmith_4bh82mp6k0xd1.jpeg","url":"https://i.redd.it/4bh82mp6k0xd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:12.228-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"imaginarycastles","post":"Château de la Mort by Greg Zdunek","filePath":"/Users/bbkane/Pictures/grabbit/imaginarycastles_Château_de_la_Mort_by_Greg_Zdunek_greg-zdunek-wit-001-seq-compfinal-e04.jpg","url":"https://cdna.artstation.com/p/assets/images/images/013/753/502/large/greg-zdunek-wit-001-seq-compfinal-e04.jpg?1540983487"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:12.228-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"imaginarycastles","post":"Heavy Is The Crown by Sébastien DL","filePath":"/Users/bbkane/Pictures/grabbit/imaginarycastles_Heavy_Is_The_Crown_by_Sébastien_DL_rxmgm7vm2pwd1.png","url":"https://i.redd.it/rxmgm7vm2pwd1.png"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:12.642-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"villageporn","post":"Pyrenees, Huesca, Spain","filePath":"/Users/bbkane/Pictures/grabbit/villageporn_Pyrenees,_Huesca,_Spain_ditk92sgd9xd1.jpeg","url":"https://i.redd.it/ditk92sgd9xd1.jpeg"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:12.643-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"villageporn","post":"Grey River, NL, Canada [Photo by Tom Cochrane]","filePath":"/Users/bbkane/Pictures/grabbit/villageporn_Grey_River,_NL,_Canada_[Photo_by_Tom_Cochrane]_pfxuv4mbvpwd1.jpeg","url":"https://i.redd.it/pfxuv4mbvpwd1.jpeg"} {"_level":"ERROR","_timestamp":"2024-10-30T22:03:13.077-0700","_caller":"grabbit/grab.go:271","_function":"main.grabSubreddit","_msg":"download file error","_pid":76777,"_version":"v4.4.20","subreddit":"roomporn","post":"Arched entrance to a subway tile-clad shower in a Salt Lake City residence, Utah [1440x1800]","url":"https://i.imgur.com/sneCo4E.jpg","err":"Could not read contentBytes: https://i.imgur.com/sneCo4E.jpg\n: EOF","errVerbose":"EOF\nCould not read contentBytes: https://i.imgur.com/sneCo4E.jpg\n\nmain.downloadImage.func1\n\t/home/runner/work/grabbit/grabbit/grab.go:70\nmain.downloadImage\n\t/home/runner/work/grabbit/grabbit/grab.go:91\nmain.grabSubreddit\n\t/home/runner/work/grabbit/grabbit/grab.go:259\nmain.grab\n\t/home/runner/work/grabbit/grabbit/grab.go:411\ngo.bbkane.com/warg.(*App).MustRun\n\t/home/runner/go/pkg/mod/go.bbkane.com/warg@v0.0.17/app.go:207\nmain.main\n\t/home/runner/work/grabbit/grabbit/main.go:171\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/asm_amd64.s:1598\nmain.grabSubreddit\n\t/home/runner/work/grabbit/grabbit/grab.go:276\nmain.grab\n\t/home/runner/work/grabbit/grabbit/grab.go:411\ngo.bbkane.com/warg.(*App).MustRun\n\t/home/runner/go/pkg/mod/go.bbkane.com/warg@v0.0.17/app.go:207\nmain.main\n\t/home/runner/work/grabbit/grabbit/main.go:171\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/asm_amd64.s:1598"} {"_level":"ERROR","_timestamp":"2024-10-30T22:03:13.580-0700","_caller":"grabbit/grab.go:238","_function":"main.grabSubreddit","_msg":"can't download image","_pid":76777,"_version":"v4.4.20","subreddit":"wallpapers","post":"Jupiter amoled ☄️ [3840x2160]","url":"https://www.reddit.com/gallery/1gd975f","err":"urlFileName doesn't end in allowed extension: \"1gd975f\" , []string{\".jpg\", \".jpeg\", \".png\"}\n ","errVerbose":"urlFileName doesn't end in allowed extension: \"1gd975f\" , []string{\".jpg\", \".jpeg\", \".png\"}\n \nmain.validateImageURL\n\t/home/runner/work/grabbit/grabbit/grab.go:143\nmain.grabSubreddit\n\t/home/runner/work/grabbit/grabbit/grab.go:236\nmain.grab\n\t/home/runner/work/grabbit/grabbit/grab.go:411\ngo.bbkane.com/warg.(*App).MustRun\n\t/home/runner/go/pkg/mod/go.bbkane.com/warg@v0.0.17/app.go:207\nmain.main\n\t/home/runner/work/grabbit/grabbit/main.go:171\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/asm_amd64.s:1598"} {"_level":"INFO","_timestamp":"2024-10-30T22:03:13.581-0700","_caller":"grabbit/grab.go:262","_function":"main.grabSubreddit","_msg":"file exists!","_pid":76777,"_version":"v4.4.20","subreddit":"wallpapers","post":"Bloodborne Hunter Vs Ludwig (3502x2160) Art by @Max58Art","filePath":"/Users/bbkane/Pictures/grabbit/wallpapers_Bloodborne_Hunter_Vs_Ludwig_(3502x2160)_Art_by_@Max58Art_1jvfdkshmqwd1.png","url":"https://i.redd.it/1jvfdkshmqwd1.png"} ```

The error message I get is:

$ tw ~/.config/grabbit.jsonl
thread 'main' panicked at src/main.rs:37:39:
found more fields than defined in 'Schema'

Consider setting 'truncate_ragged_lines=true'.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
zsh: abort      tw ~/.config/grabbit.jsonl

And my version of tabiew is:

$ tw --version
tabiew 0.7.0

Thanks for looking into this, I think tabiew could really help my experience looking at logs haha!

shshemi commented 3 weeks ago

I opened the file successfully using the following command:

tw grabbit.jsonl --format jsonl
bbkane commented 2 weeks ago

Hmm I thought it would infer that from the file name. I'll start passing it explicitly, thank you!