cube2222 / octosql

OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
Mozilla Public License 2.0
4.75k stars 202 forks source link

Support common special characters in json filenames #275

Closed ChristianCiach closed 2 years ago

ChristianCiach commented 2 years ago

I am just trying out octosql for the first time and have a hard time to get it working.

For starters, this didn't work:

$ octosql 'SELECT * FROM customer.basedata.json'

Output:

Error: couldn't parse query: invalid argument syntax error at position 33

Seems like octosql doesn't like the . inside the filename. So let's change it to a dash...

$ octosql 'SELECT * FROM customer-basedata.json'

Output:

Error: couldn't parse query: invalid argument syntax error at position 24

Well..... maybe changing the dash to an underscore?

octosql 'SELECT * FROM customer_basedata.json'

Output:

Error: typecheck error: couldn't create datasource: couldn't scan lines: bufio.Scanner: token too long

Okay, now it was able to parse the argument but chokes on the contents.

I am using octosql 0.6.2 on Linux, so commit https://github.com/cube2222/octosql/commit/d4a8841c8609fb808430bea1369b24dcdb65ff7c should be included.

cube2222 commented 2 years ago

Hey @ChristianCiach! Thanks for the report!

As for filenames with special characters, you can use backticks in such situations.

'SELECT * FROM `customer.basedata.json`'

should work.

As for the token too long error:

  1. Is your file in JSONLines format? The file should be in a format where there's one JSON object per line.
  2. How long is your longest line? (I'll add the possibility to customize the maximum line length in the next release)
ChristianCiach commented 2 years ago
  1. No, it is a single json object, not one object per line.
  2. The file is about 6 MB in size and contains a single line of json.

I see now that this project doesn't fit my use case. That's fine, because I've found https://github.com/noborus/trdsql which fits my use case perfectly.

cube2222 commented 2 years ago

Sounds to me like that as well. Glad you found the right tool for your use case!