Show which part of input cause error

Izzur commented 4 years ago

When I try to create sqlite from json and not using --no-type-inference, I got OverflowError like this

Traceback (most recent call last):
File "cli.py", line 14, in <module>
File "click/core.py", line 764, in __call__
File "click/core.py", line 717, in main
File "click/core.py", line 1137, in invoke
File "click/core.py", line 956, in invoke
File "click/core.py", line 555, in invoke
File "click/decorators.py", line 17, in new_func
File "sqlitebiter/sqlitebiter.py", line 271, in file
File "sqlitebiter/subcommand/_file.py", line 100, in convert
File "sqlitebiter/subcommand/_file.py", line 140, in __convert
File "sqlitebiter/_table_creator.py", line 27, in create
File "simplesqlite/core.py", line 1297, in create_table_from_tabledata
File "simplesqlite/core.py", line 1656, in __create_table_from_tabledata
File "simplesqlite/core.py", line 586, in insert_many
OverflowError: Python int too large to convert to SQLite INTEGER
[49281] Failed to execute script cli

The problem is I don't know which part or which line from input file cause the error So its hard to edit my input file if it contains thousands or millions line in them Is it possible for sqlitebiter to show which part cause error so user can check the input?

---edit to follow issue template---

What happens?

sqlitebiter shown error when I try to convert invalid json file, but the error is not verbose enough to know which line from input cause this error

What were you expecting to happen?

sqlitebiter shown error message containing information that the input is invalid on line XX

What are the steps to reproduce this issue?

User run command sqlitebiter file example.json

Any logs, error output, etc?

Error output here: https://ghostbin.co/paste/o4cdm

Environments

Operating System: macOS Catalina 10.15.3
Python Version: Python 3.7.6
sqlitebiter Version: 0.29.2
Installation Method: brew for macOS

thombashi commented 4 years ago

@Izzur Thank you for your report.

Could you describe how to reproduce? I could not reproduce the error with my environment. Uploading the your json file would be a great help if you could. Also, please fill out the Environment section of the bug_report issue template

This may be a bug. If you use --no-type-inference option, all of the data stored as TEXT. So, OverflowError would not be expected to raise.

Izzur commented 4 years ago

@thombashi I have updated the issue to follow template

I intentionally not use --no-type-inference

For sample json you can get here example.json.zip

thombashi commented 4 years ago

@lzzur Thank you for your additional information.

I misunderstood that you use --no-type-inference option.

Now, you can get the cause of errors at sqlitebiter 0.30.0 as the following:

$ sqlitebiter file example.json
[ERROR] example.json: failed to convert:
  overflow int found: row=1, col=no_ktp (36)
[INFO] converted results: source=0, fail=1

thombashi / sqlitebiter