thombashi / sqlitebiter

A CLI tool to convert CSV / Excel / HTML / JSON / Jupyter Notebook / LDJSON / LTSV / Markdown / SQLite / SSV / TSV / Google-Sheets to a SQLite database file.
https://sqlitebiter.rtfd.io/
MIT License
850 stars 50 forks source link

Why is sqlitebiter rejecting this data as invalid json data format? #50

Closed davidshq closed 6 years ago

davidshq commented 6 years ago

Hi,

I'm trying to figure out why sqlitebiter doesn't like the json I am passing it. I've tried running it through jq, but that doesn't seem to help. I get the invalid json data format error with a message of inconvertible JSON schema.

Below is the json I'm trying to use: ``{ "name":"Zendesk Chat", "slug":"zopim-live-chat", "version":"1.4.12", "author":"<a href=\"http://www.zendesk.com/chat?iref=wp_plugin\">Zendesk", "author_profile":"https://profiles.wordpress.org/bencxr", "requires":"3.1", "tested":"4.7.10", "requires_php":false, "compatibility":[

], "rating":80, "ratings":{ "5":18, "4":1, "3":2, "2":1, "1":5 }, "num_ratings":27, "support_threads":1, "support_threads_resolved":0, "downloaded":925716, "last_updated":"2017-12-01 6:22am GMT", "added":"2010-01-20", "homepage":"http://www.zendesk.com/chat?iref=wp_plugin", "short_description":"Zendesk Chat (previously Zopim) lets you monitor and chat with visitors surfing your store in real-time. Impress them personally and ease them into th …", "download_link":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.12.zip", "screenshots":{ "4":{ "src":"https://ps.w.org/zopim-live-chat/trunk/screenshot-4.png?rev=1779235", "caption":"Dashboard View - Open new browser tab" }, "3":{ "src":"https://ps.w.org/zopim-live-chat/trunk/screenshot-3.png?rev=1779235", "caption":"Account Configuration - Linked Up with Launch Dashboard" }, "5":{ "src":"https://ps.w.org/zopim-live-chat/trunk/screenshot-5.png?rev=1779235", "caption":"Widget Customization from Dashboard" }, "1":{ "src":"https://ps.w.org/zopim-live-chat/trunk/screenshot-1.png?rev=1779235", "caption":"Chat window on your website - active chat" }, "2":{ "src":"https://ps.w.org/zopim-live-chat/trunk/screenshot-2.png?rev=1779235", "caption":"Account Configuration - Link Up" } }, "tags":{ "chat":"chat", "chat-online":"chat online", "contact-plugin":"contact plugin", "contact-us":"contact us", "customer-support":"customer support" }, "versions":{ "0.6.1":"https://downloads.wordpress.org/plugin/zopim-live-chat.0.6.1.zip", "0.7":"https://downloads.wordpress.org/plugin/zopim-live-chat.0.7.zip", "1.0":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.zip", "1.0.1":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.1.zip", "1.0.2":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.2.zip", "1.0.3":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.3.zip", "1.0.4":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.4.zip", "1.0.5":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.5.zip", "1.0.6":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.6.zip", "1.0.7":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.7.zip", "1.1.0":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.1.0.zip", "1.1.1":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.1.1.zip", "1.1.2":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.1.2.zip", "1.1.3":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.1.3.zip", "1.2.0":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.0.zip", "1.2.1":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.1.zip", "1.2.2":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.2.zip", "1.2.5":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.5.zip", "1.2.6":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.6.zip", "1.2.7":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.7.zip", "1.2.8":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.8.zip", "1.2.9":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.9.zip", "1.3.0":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.0.zip", "1.3.1":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.1.zip", "1.3.2":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.2.zip", "1.3.3":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.3.zip", "1.3.4":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.4.zip", "1.3.5":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.5.zip", "1.3.6":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.6.zip", "1.3.7":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.7.zip", "1.3.8":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.8.zip", "1.3.9":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.9.zip", "1.4.0":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.0.zip", "1.4.1":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.1.zip", "1.4.10":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.10.zip", "1.4.11":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.11.zip", "1.4.12":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.12.zip", "1.4.2":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.2.zip", "1.4.3":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.3.zip", "1.4.4":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.4.zip", "1.4.5":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.5.zip", "1.4.6":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.6.zip", "1.4.7":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.7.zip", "1.4.8":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.8.zip", "1.4.9":"https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.9.zip", "trunk":"https://downloads.wordpress.org/plugin/zopim-live-chat.zip" }, "donate_link":"" }``

thombashi commented 6 years ago

Hi,

That's because convertible JSON schemas by sqlitebiter are limited by now. However, I have some plans to enhance JSON support for the future release.

thombashi commented 6 years ago

@davidshq sqlitebiter 0.15.0 had improved JSON support. Now, you can convert the JSON as following tables:

$ sqlitebiter file issue50.json
[INFO] sqlitebiter file: convert 'issue50.json' to 'ratings' table
[INFO] sqlitebiter file: convert 'issue50.json' to 'screenshots_4' table
[INFO] sqlitebiter file: convert 'issue50.json' to 'screenshots_3' table
[INFO] sqlitebiter file: convert 'issue50.json' to 'screenshots_5' table
[INFO] sqlitebiter file: convert 'issue50.json' to 'screenshots_1' table
[INFO] sqlitebiter file: convert 'issue50.json' to 'screenshots_2' table
[INFO] sqlitebiter file: convert 'issue50.json' to 'tags' table
[INFO] sqlitebiter file: convert 'issue50.json' to 'versions' table
[INFO] sqlitebiter file: convert 'issue50.json' to 'root' table
[INFO] sqlitebiter file: number of created tables: 9
[INFO] sqlitebiter file: database path: out.sqlite

.. table:: ratings

+---+-----+
|key|value|
+===+=====+
|  5|   18|
+---+-----+
|  4|    1|
+---+-----+
|  3|    2|
+---+-----+
|  2|    1|
+---+-----+
|  1|    5|
+---+-----+

.. table:: screenshots_4

+-------+-------------------------------------------------------------------+
|  key  |                               value                               |
+=======+===================================================================+
|src    |https://ps.w.org/zopim-live-chat/trunk/screenshot-4.png?rev=1779235|
+-------+-------------------------------------------------------------------+
|caption|Dashboard View - Open new browser tab                              |
+-------+-------------------------------------------------------------------+

.. table:: screenshots_3

+-------+-------------------------------------------------------------------+
|  key  |                               value                               |
+=======+===================================================================+
|src    |https://ps.w.org/zopim-live-chat/trunk/screenshot-3.png?rev=1779235|
+-------+-------------------------------------------------------------------+
|caption|Account Configuration - Linked Up with Launch Dashboard            |
+-------+-------------------------------------------------------------------+

.. table:: screenshots_5

+-------+-------------------------------------------------------------------+
|  key  |                               value                               |
+=======+===================================================================+
|src    |https://ps.w.org/zopim-live-chat/trunk/screenshot-5.png?rev=1779235|
+-------+-------------------------------------------------------------------+
|caption|Widget Customization from Dashboard                                |
+-------+-------------------------------------------------------------------+

.. table:: screenshots_1

+-------+-------------------------------------------------------------------+
|  key  |                               value                               |
+=======+===================================================================+
|src    |https://ps.w.org/zopim-live-chat/trunk/screenshot-1.png?rev=1779235|
+-------+-------------------------------------------------------------------+
|caption|Chat window on your website - active chat                          |
+-------+-------------------------------------------------------------------+

.. table:: screenshots_2

+-------+-------------------------------------------------------------------+
|  key  |                               value                               |
+=======+===================================================================+
|src    |https://ps.w.org/zopim-live-chat/trunk/screenshot-2.png?rev=1779235|
+-------+-------------------------------------------------------------------+
|caption|Account Configuration - Link Up                                    |
+-------+-------------------------------------------------------------------+

.. table:: tags

+----------------+----------------+
|      key       |     value      |
+================+================+
|chat            |chat            |
+----------------+----------------+
|chat-online     |chat online     |
+----------------+----------------+
|contact-plugin  |contact plugin  |
+----------------+----------------+
|contact-us      |contact us      |
+----------------+----------------+
|customer-support|customer support|
+----------------+----------------+

.. table:: versions

+------+-----------------------------------------------------------------+
| key  |                              value                              |
+======+=================================================================+
|0.6.1 |https://downloads.wordpress.org/plugin/zopim-live-chat.0.6.1.zip |
+------+-----------------------------------------------------------------+
|0.7   |https://downloads.wordpress.org/plugin/zopim-live-chat.0.7.zip   |
+------+-----------------------------------------------------------------+
|1.0   |https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.zip   |
+------+-----------------------------------------------------------------+
|1.0.1 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.1.zip |
+------+-----------------------------------------------------------------+
|1.0.2 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.2.zip |
+------+-----------------------------------------------------------------+
|1.0.3 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.3.zip |
+------+-----------------------------------------------------------------+
|1.0.4 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.4.zip |
+------+-----------------------------------------------------------------+
|1.0.5 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.5.zip |
+------+-----------------------------------------------------------------+
|1.0.6 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.6.zip |
+------+-----------------------------------------------------------------+
|1.0.7 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.0.7.zip |
+------+-----------------------------------------------------------------+
|1.1.0 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.1.0.zip |
+------+-----------------------------------------------------------------+
|1.1.1 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.1.1.zip |
+------+-----------------------------------------------------------------+
|1.1.2 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.1.2.zip |
+------+-----------------------------------------------------------------+
|1.1.3 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.1.3.zip |
+------+-----------------------------------------------------------------+
|1.2.0 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.0.zip |
+------+-----------------------------------------------------------------+
|1.2.1 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.1.zip |
+------+-----------------------------------------------------------------+
|1.2.2 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.2.zip |
+------+-----------------------------------------------------------------+
|1.2.5 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.5.zip |
+------+-----------------------------------------------------------------+
|1.2.6 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.6.zip |
+------+-----------------------------------------------------------------+
|1.2.7 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.7.zip |
+------+-----------------------------------------------------------------+
|1.2.8 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.8.zip |
+------+-----------------------------------------------------------------+
|1.2.9 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.2.9.zip |
+------+-----------------------------------------------------------------+
|1.3.0 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.0.zip |
+------+-----------------------------------------------------------------+
|1.3.1 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.1.zip |
+------+-----------------------------------------------------------------+
|1.3.2 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.2.zip |
+------+-----------------------------------------------------------------+
|1.3.3 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.3.zip |
+------+-----------------------------------------------------------------+
|1.3.4 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.4.zip |
+------+-----------------------------------------------------------------+
|1.3.5 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.5.zip |
+------+-----------------------------------------------------------------+
|1.3.6 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.6.zip |
+------+-----------------------------------------------------------------+
|1.3.7 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.7.zip |
+------+-----------------------------------------------------------------+
|1.3.8 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.8.zip |
+------+-----------------------------------------------------------------+
|1.3.9 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.3.9.zip |
+------+-----------------------------------------------------------------+
|1.4.0 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.0.zip |
+------+-----------------------------------------------------------------+
|1.4.1 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.1.zip |
+------+-----------------------------------------------------------------+
|1.4.10|https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.10.zip|
+------+-----------------------------------------------------------------+
|1.4.11|https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.11.zip|
+------+-----------------------------------------------------------------+
|1.4.12|https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.12.zip|
+------+-----------------------------------------------------------------+
|1.4.2 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.2.zip |
+------+-----------------------------------------------------------------+
|1.4.3 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.3.zip |
+------+-----------------------------------------------------------------+
|1.4.4 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.4.zip |
+------+-----------------------------------------------------------------+
|1.4.5 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.5.zip |
+------+-----------------------------------------------------------------+
|1.4.6 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.6.zip |
+------+-----------------------------------------------------------------+
|1.4.7 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.7.zip |
+------+-----------------------------------------------------------------+
|1.4.8 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.8.zip |
+------+-----------------------------------------------------------------+
|1.4.9 |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.9.zip |
+------+-----------------------------------------------------------------+
|trunk |https://downloads.wordpress.org/plugin/zopim-live-chat.zip       |
+------+-----------------------------------------------------------------+

.. table:: root

+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|          key           |                                                                          value                                                                           |
+========================+==========================================================================================================================================================+
|name                    |Zendesk Chat                                                                                                                                              |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|slug                    |zopim-live-chat                                                                                                                                           |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|version                 |1.4.12                                                                                                                                                    |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|author                  |<a href=http: //www.zendesk.com/chat?iref=wp_plugin>Zendesk                                                                                               |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|author_profile          |https://profiles.wordpress.org/bencxr                                                                                                                     |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|requires                |3.1                                                                                                                                                       |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|tested                  |4.7.10                                                                                                                                                    |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|requires_php            |0                                                                                                                                                         |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|rating                  |80                                                                                                                                                        |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|num_ratings             |27                                                                                                                                                        |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|support_threads         |1                                                                                                                                                         |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|support_threads_resolved|0                                                                                                                                                         |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|downloaded              |925716                                                                                                                                                    |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|last_updated            |2017-12-01 6:22am GMT                                                                                                                                     |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|added                   |2010-01-20                                                                                                                                                |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|homepage                |http://www.zendesk.com/chat?iref=wp_plugin                                                                                                                |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|short_description       |Zendesk Chat (previously Zopim) lets you monitor and chat with visitors surfing your store in real-time. Impress them personally and ease them into th …|
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|download_link           |https://downloads.wordpress.org/plugin/zopim-live-chat.1.4.12.zip                                                                                         |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|donate_link             |                                                                                                                                                          |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
sanketdiwale commented 6 years ago

I am having a similar problem. I have an existing sqlite3 database that I use with a Django app and am trying to transfer it to a remote server. I tried using sqlitebiter as an intermediate format. I export my data to json from the sqlite database using Django's manage.py dumpdata functionality and then am trying to reconstruct a sqlite database on the remote server from the dumped json file using sqlitebiter. Is there a way to do this correctly. How do you get a compatible json file for biter?

thombashi commented 6 years ago

@sanketdiwale I'm not exactly sure what you are trying to do, but is it impossible to transfer the SQLite database file to a remote server as it is? or execute manage.py dumpdata at the local server and execute manager.py loaddata at the remote server. You would get more stable results.

sanketdiwale commented 6 years ago

Hi @thombashi. Thanks for the response. I managed to get the dumpdata - loaddata method working. I was getting Integrity errors earlier with the dump/load method and thus tried using sqlitebiter as a substitute. I am somewhat new to database management and thus wasn't familiar with the best way to do this. Thanks.

thombashi commented 6 years ago

@davidshq I'll close the issue. Feel free to reopen if you still have some problems.