zolekode / json-to-tables

Convert any JSON File to Normalised tables
GNU General Public License v3.0
15 stars 2 forks source link

when we try to append data to an existing database, id attribute starts from value 0 again. #1

Closed gkhehra closed 3 years ago

gkhehra commented 3 years ago

Screenshot (100)

As seen in the attached screenshot, while trying to append data to an existing database, id attribute is assigned value 0 as it creates table first and then saves to the database. Can we do something to fetch the last id value from database and increment in case of append operation. Thanks in advance.

zolekode commented 3 years ago

@researchDataP please can you share more details as to how you are currently calling the library?

gkhehra commented 3 years ago

@zolekode Thanks for your response. Yes, I'm trying to read a Json file(dem01.json) that I want to save as tables in the database. After that I want to read another Json file(demo2.json) with the same layout and save into the existing database and tables. But it reads demo2.json as a new file and generates ID starting from 0 and save into existing tables as shown in the screenshot attached i.e Id 0-7 for demo1.json data enteries and again Id 0- 6 for demo2.json but it should be 0-13 unique values as id needs to be a primary key. Can we modify something to make this id value always unique, even when we read and save multiple json files into the same database because we need a unique id as a foreign key to fetch data from subtables.

zolekode commented 3 years ago

Thanks for the detailed description @researchDataP I replicated the process but the ids keep increasing without any issues. My guess is that you create a new ExtentTable() object each time. Then, of course, it will reset the ID. Think about the extent table object as an object that manages the states of the tables.

So if you do extent_table = ExtentTable() a second time, you actually erase all the Infos (including the latest ids). So this is what you gotta do:

extent_table = ExtentTable()

table_maker = TableMaker(extent_table)

for json_obj in all_objects:
      table_maker.convert_json_objects_to_tables(json_obj, "your_root_table_name")

But, if for some reason, you want to save a table say on Monday and the next table on Saturday (probably because you don't have access to all the data at the same time, this is what you can do:

extent_table.save_extent_table_state() # this is something new I just added for you

extent_table = ExtentTable() # now if you happen to relauch your code another day

extent_table.load_extent_table_state() # just load the previous state from file

table_maker = TableMaker(extent_table) # and create the table maker as usual

Note that it just makes sure that all the ids are incremented properly.

Pull it and try it. I tested it and everything should be ok. You that you can pass the path of the folder to which you want to save the extent table data.

gkhehra commented 3 years ago

Thanks for your help!