i'm on latest version of dataset:
1.4.1 - updated recently. Any chance to see & verify the version in my files directly?
I see an unexpected behavior with upsert_many(). As I understand and read in sources, upsert_many() consumes a list of dicts, like:
File "<ipython-input-12-bcc35e9d30ac>", line 1, in <module>
db['calculations'].upsert_many(sample_list, 'date')
File "C:\Python\lib\site-packages\dataset\table.py", line 290, in upsert_many
self.update_many(to_update, keys, chunk_size, ensure, types)
File "C:\Python\lib\site-packages\dataset\table.py", line 243, in update_many
print(stmt) # **<- Line inserted by me for inspection**
File "C:\Python\lib\site-packages\sqlalchemy\sql\elements.py", line 491, in __str__
return str(self.compile())
File "<string>", line 1, in <lambda>
File "C:\Python\lib\site-packages\sqlalchemy\sql\elements.py", line 481, in compile
return self._compiler(dialect, bind=bind, **kw)
File "C:\Python\lib\site-packages\sqlalchemy\sql\elements.py", line 487, in _compiler
return dialect.statement_compiler(dialect, self, **kw)
File "C:\Python\lib\site-packages\sqlalchemy\sql\compiler.py", line 592, in __init__
Compiled.__init__(self, dialect, statement, **kwargs)
File "C:\Python\lib\site-packages\sqlalchemy\sql\compiler.py", line 322, in __init__
self.string = self.process(self.statement, **compile_kwargs)
File "C:\Python\lib\site-packages\sqlalchemy\sql\compiler.py", line 352, in process
return obj._compiler_dispatch(self, **kwargs)
File "C:\Python\lib\site-packages\sqlalchemy\sql\visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "C:\Python\lib\site-packages\sqlalchemy\sql\compiler.py", line 2593, in visit_update
crud_params = crud._setup_crud_params(
File "C:\Python\lib\site-packages\sqlalchemy\sql\crud.py", line 64, in _setup_crud_params
return _get_crud_params(compiler, stmt, **kw)
File "C:\Python\lib\site-packages\sqlalchemy\sql\crud.py", line 177, in _get_crud_params
raise exc.CompileError(
CompileError: **Unconsumed column names: sample_col_2, sample_col_1**
In my sqlite database's table, there is a column "date" (is PrimaryKey, takes 'DATE'). The both columns not consumed should be created automatically, right? This does not happen. When I set the ensure = True argument explicitly, still get the same behavior.
Also, the given keys argument ['date'] is stored in the original dict as '_date' -> method update() is doing this. Is it intended for the method to work on the original dict instead on a temporary dict?
found a reason
I can't reproduce an other behavior atm, as I've restructured & rewritten the code in the meantime. I think it was seen with upsert() or with update() ??:
Having a table with columns 'date' (PK, type 'DATE'), 'sample1' (type FLOAT), 'sample2' (type FLOAT).
Having a list_A of dicts [{'date'=..., 'sample3'=...}, {'date'=..., 'sample3'=...} ...]
Calling upsert(list_A, ['date']) produces the right output, but deletes the columns 'sample1' and 'sample2' in the table.
If its necessary, i try to recreate the original code - I think, this behavior is somehow connected to the problem described above.
The behavior above is caused by setting the arg ensure = False and use _sync_columns() as its written in the docstring:
If automatic schema generation is disabled (ensure is False), this will remove any keys from the row for which there is no
matching column.
But still, is it intended for _sync_columns() to completely remove columns entries in a table??
Hi,
i'm on latest version of dataset: 1.4.1 - updated recently. Any chance to see & verify the version in my files directly? I see an unexpected behavior with upsert_many(). As I understand and read in sources, upsert_many() consumes a list of dicts, like:
upsert_many() is called with the sample_list above as arg:
yielding an error & warning: db['calculations'].upsert_many(sample_list, 'date') Traceback (most recent call last):
In my sqlite database's table, there is a column "date" (is PrimaryKey, takes 'DATE'). The both columns not consumed should be created automatically, right? This does not happen. When I set the ensure = True argument explicitly, still get the same behavior. Also, the given keys argument ['date'] is stored in the original dict as '_date' -> method update() is doing this. Is it intended for the method to work on the original dict instead on a temporary dict?
found a reason I can't reproduce an other behavior atm, as I've restructured & rewritten the code in the meantime. I think it was seen with upsert() or with update() ??: Having a table with columns 'date' (PK, type 'DATE'), 'sample1' (type FLOAT), 'sample2' (type FLOAT). Having a list_A of dicts [{'date'=..., 'sample3'=...}, {'date'=..., 'sample3'=...} ...] Calling upsert(list_A, ['date']) produces the right output, but deletes the columns 'sample1' and 'sample2' in the table. If its necessary, i try to recreate the original code - I think, this behavior is somehow connected to the problem described above.
The behavior above is caused by setting the arg
ensure = False
and use _sync_columns() as its written in the docstring: If automatic schema generation is disabled (ensure
isFalse
), this will remove any keys from therow
for which there is no matching column. But still, is it intended for _sync_columns() to completely remove columns entries in a table??Am I doing something wrong?
Thank you guys for dataset!