Open graykode opened 4 years ago
Hi @graykode, thanks for posting this issue!
Quick note: In your example above, you may want to the following modifications:
tiledb.Attr(name="a2", dtype=np.float32)
to tiledb.Attr(name="a2", dtype=np.float64)
(i.e., use float64) for fairness to what you do with HDF5.ran = np.random.rand(n, 784)
outside the timing for both TileDB and HDF5How TileDB performs writes
TileDB follows a general algorithm that applies to both dense and sparse arrays, with any arbitrary filter (e.g., compression, encryption, etc) and all layouts. It works as follows:
Everything internally is heavily parallelized with TBB, but still the algorithm needs to perform the 2 fully copies I mention above.
What happens in your example
Your example is a special scenario where you have a single tile and, therefore, the cells in the numpy buffer have the same layout as the one that will be written to disk. Moreover, you are not specifying any filter. Consequently, it is possible to avoid the 2 extra copies, which is what I assume that HDF5 does. Hence the difference in performance.
The solution
We need to optimize for special scenarios like this. It is fairly easy before executing the query to determine whether it is a special scenario where no cell shuffling and filtering is involved. In those cases we should resort to directly writing from the user buffers (e.g., numpy arrays) to disk, bypassing the tile preparation and filtering steps.
Thanks for putting this on our radar. We'll implement the optimizations and provide updates soon.
@ihnorton @joe-maley @Shelnutt2
- np.float32
Thanks for your reply
First, I cannot create a schema dimension of float type. TileDBError: [TileDB::ArraySchema] Error: Cannot set domain; Dense arrays do not support dimension datatype 'FLOAT32 or 64'
Second, Removing ran = np.random.rand(n, 784)
out of timings does not make a significant difference in time with existing results.
I read your solution and understand that it simply means to save it in the form of a buffer, just np.save()
. Do you have an example code for this scenario using tiledb?
Thanks @stavrospapadopoulos
The float32
-> float64
is for the attribute a2
, not the dimensions. You currently set the type to np.float32
(32-bit float), whereas in PyTables you use Float64Atom
(64-bit float). In other words, you write less data in TileDB than HDF5. Here is all you need to do:
schema = tiledb.ArraySchema(domain=dom,
sparse=False,
attrs=[
tiledb.Attr(name="a2", dtype=np.float64) # <- this is the change here
])
Regarding the solution, you do not need to do anything :-). We need to implement the solution I suggested inside TileDB core and once we merge a patch, you will just get the performance boost without changing your code.
I hope this helps.
Regarding the solution, you do not need to do anything :-). We need to implement the solution I suggested inside TileDB core and once we merge a patch, you will just get the performance boost without changing your code.
I understand what you're saying. Then I will wait for your patch. Thanks
I compared numpy array of shapes (200000, 784) with the use of a dense-array in tileDB and the 'create_array' of pytable. However, the speed of the tileDB's I/O is significantly slower than that of the pytable.
tiledb
pytable
I'd like to know the cause. I look forward to hearing some solutions. @ihnorton