piever
commented
6 years ago Off-Topic: don't call your data table as that's a pretty useful function that you're overwriting!
What is the output of typeof(column(table, :species))? And what is the length of unique(column(table, :species))? It seems like JuliaDB is trying to use PooledArray, (which is useful to represent a vector of strings with few unique values repeated many times) but failing as you actually hava a lot of different values of species.
Can you try:
using PooledArrays
table2 = setcol(table, :species, PooledArray(column(table, :species), UInt32))
result = groupby(length, table2, :species)and see if it fixes it?
What the error is telling you is that you used (or JuliaDB.load chose it for you, I don't know) a PooledArray(v, UInt16) which can have up to 2^16 different values, which apparently is not enough for your large dataset, so you need PooledArray(v, UInt32) which can have 2^32 = 4294967296 distinct values and that is certainly enough. Maybe the load function should take care of this for you though, so that the user never have to worry about these issues.
@shashi : why do we use PooledArrays and not CategoricalArrays like DataFrames does? Is it because of having different missing data?
EDIT: this thing that I'm suggesting, I'm not sure if it works also on distributed tables, better try it on the non-distributed (= 1 worker) case.
shashi
claireh93
richardreeve
I have a large (~20GB) JuliaDB table saved in 8 chunks that I am now trying to perform some summary functions on. For example,
:speciesis a column of Strings and I would like to know how many instances I have of each unique string. However, I run into errors when using multiple workers (e.g.ERROR: Error deserializing a remote exception from worker 2 Remote(original) exception of type TypeError).Running it on just one worker seems to give me a more informative error:
I tested out this function on a smaller test table and it seemed to work fine, could there be a problem with the way I have saved or loaded the data? Thanks!