Cryptic error when merging on missing variable

[nalimilan]

If you try to merge two ffdf based on a variable that is missing from at least one of them, you get this cryptic error: Error : inherits(x, "ff_vector") & inherits(table, "ff_vector") is not TRUE

It would be nicer to print an explicit message for this common mistake. Just a small improvement! ;-)

[edwindj]

Thanks! I changed the code, it is on github.

It will be part of the next CRAN release (not sure when this will be, in a couple of weeks I expect),

[nalimilan]

Great! That's really the kind of small messages that makes your life easier when you're tired. ;-)

[asinofsky]

Hi! I am a relatively new R programmer and I have just downloaded ff and ffbase. I'm trying to use merge.ffdf and am receiving this exact error: "Error: inherits(x, "ff_vector") & inherits(table, "ff_vector") is not TRUE"

But, my "by" variable is present in both datasets. When I merge using the regular merge command it merges correctly, but produces a df, not an ffdf. So, to me that suggests my by variable etc. works.

In other words, this line works:

master<-merge(minidata, qbcoms, by='qbsal')

but this line produces the error:

rm(master)

master<-merge.ffdf(minidata, qbcoms, by='qbsal') Error: inherits(x, "ff_vector") & inherits(table, "ff_vector") is not TRUE

Googling the error only brought me to this thread. I was hoping one of you guys might by chance be able to help me, or else I'm completely stuck! Thanks for your time!

[nalimilan]

@asinofsky What the error says is that the data sets are not of the correct format. You can't call merge.ffdf directly on data frames, you need to convert them to ffdf objects using as.ffdf.

[asinofsky]

Thanks a bunch! I forgot to convert one of the dataframes.

[asinofsky]

Hey Nalimilan,

I know this isn't stack overflow, but I've found surprisingly little about ff and ffbase on stackoverflow/google and you were really helpful with my last question so I figured I'd ask another. I'm trying to do a many to many merge, which I now notice in the merge.ffdf documentation seems impossible with this function, since "Note that joining is done based on ffmatch or ffdfmatch: only the first element in y will be added to x"

Is there anyway to accomplish my goal with ffdf objects? Thanks again for your time! I'm gonna search all the ffbase and ff documentation, but nothing looks promising so far :(

[jwijffels]

Please direct your questions to stackoverflow and use the ff/ffbase tag. Someone else asked the same question: http://stackoverflow.com/questions/32516010/efficient-combination-and-operating-on-large-data-frames/32562569#32562569

On the answer. There is no such thing currently implemented doing a full outer joing. what you can do for the many-to-many case is use expand.ffgrid of your vectors which you want to a 'many to many' merge and next use a merge.ffdf. to add the information to the newly created ffdf