Closed rbagd closed 6 years ago
Hi,
Thanks a lot for your feedback! We really appreciate it.
As you rightly point out, in your example, the problem is that you do not have a single suitable match for the one observation in one of your datasets. The latter leads to numerical underflow. We are constantly working on more robust ways to adjust for numerical underflow. In the meantime, we will add one warning message to describe the problem you encountered.
By the way, p.m
is the overall probability of finding a match among all the possible pairs in the cross product of two datasets. p.m
most of the time will be small, but that does not mean there is not a match in the data. The patterns and the probability of each pattern of being a match (zeta.j
) are way more informative. Note that zeta.j
includes p.m
in its calculation, so if p.m
is equal to zero, then that is a good indication that there is little overlap in terms of matches between datasets.
Finally, to give you a better recommendation on how to proceed, we would love to know a little bit more about the type of merge you are doing. It does not have to be specific but a brief description will suffice.
We are really glad that you find fastLink
of some help!
Ted
Thanks for the input @tedenamorado. In my use case, dfA
is a kind of a blacklist and I am looking for records in dfB
that match one or more columns within that blacklist (I do matching separately by different sets of columns). False match is very undesirable in such a case so I'd rather ignore anything numerically suspicious just to be sure. I noticed indeed that zeta.j
is typically zero for such edge cases, so no match ever happens anyway (hopefully) and the warning seems like not really a problem for my use case.
Again, thanks a lot for your feedback. Believe me, every day we are trying to think of better ways to detect and handle false positives =)
We have pushed an updated version of emlinkMARmov
that includes a more informative warning so that people facing knife-edge cases like the one you brought to our attention. In addition, we will release version 0.3 on CRAN really soon. The new release, besides incorporating your suggestion, will include new functions which will allow the comparison of numeric variables and confusion tables.
Please, if you find any further issues when using fastLink do not hesitate to let us know. We deeply care about what people using fastLink have to say.
Happy holidays!
I've been looking quite extensively into
fastLink
algorithm and notably the edge cases. In doing so, I think I've encountered a bug inemlinkMARmov
in the case of a poor match. If you unzip the attached file, you may runIf you try the above snippet out, you will obtain in most cases a warning that
p.old
is of different length thanp.new
. It doesn't happen always due to probable randomization. The reason is that after a few iterations,p.m = 0
and consequentlynum.prod = 0
. Due to zero division,p.gamma.k.m
becomesNaN
and sorting onNaN
leads to an empty vector - hence, the difference in length ofp.old
andp.new
. Sincep.old
andp.new
are of different lengths, it doesn't really make sense to subtract one from the other as you are recycling.I was thinking that
p.m = 0
essentially means there is no match, hence it doesn't really matter if we discontinue EM-algorithm and get out of the loop, i.e.if (p.m == 0) break
. I was wondering if you have some other ideas around it.By the way,
fastLink
is very nice work. Thanks a lot for it.fastlink.zip