Function writeNumericPart does not handle large length of values vector

[intelinsight]

I am trying to write a .mat file using the writeMat function and faced the error message:

"Error in if (padding < 0) { : missing value where TRUE/FALSE needed"

Upon inspection, I found that: At line 18, if "length(values) * sizeOf" becomes greater than ".Machine$integer.max" then nbrOfBytes gets set to NA, resulting in failure of the program

Can you please fix it? Maybe, setting the sizeOf variable to '8' instead of '8L' (and similarly for miINT32 type) might fix it.

[HenrikBengtsson]

Thanks for the report. Your troubleshooting sounds correct to me. Would you mind putting together a minimum reproducible example causing writeMat() to give this error that I can work with to fix the problem?

[intelinsight]

Hi Henrik

Yes, sure:

Running:

library(R.matlab) writeMat(con='ParentDir/testLarge.mat',laregIntObj=seq.int(from = 1,to = .Machine$integer.max,by = 1))

Gives:

Error in if (padding < 0) { : missing value where TRUE/FALSE needed In addition: Warning message: In length(values) * sizeOf : NAs produced by integer overflow

[HenrikBengtsson]

A slightly smaller (in RAM) example:

> library("R.matlab")
> x <- integer(length=.Machine$integer.max/4+1)
> writeMat(con="longvector.mat", x=x)
Error in if (padding < 0) { : missing value where TRUE/FALSE needed
In addition: Warning message:
In length(values) * sizeOf : NAs produced by integer overflow
> traceback()
8: writeNumericPart(data)
7: writeNumericArray(name = name, data = value)
6: writeDataElement(con = NULL, object = object, nbrOfBytes = 0)
5: writeDataElement(con, object)
4: writeAll(con, objects)
3: writeMat5(con, objects = args, onWrite = onWrite)
2: writeMat.default(con = "longvector.mat", x = x)
1: writeMat(con = "longvector.mat", x = x)

[HenrikBengtsson]

Although the original error message can be fixed, there is a limitation of the MAT file format on how many bytes can be written per variable. This is a limitation by design, cf. http://www.mathworks.com/help/matlab/import_export/mat-file-versions.html.

It is only the MAT v7.3 file format that supports larger objects than 2^31 bytes. writeMat() only writes MAT v5 files. MAT v7.3 is a completely different creature and it wont be supported any time soon, cf. Issues #20 and #23.

I've updated R.matlab (develop branch) to now give the following error:

> library("R.matlab")
> x <- integer(length=.Machine$integer.max/4+1)
> writeMat("longvector.mat", x=x)
[2015-12-27 17:24:13] Exception: MAT file format error: Object is too large to be written to
a MAT v5 file, which only supports variables of maximum 2^31 bytes. The object that cannot
be written has 536870912 elements each of 4 bytes totalling 2147483648 bytes:
'./longvector.mat.tmp'

[HenrikBengtsson]

@intelinsight, I'm closing this issue since it's be resolved as far as possible.