search

dnychka / fieldsRPackage

GNU General Public License v2.0
15 stars 3 forks source link

Use of fields with large sparse matrices #5

Open mcuma opened 9 months ago

mcuma commented 9 months ago

Hello,

I am an user support person with limited R experience working with an user who uses the fields package to process large data sets. Running mKrig on ~100000 data points, trying to use the Wendland covariance to force sparse matrix storage of the Cholesky matrix, I am getting integer overflows:

Error in if (force64 || fortran || sizemax > 2147483647 - 1) { : 
  missing value where TRUE/FALSE needed
In addition: Warning message:
In n2 * n1 : NAs produced by integer overflow

It appears this is coming from the spam package, so, I am looking into plugging in the spam64, but, I am still getting this error. So, I am wondering if I can build/tell fields to use spam64, or, if this is something else that is fixable through some other R coding trickery.

Thanks.

dnychka commented 9 months ago

My guess is that you are setting up the sparse model OK but the matrices are still to large.

Try using the fastTps function as part of fields and also pay attention to setting the value of the aRange parameter so that there are a limited number of nearest neighbors to each observation location. Best,

Douglas Nychka Personal email Cell 3037253199

From: mcuma @.> Date: Sunday, January 14, 2024 at 2:00 PM To: dnychka/fieldsRPackage @.> Cc: Subscribed @.***> Subject: [dnychka/fieldsRPackage] Use of fields with large sparse matrices (Issue #5)

Hello,

I am an user support person with limited R experience working with an user who uses the fields package to process large data sets. Running mKrig on ~100000 data points, trying to use the Wendland covariance to force sparse matrix storage of the Cholesky matrix, I am getting integer overflows:

Error in if (force64 || fortran || sizemax > 2147483647 - 1) { :

missing value where TRUE/FALSE needed

In addition: Warning message:

In n2 * n1 : NAs produced by integer overflow

It appears this is coming from the spam package, so, I am looking into plugging in the spam64, but, I am still getting this error. So, I am wondering if I can build/tell fields to use spam64, or, if this is something else that is fixable through some other R coding trickery.

Thanks.

— Reply to this email directly, view it on GitHubhttps://github.com/dnychka/fieldsRPackage/issues/5, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADHJ7U76WHIYKBQMEQXEWODYORBPHAVCNFSM6AAAAABB2LPHOWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA4DAOBZGU4DSMA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

mcuma commented 9 months ago

Thanks, I suggested the TPS to the researcher. I also discovered your LatticeKrig package which seems to work as well.

dnychka commented 9 months ago

Great! LatticeKrig is my more “modern” solution to this problem. Please encourage your researcher to contact me directly about help if you encounter any issues. Best, Doug

Douglas Nychka Personal email Cell 3037253199

From: mcuma @.> Date: Tuesday, January 16, 2024 at 1:28 PM To: dnychka/fieldsRPackage @.> Cc: Douglas Nychka @.>, Comment @.> Subject: Re: [dnychka/fieldsRPackage] Use of fields with large sparse matrices (Issue #5)

Thanks, I suggested the TPS to the researcher. I also discovered your LatticeKrig package which seems to work as well.

— Reply to this email directly, view it on GitHubhttps://github.com/dnychka/fieldsRPackage/issues/5#issuecomment-1894463021, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADHJ7U5EUPA35VMJOSGC73TYO3PE7AVCNFSM6AAAAABB2LPHOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJUGQ3DGMBSGE. You are receiving this because you commented.Message ID: @.***>