Parallel vs serialized calc.sst and calc.ohc functions

[marosteg]

I am experiencing issues with the parallelized versions of the sst and ohc likelihood calculation functions. Notably, calc.sst() yields likelihoods while calc.sst.par yields: Error in L.sst[, , i] <- lik.sst[[i]] : number of items to replace is not a multiple of replacement length which stems from code unique to the parallelized function. Similarly, calc.ohc yields likelihoods while calc.ohc.par does not (it does not yield an error though).

[camrinbraun]

Can you provide a MWE?

I assume you're working from the dev branch? These functions have both been working fine for me lately so perhaps something inherent to your specific use case.

[camrinbraun]

I am experiencing issues with the parallelized versions of the sst and ohc likelihood calculation functions. Notably, calc.sst() yields likelihoods while calc.sst.par yields: Error in L.sst[, , i] <- lik.sst[[i]] : number of items to replace is not a multiple of replacement length which stems from code unique to the parallelized function.

This issue stemmed from the use of MUR sst data which was being automatically aggregated (coarsened) to improve likelihood calculation speed. The aggregated sst data was not matched in dimensions by the attempted likelihood calculation, causing an error. This has been fixed with #43. Please review and/or confirm the fix and we'll merge to dev.

[camrinbraun]

Similarly, calc.ohc yields likelihoods while calc.ohc.par does not (it does not yield an error though).

I am unable to reproduce this issue. I can calculate OHC based likelihoods using either approach. Here's the output of plot(L.ohc.par[[1:5]]) with your example data (note that days 4 and 5 are missing input pdt data):

[marosteg]

I just get five blank days (rather than three with likelihoods and two blank). Perhaps compare sessions?

sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] HMMoce_1.1.0      analyzePSAT_0.1.0 raster_3.4-5      sp_1.4-4          reshape2_1.4.4   
 [6] tidyfast_0.2.1    tags2etuff_1.0.0  forcats_0.4.0     stringr_1.4.0     dplyr_1.0.2      
[11] purrr_0.3.4       readr_1.3.1       tidyr_1.1.2       tibble_3.0.4      tidyverse_1.3.0  
[16] ggplot2_3.3.2     lubridate_1.7.9.2 fields_11.6       spam_2.5-1        dotCall64_1.0-0  
[21] pracma_2.2.9      Matrix_1.2-18     oce_1.2-0         gsw_1.0-5         testthat_2.3.2   

loaded via a namespace (and not attached):
 [1] colorspace_1.4-1   ellipsis_0.3.1     rgdal_1.4-8        rprojroot_1.3-2    fs_1.3.1          
 [6] rstudioapi_0.11    farver_2.0.3       remotes_2.2.0      fansi_0.4.1        xml2_1.2.2        
[11] codetools_0.2-16   ncdf4_1.17         doParallel_1.0.16  polyclip_1.10-0    pkgload_1.1.0     
[16] readbitmap_0.1.5   jsonlite_1.7.1     broom_0.7.0        dbplyr_1.4.2       png_0.1-7         
[21] rgeos_0.5-5        ggforce_0.3.2      compiler_3.6.3     httr_1.4.2         backports_1.1.8   
[26] assertthat_0.2.1   bmp_0.3            cli_2.0.2          tweenr_1.0.1       prettyunits_1.1.1 
[31] tools_3.6.3        igraph_1.2.5       gtable_0.3.0       glue_1.4.2         maps_3.3.0        
[36] Rcpp_1.0.5         cellranger_1.1.0   RNetCDF_2.4-2      vctrs_0.3.2        iterators_1.0.12  
[41] ps_1.3.4           rvest_0.3.6        lifecycle_0.2.0    devtools_2.3.2     MASS_7.3-53       
[46] scales_1.1.1       hms_0.5.3          parallel_3.6.3     RColorBrewer_1.1-2 curl_4.3          
[51] memoise_1.1.0      gridExtra_2.3      stringi_1.4.6      imager_0.42.3      maptools_1.0-2    
[56] desc_1.2.0         foreach_1.5.1      tiff_0.1-5         pkgbuild_1.1.0     rlang_0.4.7       
[61] pkgconfig_2.0.3    bitops_1.0-6       lattice_0.20-41    labeling_0.3       processx_3.4.4    
[66] tidyselect_1.1.0   fasttime_1.0-2     plyr_1.8.6         magrittr_2.0.1     R6_2.4.1          
[71] generics_0.0.2     DBI_1.1.0          pillar_1.4.6       haven_2.3.1        foreign_0.8-75    
[76] withr_2.3.0        RCurl_1.98-1.2     modelr_0.1.5       crayon_1.3.4       utf8_1.1.4        
[81] jpeg_0.1-8.1       usethis_1.6.3      locfit_1.5-9.4     readxl_1.3.1       data.table_1.12.8 
[86] callr_3.5.1        reprex_0.3.0       digest_0.6.25      munsell_0.5.0      sessioninfo_1.1.1 

[camrinbraun]

I just get five blank days (rather than three with likelihoods and two blank). Perhaps compare sessions?

sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] HMMoce_1.1.0      analyzePSAT_0.1.0 raster_3.4-5      sp_1.4-4          reshape2_1.4.4   
 [6] tidyfast_0.2.1    tags2etuff_1.0.0  forcats_0.4.0     stringr_1.4.0     dplyr_1.0.2      
[11] purrr_0.3.4       readr_1.3.1       tidyr_1.1.2       tibble_3.0.4      tidyverse_1.3.0  
[16] ggplot2_3.3.2     lubridate_1.7.9.2 fields_11.6       spam_2.5-1        dotCall64_1.0-0  
[21] pracma_2.2.9      Matrix_1.2-18     oce_1.2-0         gsw_1.0-5         testthat_2.3.2   

loaded via a namespace (and not attached):
 [1] colorspace_1.4-1   ellipsis_0.3.1     rgdal_1.4-8        rprojroot_1.3-2    fs_1.3.1          
 [6] rstudioapi_0.11    farver_2.0.3       remotes_2.2.0      fansi_0.4.1        xml2_1.2.2        
[11] codetools_0.2-16   ncdf4_1.17         doParallel_1.0.16  polyclip_1.10-0    pkgload_1.1.0     
[16] readbitmap_0.1.5   jsonlite_1.7.1     broom_0.7.0        dbplyr_1.4.2       png_0.1-7         
[21] rgeos_0.5-5        ggforce_0.3.2      compiler_3.6.3     httr_1.4.2         backports_1.1.8   
[26] assertthat_0.2.1   bmp_0.3            cli_2.0.2          tweenr_1.0.1       prettyunits_1.1.1 
[31] tools_3.6.3        igraph_1.2.5       gtable_0.3.0       glue_1.4.2         maps_3.3.0        
[36] Rcpp_1.0.5         cellranger_1.1.0   RNetCDF_2.4-2      vctrs_0.3.2        iterators_1.0.12  
[41] ps_1.3.4           rvest_0.3.6        lifecycle_0.2.0    devtools_2.3.2     MASS_7.3-53       
[46] scales_1.1.1       hms_0.5.3          parallel_3.6.3     RColorBrewer_1.1-2 curl_4.3          
[51] memoise_1.1.0      gridExtra_2.3      stringi_1.4.6      imager_0.42.3      maptools_1.0-2    
[56] desc_1.2.0         foreach_1.5.1      tiff_0.1-5         pkgbuild_1.1.0     rlang_0.4.7       
[61] pkgconfig_2.0.3    bitops_1.0-6       lattice_0.20-41    labeling_0.3       processx_3.4.4    
[66] tidyselect_1.1.0   fasttime_1.0-2     plyr_1.8.6         magrittr_2.0.1     R6_2.4.1          
[71] generics_0.0.2     DBI_1.1.0          pillar_1.4.6       haven_2.3.1        foreign_0.8-75    
[76] withr_2.3.0        RCurl_1.98-1.2     modelr_0.1.5       crayon_1.3.4       utf8_1.1.4        
[81] jpeg_0.1-8.1       usethis_1.6.3      locfit_1.5-9.4     readxl_1.3.1       data.table_1.12.8 
[86] callr_3.5.1        reprex_0.3.0       digest_0.6.25      munsell_0.5.0      sessioninfo_1.1.1 

This doesnt look out of the ordinary. Try using the calc.ohc.par() function this recent push https://github.com/camrinbraun/HMMoce/commit/1a54e99e161cbe811615014961f27cabf6ff9250

[camrinbraun]

@marosteg is this the lingering issue you mentioned to me? still having problems with this?

[marosteg]

I don't seem to have issues on this front anymore. The lingering thing is for you to test and approve #52.