search

expersso / BIS

Programmatic access to BIS data
19 stars 10 forks source link

get_datasets drops amount outstanding measure in locational banking statistics #5

Open econ-erik opened 3 years ago

econ-erik commented 3 years ago 
> bis_location <- get_bis(bis_datasets$url[bis_datasets$name == "Locational banking statistics"], quiet = TRUE)
Parsed with column specification:
cols(
  .default = col_character()
)
See spec(...) for full column specifications.
|====================================================================================================| 100%  337 MB
Parsed with column specification:
cols(
  .default = col_double(),
  FREQ = col_character(),
  Frequency = col_character(),
  L_MEASURE = col_logical(),
  Measure = col_character(),
  L_POSITION = col_character(),
  `Balance sheet position` = col_character(),
  L_INSTR = col_character(),
  `Type of instruments` = col_character(),
  L_DENOM = col_character(),
  `Currency denomination` = col_character(),
  L_CURR_TYPE = col_character(),
  `Currency type of reporting country` = col_character(),
  L_PARENT_CTY = col_character(),
  `Parent country` = col_character(),
  L_REP_BANK_TYPE = col_character(),
  `Type of reporting institutions` = col_character(),
  L_REP_CTY = col_character(),
  `Reporting country` = col_character(),
  L_CP_SECTOR = col_character(),
  `Counterparty sector` = col_character()
  # ... with 6 more columns
)
See spec(...) for full column specifications.
|====================================================================================================| 100%  337 MB
Warning: 154340 parsing failures.
   row       col           expected   actual                                                                                                   file
141051 L_MEASURE 1/0/T/F/TRUE/FALSE S        '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
141051 1977-Q4   1/0/T/F/TRUE/FALSE 24.000   '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
141052 L_MEASURE 1/0/T/F/TRUE/FALSE S        '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
141052 1977-Q4   1/0/T/F/TRUE/FALSE 1060.000 '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
141053 L_MEASURE 1/0/T/F/TRUE/FALSE S        '/var/folders/9p/q05l96xj0cl_z2vb96crjzqh0000gn/T//RtmpVvNWQC/WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv'
...... ......... .................. ........ .......................................... [... truncated]
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] BIS_0.2.1           fredr_1.1.0.9000    censusapi_0.7.1     htmlTable_2.1.0     compareDF_2.3.0    
 [6] blsAPI_0.2.1        pdftools_2.3.1      magrittr_1.5        timetk_2.4.0        seasonal_1.7.1     
[11] HDInterval_0.2.2    smoother_1.1        TTR_0.24.2          cansim_0.3.5        jsonlite_1.7.1     
[16] geofacet_0.2.0      zoo_1.8-8           janitor_2.0.1       ggrepel_0.8.2       openxlsx_4.2.2     
[21] readxl_1.3.1        openintro_2.0.0     usdata_0.1.0        cherryblossom_0.1.0 airports_0.1.0     
[26] httr_1.4.2          rvest_0.3.6         xml2_1.3.2          RCurl_1.98-1.2      pander_0.6.3       
[31] DT_0.16             scales_1.1.1        lubridate_1.7.9     forcats_0.5.0       stringr_1.4.0      
[36] dplyr_1.0.2         purrr_0.3.4         readr_1.4.0         tidyr_1.1.2         tibble_3.0.4       
[41] ggplot2_3.3.2       tidyverse_1.3.0    

loaded via a namespace (and not attached):
  [1] backports_1.1.10    selectr_0.4-2       lazyeval_0.2.2      sp_1.4-4            splines_4.0.2      
  [6] listenv_0.8.0       digest_0.6.26       htmltools_0.5.0     fansi_0.4.1         checkmate_2.0.0    
 [11] recipes_0.1.14      globals_0.13.1      modelr_0.1.8        gower_0.2.2         xts_0.12.1         
 [16] sandwich_3.0-0      askpass_1.1         rsample_0.0.8       jpeg_0.1-8.1        strucchange_1.5-2  
 [21] colorspace_1.4-1    blob_1.2.1          haven_2.3.1         xfun_0.18           crayon_1.3.4       
 [26] survival_3.2-7      glue_1.4.2          gtable_0.3.0        ipred_0.9-9         imguR_1.0.3        
 [31] qpdf_1.1            DBI_1.1.0           Rcpp_1.0.5          units_0.6-7         lava_1.6.8         
 [36] prodlim_2019.11.13  htmlwidgets_1.5.2   geogrid_0.1.1       RColorBrewer_1.1-2  x13binary_1.1.39-2 
 [41] ellipsis_0.3.1      pkgconfig_2.0.3     farver_2.0.3        nnet_7.3-14         dbplyr_1.4.4       
 [46] tidyselect_1.1.0    labeling_0.4.2      rlang_0.4.8         munsell_0.5.0       cellranger_1.1.0   
 [51] tools_4.0.2         cli_2.1.0           generics_0.0.2      broom_0.7.2         evaluate_0.14      
 [56] yaml_2.2.1          knitr_1.30          fs_1.5.0            zip_2.1.1           future_1.19.1      
 [61] nlme_3.1-149        compiler_4.0.2      rstudioapi_0.11     curl_4.3            png_0.1-7          
 [66] e1071_1.7-4         reprex_0.3.0        stringi_1.5.3       rgeos_0.5-5         lattice_0.20-41    
 [71] Matrix_1.2-18       classInt_0.4-3      urca_1.3-0          vctrs_0.3.4         pillar_1.4.6       
 [76] lifecycle_0.2.0     vars_1.5-3          furrr_0.2.0         lmtest_0.9-38       data.table_1.13.0  
 [81] bitops_1.0-6        R6_2.4.1            KernSmooth_2.23-17  gridExtra_2.3       codetools_0.2-16   
 [86] MASS_7.3-53         assertthat_0.2.1    rjson_0.2.20        withr_2.3.0         rnaturalearth_0.1.0
 [91] mgcv_1.8-33         parallel_4.0.2      hms_0.5.3           rpart_4.1-15        timeDate_3043.102  
 [96] class_7.3-17        rmarkdown_2.4       snakecase_0.11.0    sf_0.9-6            tinytex_0.26
econ-erik commented 3 years ago

It appears the issue is with the L_MEASURE column as the quarterly changes have a value of "F" for more than 1000 rows. When I ran the following manual read_csv, I ended up with all of the rows.

bis_cols <- paste(c(rep("c",25),rep("d",171)), collapse = "", sep = "") bis_location <- read_csv(paste(bis_path,"WEBSTATS_LBS_D_PUB_DATAFLOW_csv_col.csv", sep = ""),col_names = TRUE, col_types = bis_cols) 

> dim(bis_location)
[1] 282137    196