Run data preparation back to 1994

[athowes]

• [x] CMR issue: resolved by.. not including any surveys before 1999 :magic_wand:

  > surveys <- surveys_add_dhs_regvar(surveys, survey_region_boundaries)
  [ failed     ]  2021-12-06 13:17:42
  [ elapsed    ]  Failed report in 14.0313 secs
  Error in surveys_add_dhs_regvar(surveys, survey_region_boundaries) : 
  Survey regions dataset has multiple REGVAR: CMR1998DHS
  In addition: There were 26 warnings (use warnings() to see them)

• [x] TZA issue: (solved by just excluding 1999, which according to statcompiler doesn't have behavioural data anyway)

  > survey_clusters <- assign_dhs_cluster_areas(survey_clusters, survey_region_areas)
  [ failed     ]  2021-12-06 14:46:30
  [ elapsed    ]  Failed report in 1.961909 mins
  Error in assign_dhs_cluster_areas(survey_clusters, survey_region_areas) : 
  Survey clusters have survey_region_id not contained in survey_regions:
   survey_id survey_region_id cluster_id
  1   TZA1999DHS                1          1
  2   TZA1999DHS                2          2
  3   TZA1999DHS                2          4
  4   TZA1999DHS                2          6
  5   TZA1999DHS                2         10
  6   TZA1999DHS                2         12
  7   TZA1999DHS                2         14
  8   TZA1999DHS                1         17
  9   TZA1999DHS                1         19
  10  TZA1999DHS                2         21
  11  TZA1999DHS                2         23
  12  TZA1999DHS                2         25
  13  TZA1999DHS                2         27
  14  TZA1999DHS                2         29
  15  TZA1999DHS                2         31
  16  TZA1999DHS                1         33
  17  TZA1999DHS                1         35
  18  TZA1999DHS                2         36
  19  TZA1999DHS                2         38
  20  TZA1999DHS                2         42
  21  TZA1999DHS 

• [x] ZAF issue:

  > survey_region_boundaries <- create_survey_boundaries_dhs(surveys)
  Downloading DHS region boundaries: ZA1998DHS
  [ failed     ]  2021-12-06 14:31:21
  [ elapsed    ]  Failed report in 3.91725 secs
  Error in download.file(url, tf2, quiet = quiet_download) : 
  cannot open URL 'https://gis.dhsprogram.com/arcgis/rest/directories/arcgisjobs/tools/downloadsubnationaldata_gpserver/jc3e6bfce21a4481b93ffb219175ca04b/scratch/sdr_subnational_boundaries_2021-12-06.zip'
  In addition: There were 50 or more warnings (use warnings() to see the first 50)

  > individuals <- create_individual_hiv_dhs(surveys)
  [ failed     ]  2021-12-06 14:50:55
  [ elapsed    ]  Failed report in 9.101696 secs
  Error in `[.data.frame`(filenames, , nm_type) : 
  undefined columns selected 

  [ failed     ]  2021-12-07 17:53:43
  [ elapsed    ]  Failed report in 55.94867 secs
  Error: Problem with `filter()` input `..1`.
  ℹ Input `..1` is `>=...`.
  x must be a character vector, not an integer vector.

• [x] ZWE issue:

  Downloading DHS region boundaries: ZW1999DHS
  [ failed     ]  2021-12-06 14:34:08
  [ elapsed    ]  Failed report in 2.4942 secs
  Error in download.file(url, tf2, quiet = quiet_download) : 
  cannot open URL 'https://gis.dhsprogram.com/arcgis/rest/directories/arcgisjobs/tools/downloadsubnationaldata_gpserver/jd36876ad17924d39b399e0ad208108f5/scratch/sdr_subnational_boundaries_2021-12-06.zip'
  In addition: There were 50 or more warnings (use warnings() to see the first 50)

  
  > #' Survey indicator dataset
  > survey_indicators <- calc_survey_indicators(

• survey_meta,
• survey_regions,
• survey_clusters,
• survey_individuals,
• survey_biomarker,
• survey_other,
• st_drop_geometry(areas),
• sex = sex,
• age_group_include = age_group_include
• ) [ failed ] 2021-12-06 14:54:46 [ elapsed ] Failed report in 1.683393 mins Error: Problem with filter() input ..1. ℹ Input ..1 is >=.... x must be a character vector, not an integer vector.

[athowes]

First inclusion of any suitable sexual behaviour questions is in 1999, suggest that as the cut-off (Zimbabwe 1999 DHS).

[athowes]

On why there was an error in ZWE and ZAF relating to strings and integers:

Eaton, Jeffrey W

the reason is because that call includes the 1988 ZWE DHS which did not include men So for that survey a "" string is returned, which R auto-parses the column as a character column rather than an integer should add to naomi.utils to cast those columns to integer, which will convert the "" to an NA

Fixed here https://github.com/mrc-ide/naomi.utils/issues/13

[athowes]

Use the plot_available-surveys to clean up any remaining missing sex paid question surveys and then this can be closed.

[athowes]

Should be finished. :disappointed_relieved:

[athowes]

Issue here with TZA2015DHS?