survey integration union_all call is failing

[alee]

There is an issue with the analysis pipeline integrating round 1 pre surveys with round 2+ pre surveys. Not sure if this is a data issue where the round 1 pre survey has changed and the round 2+ pre surveys did not or vice versa.

@MarcoAJanssen

<error/rlang_error>
Error in `dplyr::union_all()` at R/survey.R:301:2:
! `x` and `y` are not compatible.
✖ Different number of columns: 111 vs 122.
---
Backtrace:
    ▆
 1. └─global tournament_load(tournament_dir, max_game_rounds)
 2.   └─global survey_tournament_load(...) at R/tournament.R:7:2
 3.     ├─dplyr::union_all(r1, r2plus) at R/survey.R:301:2
 4.     └─dplyr:::union_all.data.frame(r1, r2plus)

[MarcoAJanssen]

We did not change the surveys

[alee]

I dug a little deeper and it looks like this is an issue with dplyr version 1.1.0 changing the behavior of union_all to require that the two dataframes being unioned have the same shape. (changelog: https://github.com/tidyverse/dplyr/releases/tag/v1.1.0)

  dplyr::union_all(
      r1,
      r2plus
  )

Here I believe Calvin's code is is attempting to unify the survey results from round 1 (pre + post) and round 2+ but the union_all now fails. There's probably an easy fix for this but would be faster for some R experts like @kaclaborn to take a look at the code in survey.R and suggest patches / refactoring.

I did try changing the union_all to a full_join or row_bind naively thinking it would be equivalent but no dice.

  dplyr::full_join(
      r1,
      r2plus,
      by = survey_join_keys
  )

This causes a downstream failed assertion Error: nrow(game_tournament) not equal to nrow(survey_tournament) which I vaguely recall from previous runs.

Switching to an older version of dplyr is another option but currently challenging because of all of the other package dependencies that need to be included and RStudio's tendencies to install the latest version of things. Still that's pretty much the next avenue to explore using Docker images at https://hub.docker.com/r/rocker/tidyverse/tags

[alee]

root cause: https://github.com/virtualcommons/planning/issues/62

[sgfost]

The script will also get tripped up on duplicate survey records for a participant at this assertion:

  assertthat::assert_that(nrow(game_tournament) == nrow(survey_tournament))

survey_tournament will have additional rows for each dupe

In the 2023-10 dataset there's one instance of two post surveys recorded for one participant (inviteId = 1397) with the first being completed and the second only started several days later for some reason

Definitely something to handle within the script so I think we can just select the more complete (or newest if same completion %) one