Exporting data with filter fields for KoBo-specific question types results in missing data

[joshuaberetta]

Description

Community report: https://community.kobotoolbox.org/t/i-ve-got-3-projects-that-are-not-downloading-mix-matrix-ranking-and-some-even-text-questions/23682/11 Zulip discussion: https://chat.kobotoolbox.org/#narrow/stream/4-KoBo-Dev/topic/Custom.20exports/near/66892

KoBo-specific question types don't export submission data correctly when filtering fields in "Advanced options".

Steps to Reproduce

1. Use the a form with a KoBo question type (this one is from the support page on Matrix questions)

survey

type	name	label	required	kobo--matrix_list
begin_kobomatrix	M1	Items		assets
select_one yn	Q1	Q1. Which assets do you have at home	TRUE
integer	Q2	Q2. Number of assets	TRUE
end_kobomatrix

choices

list_name	name	label
assets	car	Car
assets	bike	Bike
assets	tv	TV
yn	yes	Yes
yn	no	No

settings

style
theme-grid no-text-transform

2. Submit some data
3. Export without any filter fields 👉 all data is exported
4. Export with any or all fields selected 👉 data from KoBo question type not exported. The following is is sent to the BE for filtering:

{
  ...
  "fields": [
    "M1/Q1",
    "M1/Q2",
  ],
  ...
}

If we look at the asset content.survey JSON, we see why this is the case:

[
  {
    "name": "M1",
    "type": "begin_kobomatrix",
    "$kuid": "Mex7rGeF6",
    "label": [
      "Items"
    ],
    "$autoname": "M1",
    "kobo--matrix_list": "assets"
  },
  {
    "name": "Q1",
    "type": "select_one",
    "$kuid": "Vwi0pHZ4A",
    "label": [
      "Q1. Which assets do you have at home"
    ],
    "required": true,
    "$autoname": "Q1",
    "select_from_list_name": "yn"
  },
  {
    "name": "Q2",
    "type": "integer",
    "$kuid": "MlNQNjPMM",
    "label": [
      "Q2. Number of assets"
    ],
    "required": true,
    "$autoname": "Q2"
  },
  {
    "type": "end_kobomatrix",
    "$kuid": "wNIn8EeUO"
  }
]

However, the kobomatrix group structure is transformed behind the scenes resulting in this survey structure:

survey

type	name	label	appearance	required
begin_group	M1_header		w5
note	M1_header_note	Items	w1	false
note	M1_header_Q1	Q1. Which assets do you have at home	w2	false
note	M1_header_Q2	Q2. Number of assets	w2	false
end_group
begin_group	M1_car		w5
note	M1_car_note	##### Car	w1	false
select_one yn	M1_car_Q1		w2 horizontal-compact	true
integer	M1_car_Q2		w2 no-label	true
end_group
begin_group	M1_bike		w5
note	M1_bike_note	##### Bike	w1	false
select_one yn	M1_bike_Q1		w2 horizontal-compact	true
integer	M1_bike_Q2		w2 no-label	true
end_group
begin_group	M1_tv		w5
note	M1_tv_note	##### TV	w1	false
select_one yn	M1_tv_Q1		w2 horizontal-compact	true
integer	M1_tv_Q2		w2 no-label	true
end_group

Therefore to correctly include all the matrix fields, the export data should look like this instead:

{
  ...
  "fields": [
    "M1_car/M1_car_Q1",
    "M1_car/M1_car_Q2",
    "M1_bike/M1_bike_Q1",
    "M1_bike/M1_bike_Q2",
    "M1_tv/M1_tv_Q1",
    "M1_tv/M1_tv_Q1"
  ],
  ...
}

Expected behavior

Export includes all selected data.

Actual behavior

Data is missing.

[jnm]

I think this may be another case of something that would be fixed by having get_fields_for_versions() in formpack return a list of fields to the front end, which is then presented in selection controls like the export filter.