"_sm" Added to the end of Grid Variables in the Raw Tables

[KELSEYDOWLING7]

There is now "_sm" added to the ends of some grid variables in the raw data that shouldn't be there. This is causing a problem for IMS to properly make changes to the markdown, and is mentioned in issue #299 . This is not on all grid variables, only the ones shown below.

The participants who have data in these _sm variables started as early as 11/27: Mod2- started 11/27 or 11/28, finished 11/30 Mod3- started 11/27 and 11/28, finished 11/28-12/6 Mod4 started 11/26 or 11/27, finished between 11/29-12/6 BUM- started 11/27, finished 11/27-12/1 BU- started 11/27 or 11/28, finished 11/29-11/30 COVID- started 11/27 or 11/28 and finished 11/29 or 11/30

[joshid-ims]

How about Module 4 questions that have displayifs based on previous grid responses like CURWORKT11-13? These are not displaying as well when we tested.

[KELSEYDOWLING7]

@joshid-ims Yes, sorry for the delay I just updated the issue description above now that all the tables are re-flattened. But we only saw the one variable that had an issue in Mod4 and that was CURWORK2TA

[danielruss]

The _sm data is from quest. This is grid data. The grids switches from looking like a grid to a set of multiple choice questions on small factor devices. So whenever you see a grid there are 2 html elements one that looks like the grid (D_xxx_D_xxx) and one that looks like the individuals questions D_xxx_D_xxx_sm. They have the same data.

In the last month, I completely re-wrote the grids using CSS grids. You no longer need to worry about the "_sm" data, because CSS grids handles the visual differences between different size devices.

[KELSEYDOWLING7]

@danielruss @FrogGirl1123 Should we just consider the _sm bad data? Those that have CID_sm variable data don't have data in the regular CID variables.

Also @joshid-ims does this answer your question about the markdown/configurations issue or how to proceed?

[joshid-ims]

Without the resolution, we are not sure how to proceed. I do not think any code change is required here but I am not sure.

[danielruss]

What is the issue? I create the "_sm" elements from the id's of the question. For each "_sm", there should be a column without the "_sm" containing the same data. I am concerned that the "_sm" was not seen since day 1, but maybe the PWA filters it out. In general, ignore the _sm columns.

[joshid-ims]

The grids and questions which are displayed based on previous grid responses are not displaying. Like checking this:

equals(CURWORKT10A_1,1)

[danielruss]

@joshid-ims this markdown is not causing this issue. So there is no fix on your side.

[KELSEYDOWLING7]

For the data, there are the CID and CID_sm rows, but the non _sm equivalent of the variable doesn't have the data for the participants. For example, these Connect IDs (6957683423, 5302363888, 4845519625, 7430437721) that have a response for D_487532606_D_520755310_sm in prod don't have a response for D_487532606_D_520755310 in prod

Also the CID_sm rows only showed up for a few people that started their surveys between 11/26-11/28 and only for the variables above

[anthonypetersen]

The PWA has no logic for filtering out "_sm" from question IDs. If these are showing up now, it's coming from Quest.

[FrogGirl1123]

In trying to summarize, the _sm was perfectly normal for grids taken on a small device and their presence is not related to a markdown or Quest issue. The CSS grid fix results in _sm no longer being added to the variable name for small devices.

Of concern is that the _sm has not been observed in the data earlier and more frequently. If the use of the CSS grids was part of the Nov. 24 Quest update then we should have seen _sm before Nov. 24 and after Nov. 28/29 when the Nov. 24 update was rolled back. That said, missingness and other survey analyses @KELSEYDOWLING7 has done are not showing any issues with missing data that would be related to the grids not rendering or storing as expected on small devices. The only way to investigate further is if we had additional data on what device was used for the surveys. @anthonypetersen I don't suppose this is something available in the logs or with data dog?

@danielruss Have I missed or misunderstood anything? Were the CSS grids part of the Nov. 24 update?

[danielruss]

_sm was perfectly normal for grids taken on a small device and their presence is not related to a markdown or Quest issue

yes, But all devices should have _sm elements that are in sync with the elements without the _sm. The elements get synced automatically even though one is not visible.

Of concern is that the _sm has not been observed in the data earlier

yes, i need to look at the code. Many coders have stirred the quest stew. @anthonypetersen mentioned that the PWA does not filter out the _sm data. Maybe quest is doing it. If @KELSEYDOWLING7 does not see any missing data... we can ignore the _sm.

Were the CSS grids part of the Nov. 24 update?

Yes, grids were unable to have displayif or pipe text. I completely redesigned the grids to handle these case

Have I missed or misunderstood anything?

No, you nailed it pretty well

[KELSEYDOWLING7]

@danielruss @FrogGirl1123 I can confirm the data all look as expected and that I'm not seeing any patterns of missingness. @anthonypetersen Would you be able to adjust the variables in Firestore so we won't have to do variable merges on these in all future module versions?

[anthonypetersen]

Yup, we will just need all affected Connect IDs and the surveys / columns that are affected.

[KELSEYDOWLING7]

@anthonypetersen Thank you! All variables affected should be included in the screenshots above

[anthonypetersen]

I more meant it's needed for documentation in the production change log

[KELSEYDOWLING7]

@anthonypetersen Ok, do you need anything from me?

[anthonypetersen]

Yes, please pass along all the information (Connect IDs, Surveys, Columns) to Madhuri so it can get into the prod change log.

[KELSEYDOWLING7]

@anthonypetersen Forgot to let you know I sent that all to her so she's should be adding them to the change log if she hasn't already

[cusackjm]

Is there anything I need to test for this?

[KELSEYDOWLING7]

@anthonypetersen Hi Tony, it looks like all of the sm variables are still there. I sent the full list to Madhuri so they should have been in the prod change log a few weeks back. Do you mind double checking the fix?

[anthonypetersen]

@KELSEYDOWLING7 I spot checked some of the surveys in Firestore as well as BQ in the Connect datasets and the columns that were mentioned in the prod change log do not appear to have "_sm" in them anymore.

[KELSEYDOWLING7]

@anthonypetersen Ok thanks, it must be a flattening issue then. I'll let Jake know