NoneType object causing crash at survey creation

[Globulle]

Hi,

I am not able to create a survey object from my data (manually exported), despite I carefully set heading type to 'code' and reponse type to 'short'.

My code is:

from limepy.wrangle import Survey, Question
import pandas as pd

df = pd.read_csv('results-survey618718_code.csv', sep=',')
with open('limesurvey_survey_618718.lss') as f:
    my_structure = f.read()

my_survey = Survey(df, my_structure)

And I get:

  File "/home/xxxxxxx/anaconda3/lib/python3.9/site-packages/limepy/wrangle.py", line 262, in create_readable_df
    colname = colname.replace('\n', ' ')

AttributeError: 'NoneType' object has no attribute 'replace'

Is it a bug? Or am I misusing the tool? Thanks for your help.

Here is the CSV file I use:

"id","submitdate","lastpage","startlanguage","startdate","datestamp","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","matiere_SQ001","matiere_SQ002","matiere_SQ003","matiere_SQ004","matiere_SQ005","matiere_SQ006","suggestion","general_SQ001","general_SQ002","general_SQ003","general_SQ004","general_SQ005","ptspositifs_SQ001_SQ001","ptspositifs_SQ002_SQ001","ptspositifs_SQ003_SQ001","ptsnegatifs_SQ001_SQ001","ptsnegatifs_SQ002_SQ001","ptsnegatifs_SQ003_SQ001","commentaire","poisson"
"10","2022-04-01 07:55:43","17","fr","2022-04-01 07:38:41","2022-04-01 07:55:43","A2","A3","A1","A2","A2","A2","+ d'applications en cinématiques ","A2","A1","A1","A1","A1","A1","DS parfois trop long","A2","A2","A2","A2","A2","A2","","A2","A1","A1","A1","A1","A1","","A2","A1","A1","A1","A1","A1","","A2","A2","A1","A1","A1","A1","","A1","A1","A1","A1","A1","A1","","A2","A1","A2","A2","A2","A2","","A1","A1","A1","A1","A1","A1","","A1","A1","A2","A1","A1","A2","","A2","A2","A1","A1","A1","A1","","A2","A2","A1","A1","A2","A1","","A1","A2","A1","A1","A1","A2","","A2","A1","A1","A1","A1","A1","","A2","A3","A4","A4","A2","A2","Rendre les cours plus concrets","A1","A2","A2","A1","A1","Professionalisation ","Bonne ambiance ","Échelle humaine","","","","","Bonne idée ! "
"11","2022-04-01 07:49:16","17","fr","2022-04-01 07:38:42","2022-04-01 07:49:16","A2","A2","A4","A4","A4","A4","Problème de compréhension  avec le prof ","A2","A2","A2","A2","A2","A2","","A1","A1","A1","A1","A1","A1","","A2","A1","A2","A2","A2","A2","","A2","A2","A2","A2","A2","A2","","A1","A1","A1","A1","A1","A1","","A2","A2","A2","A2","A2","A2","","A2","A2","A2","A2","A2","A2","","A2","A2","A2","A2","A3","A2","","A2","A2","A3","A3","A2","A2","","A2","A2","A2","A2","A2","A2","","A1","A1","A1","A1","A1","A1","","A2","A2","A3","A2","A2","A2","","A3","A3","A3","A2","A2","A2","","A2","A2","A2","A2","A2","A2","","A1","A1","A1","A1","A3","Le niveau ","L'ambiance ","Soirée ","Le contenu","Certains profs ","Horaire ","","C'est une bonne idée "

[DIRKMJK]

Hi, thanks for your feedback. The code looks ok, It’s not immediately clear to me what might have gone wrong. Perhaps if you could share the lss file I could try to figure out what caused the bug?

[Globulle]

Hi, here is my lss file : https://drop.infini.fr/r/4CJ389qS1C#RJ2n/6rSFe6fECUkoljtYWTwIDVzDtQZwkruPqybTxw=

Thanks for your help.

[DIRKMJK]

Thanks; it appears the problem was that missing values in the lss wasn’t handled correctly. I’ve updated the code; hopefully this solves the problem

[Globulle]

It works! Thanks!

[Globulle]

I am now able to create a Survey object, but I encounter some other problems:

• my_survey.respondent(10) raises an error:

  
  File "/home/xxxxxx/anaconda3/lib/python3.9/site-packages/limepy/wrangle.py", line 326, in respondent
  respondent += question['question'] + '\n'

TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

I wonder if this is not linked to previous bug.

---

* `my_question = Question(my_survey, 193659)` also produces an error:
`KeyError: 'matiere[SQ001]'`
Maybe this is due to the fact that this question was a table?

[DIRKMJK]

Hi, it appears to have something to do with how Limesurvey generates column names from question and subquestion codes. For example, your csv has a column named matiere_SQ001; if I export the data from my own installation of Limesurvey, I get matiere[SQ001]. I’m not sure whether this is because your version is older or newer; which version of Limesurvey are you using?

[Globulle]

which version of Limesurvey are you using?

Version 2.65.0+170502 (a bit old, I'm afraid :grimacing: )

[DIRKMJK]

Thanks; I guess that might explain it :) Do you think it would be possible to upgrade? I was able to import your lss file into 5.3 so you should still be able to use the survey you created

[Globulle]

EDIT : I'm able to affect this column name with an option: if I put "Use Expression Manager code:" to "On" I get matiere_SQ001. Otherwise I obtain matiere[SQ001] like you.

What is the best setting?

Do you think it would be possible to upgrade?

Unfortunately I use an institutional install.

[DIRKMJK]

Ah, I see, I never paid attention to that setting (in my version it's called Use ExpressionScript code). Yes could you try exporting with that setting set to Off and see if that works?

[DIRKMJK]

There’s another thing that might create problems: it appears your survey has questions with identical question codes, in which case pandas will add a suffix to differentiate them, for example matiere_SQ001.1

[Globulle]

Here are the settings I have used:

And here are some lines of the resulting csv:

"id","submitdate","lastpage","startlanguage","startdate","datestamp","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","matiere[SQ001]","matiere[SQ002]","matiere[SQ003]","matiere[SQ004]","matiere[SQ005]","matiere[SQ006]","suggestion","general[SQ001]","general[SQ002]","general[SQ003]","general[SQ004]","general[SQ005]","ptspositifs[SQ001_SQ001]","ptspositifs[SQ002_SQ001]","ptspositifs[SQ003_SQ001]","ptsnegatifs[SQ001_SQ001]","ptsnegatifs[SQ002_SQ001]","ptsnegatifs[SQ003_SQ001]","commentaire","poisson"
"10","2022-04-01 07:55:43","17","fr","2022-04-01 07:38:41","2022-04-01 07:55:43","Plutôt d'accord","Plutôt pas d'accord","Tout à fait d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","+ d'applications en cinématiques ","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","DS parfois trop long","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Plutôt d'accord","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Plutôt d'accord","Tout à fait d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Tout à fait d'accord","Tout à fait d'accord","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Plutôt d'accord","","Plutôt d'accord","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Plutôt d'accord","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Plutôt d'accord","Tout à fait d'accord","","Tout à fait d'accord","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Plutôt d'accord","","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Plutôt d'accord","Plutôt pas d'accord","Pas du tout d'accord","Pas du tout d'accord","Plutôt d'accord","Plutôt d'accord","Rendre les cours plus concrets","Tout à fait d'accord","Plutôt d'accord","Plutôt d'accord","Tout à fait d'accord","Tout à fait d'accord","Professionalisation ","Bonne ambiance ","Échelle humaine","","","","","Bonne idée ! "
"11","2022-04-01 07:49:16","17","fr","2022-04-01 07:38:42","2022-04-01 07:49:16","Plutôt d'accord","Plutôt d'accord","Pas du tout d'accord","Pas du tout d'accord","Pas du tout d'accord","Pas du tout d'accord","Problème de compréhension  avec le prof ","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Plutôt d'accord","Tout à fait d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt pas d'accord","Plutôt d'accord","","Plutôt d'accord","Plutôt d'accord","Plutôt pas d'accord","Plutôt pas d'accord","Plutôt d'accord","Plutôt d'accord","","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","","Plutôt d'accord","Plutôt d'accord","Plutôt pas d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Plutôt pas d'accord","Plutôt pas d'accord","Plutôt pas d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","Plutôt d'accord","","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Tout à fait d'accord","Plutôt pas d'accord","Le niveau ","L'ambiance ","Soirée ","Le contenu","Certains profs ","Horaire ","","C'est une bonne idée  "

As in my first post, the command my_survey = Survey(df, my_structure) raises the following error:

  File "/home/xxxxxx/anaconda3/lib/python3.9/site-packages/limepy/wrangle.py", line 264, in create_readable_df
    colname = colname.replace('\n', ' ')

AttributeError: 'NoneType' object has no attribute 'replace'

---

it appears your survey has questions with identical question codes

yes I noticed that... in Limesurvey they are in separated groups of questions but the export seems to ignore it... That's strange.

[Globulle]

But if you're not able to reproduce the bug, I will assume this is due to my old version of Limesurvey and I will use another strategy, nevermind. :wink:

[DIRKMJK]

I do get the same error message using your csv... It appears to have something to do with identical question codes. I have made a small change, it appears to work now, though I’d recommend checking if you get correct results. Or better, changing question codes to make them unique :)

[Globulle]

It seems to work, so far. Thanks!

[Globulle]

Sorry to disturb you again: I think there still is a bug (maybe I should open another issue, I don't know if it's related to previous ones) :

When I ask my_question = Question(my_survey, 193659), the array corresponding to my_question.summary.values contains only zeroes. I observe the same with other similar questions.

[DIRKMJK]

Hi, the problem seems to be in how the answers are exported. Do you have an option to export answer codes (or something similar)? If so, could you try if that works?

[Globulle]

That's it! Thank you! Sorry, I see it was written in the Readme, though.

[DIRKMJK]

Thanks, I’ll fix it in the Readme!

[DIRKMJK]

Incidentally, I’m still a bit concerned that duplicate question codes may lead to problems further down the line, since Limesurvey creates column names from (sub-) question codes, and duplicate column names will create problems when looking up the correct answers to a question

[Globulle]

OK you're right: all the questions get the same values as the first one. I will make the question code unique.

[DIRKMJK]

Ok, hope that works!

[Globulle]

It's OK now. Fortunately Limesurvey allows to edit question codes even if the survey has already been activated. Thank you very much for this great library and for your help!

[DIRKMJK]

Good to hear, & thanks for the feedback!