mtwestra
commented
11 years ago We might want to delete some data first, for example, for example, the questionAnswerSummary objects for non-option questions
Closed caetie closed 10 years ago
mtwestra
commented
11 years ago We might want to delete some data first, for example, for example, the questionAnswerSummary objects for non-option questions
iperdomo
commented
11 years ago After a quick testing in the upgraded instance:
OPTION questions don't show up in the dropdown (needs investigation)
caetie
commented
11 years ago Initial assessment 13 august 2013, cmo
// testing with 2012 survey group
Results: survey group view
Type = Water Point in 1.0 displays blank in the 1.6 interface, because we replaced Water Point with Point. Many surveys whose name have Household or PI (Public Institution) in them were still classified as Water Point in their 1.0 dashboard. These will also need to be reset manually, although maybe we can batch reset them.// testing with 2012 Surveys / Honduras Water Point 2012 / 7510009
Results: Type dropdown
Results: Language dropdown (master language)
Uncaught TypeError: Cannot read property 'name' of undefinedtype and language
GET http://wfp-161.appspot.com/rest/translations?surveyId=7504497 500 (Internal Server Error)
for 2012 Surveys / Honduras Water Point 2012
GET http://wfp-161.appspot.com/rest/translations?surveyId=7510009 500 (Internal Server Error)Results: Question detail screens
// testing with akvo test apk / test upgrade / 53115
Results:
Results: all as expected
// testing with akvo test apk / test upgrade / 53115
Results: all as expected
Needs:
Results: all as expected
Results: all as expected
// testing with 2012 Surveys / Honduras Water Point 2012 / 7510009
Results:
option questions don't load in the dropdown, but
- Who makes the applicacion of chlorine - Question displays and chart builds - SEE SCREENSHOT
- Is the water system shared with other communities - Question displays and chart builds - SEE SCREENSHOT
Raw data report: runs as expected
Comprehensive report (16 MB): runs as expected
Raw text file: runs as expected
Survey form: runs as expected
Results:
Needs:
Results:
Result: appears as expected
caetie
commented
11 years ago // testing with 2012 Surveys / Honduras Water Point 2012 / 7510009
I also tested the dependencies in an existing survey in detail by assigning it to my device and checking it against the version in their 1.0 dashboard (using a web authorization as a "preview"), and every single question in that survey behaved as expected.
iperdomo
commented
11 years ago
- Of the surveys on the 1.6.1 dash, some have carried over the language setting to English and others have not. I cannot tell why some have and some haven't.
- If you look in the survey group 2012 Surveys for instance in 1.6.1, some are set to English and others are not. If you check those same surveys that are not set to English in 1.6.1, they are set to English in the 1.0 dashboard.
Related to this topic, the attribute in the datastore is null and is treated as as English. The data transfer part was done successfully.
Example:
It's shown as English in the classic UI, but it has null in the datastore.
And the same value was transferred.
There are 49 Survey entities that have the defaultLanguageCode set to null
iperdomo
commented
11 years ago I've just reset the language to en in the upgraded instance for those Survey that have the defaultLanguageCode to null.
More info at: https://gist.github.com/iperdomo/6279918#file-remoteapiclient-java-L52
iperdomo
commented
11 years ago The same happens with the pointType attribute, If is set to null the UI sets a default to Water Point
And null in the datastore
caetie
commented
11 years ago type = water point to type = pointiperdomo
commented
11 years ago question_groups that is not necessary[]
iperdomo
commented
11 years ago Added a new issue #345 related to Chart Builder because is a generic problem and not directly related to the upgrade
iperdomo
commented
11 years ago SurveyedLocale for the placemark service, and there is a way of re-creating this entities: http://instance.appspot.com/app_worker/surveyalservlet?action=rerun&surveyId=123mergeNearbyLocales in appengine-web.xml needs to be set to falsecaetie
commented
11 years ago Another note about the maps - we currently limit point loads on the map to 500 when the user loads the map page. WFP have many more than this, so will either need to let them know or adjust the limit for their page higher, perhaps to 2000 like Cardno.
iperdomo
commented
11 years ago Thanks to the work of @mtwestra now we have a way of translating questions per question group. That allow us work with large surveys.
iperdomo
commented
11 years ago I tested pulling all points in Malawi for 2012 surveys, and got 7783 points. Constructing an internal list with almost 8K Placemark objects result in ~120MB of JavaScript heap size. Depending on the client machine could be handled or it can make the browser crash.
The reliable solution is Maps clustering, that it's potentially targeting v1.6.5 (no commitment)
KeriKugler
commented
11 years ago Hi Mark, Ivan and Others,
I was asked to put my comments, questions and issues here while testing our version of 1.6. I am still going through things but this is what I have come up with so far.
Questions I have: 1) It looks like none of our surveys have been published – do I need to go through and publish everything? 2) Previewing the Survey – it looks like you can only preview the survey in the master language, is that correct? Is there any way to preview the survey with translations?
Significant Issues: (These are things I have come across that are significant enough that for us we don't think we can deploy 1.6 to our filed until they are addressed because they would significantly disrupt operations)
1) Translations in Existing Surveys -- Our translations don’t seem to have been copied over fully into the new dashboard – they seem to be partially there probably on the back end but I can’t see them on the dashboard.
This is the behavior I encounter. I choose a survey that I know has been translated into Spanish (example: Bolivia Household 2013 in the 2013 Survey Group) I wait for the survey to fully load then choose Edit Translations. No translations are visible (but they should be this survey has been translated into Spanish) however, it says at the top that there are Existing Spanish translations, if I click on that nothing happens, if I try to add new Spanish translations by choosing Spanish in the Add New Translation box the program gives me an error message that: "Translation already present: You already have a translation in the language you have selected" but I can’t see or edit the Spanish anywhere.
2) There is no drop down at the top of the home page to change the language of the Dashboard to Spanish or French
3) Copying Surveys – I tried twice to copy a very large survey (2013 Survey Group, India Water Point 2013 copied first into Testing Copy India Water Point 2013 (copy), then into Testing India Water Point 2013 copy 2) the surveys appeared to copy and the saving bar disappeared but when I clicked on the copies they had only part of the surveys in them. I tried copying a smaller survey (Peru Household 2013 copied to Testing Peru Household 2013) and they survey copied over but the translations did not.
4) Bulk Import of Surveys Failed – I did a bulk import of data into the Malawi 2013 surveys that I also imported into our 1.0 instance. On the 1.0 instance that resulted in a raw data report for the Malawi Water Point 2013 survey with 870 rows, I did the same bulk import on the 1.6 instance and the raw data report for the Malawi Water Point survey only has 583 rows. I waited until the next day and ran the report again on the 1.6 instance to see if the instance just needed time to process the data but there were still only 583 rows.
5) Inspect Data Tab – Can’t search for data by instance ID anymore (this is something that we could previously do) and is necessary to delete test data/duplicates out of data – field staff identify test data and find it by instance ID to delete – with sometimes thousands of records in a survey it is no plausible to scroll through the whole list and find each individual survey in the inspect data tab and delete them, there needs to be a way to find and delete individual records.
Medium Issues (We can implement around these and can live with them for a short time but they are important): 1) Reports are not Bi-lingual – It used to be reports could be run in the master language and the translations, now they can only be run in the master language. It doesn’t make much sense to have surveys that are multi-lingual but not reports that are as well – we need this functionality back
2) Comprehensive Report won’t run on some of our bigger surveys (India Waterpoint 2012, India PI 2012, Rwanda Household 2012)
3) Maps – Data in pop up bubble seems to be in a totally random order –not the order of the survey questions or any other discernible order – just a jumbled data dump
Minor Issues (We would like these fixed but they won’t hold anything up): 1) When you click on the Info button next to Master Language it displays the help tip for Version Number not the help tip for Master Language
New things I especially like: 1) Can enter new translations all at once (not question by question) 2) No confusing double scroll bars 3) Previewing a survey without having to publish it and deploying it to a phone to test 4) Running a report without a java client 5) Bulk Upload – can drag and drop in multiple files instead of doing them one at a time 6) Messages to tell you when things have uploaded, processed, when there are errors --- less guessing about what is going on 7) Lat/Long/Elevation broken out into separate columns in reports
mtwestra
commented
11 years ago Hi Keri,
thanks for your comments! I am a little late in responding - Iñigo, one of our new developers working on FLOW, is visiting for the first time in Amsterdam, and I have been talking with him all day.
My comments are below,
cheers, mark
On Sep 25, 2013, at 14:43, Keri notifications@github.com wrote:
Hi Mark, Ivan and Others,
I was asked to put my comments, questions and issues here while testing our version of 1.6. I am still going through things but this is what I have come up with so far.
Questions I have: 1) It looks like none of our surveys have been published – do I need to go through and publish everything?
The testing instance wfp-161 is just that, a testing instance. We copied over all the data, including surveys and survey data. Therefore, all the surveys have not been published on that instance. The plan is to test everything on this instance, and if we are satisfied, bring your current watermapmonitordev instance to the FLOW 1.6 version. The test version of wfp-161 will then be discarded. So - feel free to experiment there, it will not affect the WFP instance. 2) Previewing the Survey – it looks like you can only preview the survey in the master language, is that correct? Is there any way to preview the survey with translations?
Yes, you are right. We viewed the survey preview mainly as a tool to check if the structure of the survey is correct. I understand that you would like to view the translations in the preview as well? This would be a new feature, that we could place on our roadmap if the use case is clear. Significant Issues: (These are things I have come across that are significant enough that for us we don't think we can deploy 1.6 to our filed until they are addressed because they would significantly disrupt operations) 1) Translations in Existing Surveys -- Our translations don’t seem to have been copied over fully into the new dashboard – they seem to be partially there probably on the back end but I can’t see them on the dashboard.
That is interesting... We will investigate this with the example of the Bolivia household example you quote below. It is possible that there is a suble difference in how the translations are defined in FLOW 1.0 and 1.6, for example how the language code is stored. We'll have a look and let you know. a. This is the behavior I encounter. I choose a survey that I know has been translated into Spanish (example: Bolivia Household 2013 in the 2013 Survey Group) I wait for the survey to fully load then choose Edit Translations. No translations are visible (but they should be this survey has been translated into Spanish) however, it says at the top that there are Existing Spanish translations, if I click on that nothing happens, if I try to add new Spanish translations by choosing Spanish in the Add New Translation box the program gives me an error message that: Translation already present: You already have a translation in the language you have selected. But I can’t see or edit the Spanish anywhere
— Reply to this email directly or view it on GitHub.
KeriKugler
commented
11 years ago I found one more thing:
Significant Issues (We can’t implement in the field till these are fixed): 1) Chart Builder Numbers Are Not Correct Sometimes – I tested this on a number of surveys but the number of answers in the center of the donut chart on chart builder is not correct in many cases and doesn’t match up with the actual number of survey responses submitted. In all cases for this issue I tested using the question “Is this an improved water source/system?” because it is a standard question on all of our core surveys and it is a mandatory question that is not multi-option so there will not be more responses than there are record instances in any particular survey. For example I pulled the raw data report for Bolivia Water Point 2013 survey, there were 175 surveys submitted, the chart builder says there were 329 responses to that question. This was also the case on the survey for Malawi Household 2013 only in that case the Chart Builder reported less responses than there actually were (125 responses in the raw data report, chart builder reported 82) and in that case the percentages in chart builder were also off by a couple of percentages. The problem was more significant for Guatemala PI 2012 survey the raw data report has 193 responses but chart build says there are no responses. Chart Build said there was no data for many of the surveys in the 2012 survey group (Bolivia Water Point 2012, Guatemala Water Point 2012, Honduras Water Point 2012, Honduras PI 2012, Malawi PI 2012 …).
mtwestra
commented
11 years ago Hi Keri,
We will also have a look at this, but the likely cause is that the the aggregation data we copied over from the existing WFP instance is incorrect. In fact, we have made significant changes to the quality of the aggregates, and we think that the functionality is reliable now. The existing aggregation data that you now see in the chart builder was created by the FLOW 1.0 system, and copied over.
The idea is that when we move to FLOW 1.6 on the original WFP instance, we rebuilld all the question statistics. To test this out, I will manually rebuild the statistics for the survey you quote, and compare the numbers. I'll let you know the results. I need to do it tomorrow though - I am in the train going home now.
cheers, mark
On Sep 25, 2013, at 20:58, Keri notifications@github.com wrote:
I found one more thing:
Significant Issues (We can’t implement in the field till these are fixed): 1) Chart Builder Numbers Are Not Correct Sometimes – I tested this on a number of surveys but the number of answers in the center of the donut chart on chart builder is not correct in many cases and doesn’t match up with the actual number of survey responses submitted. In all cases for this issue I tested using the question “Is this an improved water source/system?” because it is a standard question on all of our core surveys and it is a mandatory question that is not multi-option so there will not be more responses than there are record instances in any particular survey. For example I pulled the raw data report for Bolivia Water Point 2013 survey, there were 175 surveys submitted, the chart builder says there were 329 responses to that question. This was also the case on the survey for Malawi Household 2013 only in that case the Chart Builder reported less responses than there actually were (125 responses in the raw data report, chart builder reported 82) and in that case the percentages in chart builder were also off by a couple of percentages. The problem was more significant for Guatemala PI 2012 survey the raw data report has 193 responses but chart build says there are no responses. Chart Build said there was no data for many of the surveys in the 2012 survey group (Bolivia Water Point 2012, Guatemala Water Point 2012, Honduras Water Point 2012, Honduras PI 2012, Malawi PI 2012 …).
— Reply to this email directly or view it on GitHub.
mtwestra
commented
11 years ago 1) It looks like none of our surveys have been published – do I need to go through and publish everything?
MTW: We have just copied over the existing data from the 1.0 WFP instance. If a survey appears as not published in the 1.6 Dashboard, it was not published in the 1.0, and you will have to republish the ones you need. Of course, it could be the case that a change was made after it was published at some time, which would mean that the status might be 'not-published', while an older version of that survey is being used by devices.
2) Previewing the Survey – it looks like you can only preview the survey in the master language, is that > correct? Is there any way to preview the survey with translations?
MTW: Yes, you are right. We viewed the survey preview mainly as a tool to check if the structure of the survey is correct. I understand that you would like to view the translations in the preview as well? This would be a new feature, that we could place on our roadmap if the use case is clear.
Significant Issues: (These are things I have come across that are significant enough that for us we don't think we can deploy 1.6 to our filed until they are addressed because they would significantly disrupt operations)
1) Translations in Existing Surveys -- Our translations don’t seem to have been copied over fully into the new dashboard – they seem to be partially there probably on the back end but I can’t see them on the dashboard.
This is the behavior I encounter. I choose a survey that I know has been translated into Spanish (example: Bolivia Household 2013 in the 2013 Survey Group) I wait for the survey to fully load then choose Edit Translations. No translations are visible (but they should be this survey has been translated into Spanish) however, it says at the top that there are Existing Spanish translations, if I click on that nothing happens, if I try to add new Spanish translations by choosing Spanish in the Add New Translation box the program gives me an error message that: "Translation already present: You already have a translation in the language you have selected" but I can’t see or edit the Spanish anywhere.
MTW: we have checked the bolivia example, and all the translations are there, but the loading is very slow. Please try this: first go to the bolivia survey, and make sure the loading is complete. Because it has to load all the questions, this might take up to 40-50 seconds. Then click translations, and give it time again, to load the translations. This will again take some 40-50 seconds. After that, on my computer all the translations show up as expected.
The reason for the slow speed is the large number of items that need to be assembled, and the relative unsuitability of Google App Engine to do this. That is why we are currently planning a refactor of the system on our own hardware, which should achieve much higher speeds.
2) There is no drop down at the top of the home page to change the language of the Dashboard to Spanish or French
MTW: we will get back that functionality when we translate all the strings. We use a translation functionality called 'transifex', which we will share login details for. In that interface, you can simply provide spanish translations for the strings in the app. Could you please create a login here: https://www.transifex.com/
3) Copying Surveys – I tried twice to copy a very large survey (2013 Survey Group, India Water Point 2013 copied first into Testing Copy India Water Point 2013 (copy), then into Testing India Water Point 2013 copy 2) the surveys appeared to copy and the saving bar disappeared but when I clicked on the copies they had only part of the surveys in them. I tried copying a smaller survey (Peru Household 2013 copied to Testing Peru Household 2013) and they survey copied over but the translations did not.
MTW: This is a known issue. The problem we have seen is that it is a timeout: the copying is not completed within one minute. The solution that we have planned it so let the copying be done in a backend process on GAE, which does not have the 60 second restriction. However, to do this we need to make some changes to the dashboard user interface, for example to block editing of the survey while it is being copied. This will take some time. Please let us know if this really is a show-stopper right now, or that you can work around it for a week or two, after which we expect to have solved this.
4) Bulk Import of Surveys Failed – I did a bulk import of data into the Malawi 2013 surveys that I also imported into our 1.0 instance. On the 1.0 instance that resulted in a raw data report for the Malawi Water Point 2013 survey with 870 rows, I did the same bulk import on the 1.6 instance and the raw data report for the Malawi Water Point survey only has 583 rows. I waited until the next day and ran the report again on the 1.6 instance to see if the instance just needed time to process the data but there were still only 583 rows.
MTW: This functionality has changed, because we needed to get rid of Java on the client computer. Where before you were able to select a folder, and the dashboard would read all the files inside, now you need to create a zip file of all the data you want to load (a zip of the surveyal folder), and upload that zip file. The reason we need to do this is that browsers can only handle only the selection of a single file - the folder-upload was handled by Java, which we now don't use any more. Perhaps you could provide us with a bit more info of what exactly you did to bulk upload the data. We can also walk you through the new way of doing that.
5) Inspect Data Tab – Can’t search for data by instance ID anymore (this is something that we could previously do) and is necessary to delete test data/duplicates out of data – field staff identify test data and find it by instance ID to delete – with sometimes thousands of records in a survey it is no plausible to scroll through the whole list and find each individual survey in the inspect data tab and delete them, there needs to be a way to find and delete individual records.
MTW: Yes, that is a good point. We will bring back this functionality at some point, although we would like to tackle the other issues you have identified first, as those seem more urgent. One way of working around this might be to not do test data in a survey that is also used for real data collection, but make a copy of the survey, and use that for testing? In that way you can find the test data back in an easy way.
Medium Issues (We can implement around these and can live with them for a short time but they are important): 1) Reports are not Bi-lingual – It used to be reports could be run in the master language and the translations, now they can only be run in the master language. It doesn’t make much sense to have surveys that are multi-lingual but not reports that are as well – we need this functionality back
MTW: As I understand it, only the headings of the colums were translated, right? At the moment these values are only available in spanish. What we can do is put a dropdown in the export popup, letting the user choose between english and spanish for the headings. Would that be useful?
2) Comprehensive Report won’t run on some of our bigger surveys (India Waterpoint 2012, India PI 2012, Rwanda Household 2012)
MTW: Actually, we have tried all the reports, and the reports are generated. It just takes a long time. When I tried to run the India PI 2012 report, it returned immediately, probaby because you had started it when you tried it. The system now caches reports, so that they are not created more then once if no additional data has come in. Please try again on these files, and let us know the result. I saw the India PI 2012 is 66 MB, so it will have taken a while.
Please remember that the reports are not created by a Java process on your local computer any more, but by an Akvo server. The only thing you will see is a loading indicator. The large reports can take quite some time. Because of the caching, you can always just start the export, and try again later. The generation does not stop, and will only be done once, if no additional data comes in in the mean time.
You will notice that when you select a survey in the reports tab, the loading indicator is present for a while. This is because at the moment, when we select a survey, the system automatically downloads all questions. This is not necessary here, and this behaviour will be changed in a few weeks time.
3) Maps – Data in pop up bubble seems to be in a totally random order –not the order of the survey questions or any other discernible order – just a jumbled data dump
MTW: This is a know issue for which we have a github issue. This will be fixed on a short timescale.
Minor Issues (We would like these fixed but they won’t hold anything up): 1) When you click on the Info button next to Master Language it displays the help tip for Version Number not the help tip for Master Language
MTW: this is a know issue, and we will fix it on a short timescale.
1) Chart Builder Numbers Are Not Correct Sometimes – I tested this on a number of surveys but the number of answers in the center of the donut chart on chart builder is not correct in many cases and doesn’t match up with the actual number of survey responses submitted. In all cases for this issue I tested using the question “Is this an improved water source/system?” because it is a standard question on all of our core surveys and it is a mandatory question that is not multi-option so there will not be more responses than there are record instances in any particular survey. For example I pulled the raw data report for Bolivia Water Point 2013 survey, there were 175 surveys submitted, the chart builder says there were 329 responses to that question. This was also the case on the survey for Malawi Household 2013 only in that case the Chart Builder reported less responses than there actually were (125 responses in the raw data report, chart builder reported 82) and in that case the percentages in chart builder were also off by a couple of percentages. The problem was more significant for Guatemala PI 2012 survey the raw data report has 193 responses but chart build says there are no responses. Chart Build said there was no data for many of the surveys in the 2012 survey group (Bolivia Water Point 2012, Guatemala Water Point 2012, Honduras Water Point 2012, Honduras PI 2012, Malawi PI 2012 …).
MTW: We are investigating this. We made quite a number of changes to make the counting more accurate, and in the running instances this works well. Probably it is caused by the fact that we have copied the data, and need to run some aggregaters again. by hand. We will investigate further and let you know.
KeriKugler
commented
11 years ago Hi Mark,
Thanks for getting back to me on all of those. In general for time line, we can live with anything for a week or two so that is fine, but living with things for months would be harder as these are all functions we use a lot. Just a few comments on your comments below:
2) Previewing the Survey – it looks like you can only preview the survey in the master language, is that > correct? Is there any way to preview the survey with translations?MTW: Yes, you are right. We viewed the survey preview mainly as a tool to check if the structure of the survey is correct. I understand that you would like to view the translations in the preview as well? This would be a new feature, that we could place on our roadmap if the use case is clear.
KK: That makes sense. I think it would be a useful feature down the line to be able to preview with the translations at least for us since we often have core surveys that are in English the translated into Spanish, Kinyerwanada and other languages, since some of our staff doesn't speak any English I am sure they would like to be able to preview with the translations but it is not critical.
1) Translations in Existing Surveys -- Our translations don’t seem to have been copied over fully into the new dashboard – they seem to be partially there probably on the back end but I can’t see them on the dashboard.
This is the behavior I encounter. I choose a survey that I know has been translated into Spanish (example: Bolivia Household 2013 in the 2013 Survey Group) I wait for the survey to fully load then choose Edit Translations. No translations are visible (but they should be this survey has been translated into Spanish) however, it says at the top that there are Existing Spanish translations, if I click on that nothing happens, if I try to add new Spanish translations by choosing Spanish in the Add New Translation box the program gives me an error message that: "Translation already present: You already have a translation in the language you have selected" but I can’t see or edit the Spanish anywhere.MTW: we have checked the bolivia example, and all the translations are there, but the loading is very slow. Please try this: first go to the bolivia survey, and make sure the loading is complete. Because it has to load all the questions, this might take up to 40-50 seconds. Then click translations, and give it time again, to load the translations. This will again take some 40-50 seconds. After that, on my computer all the translations show up as expected.
The reason for the slow speed is the large number of items that need to be assembled, and the relative unsuitability of Google App Engine to do this. That is why we are currently planning a refactor of the system on our own hardware, which should achieve much higher speeds.
KK: You are right these are showing up if I wait long enough, though for me it can take quite a while. Will this take longer or potentially time out on poor internet connections (the worst I think we have is one office with all 12 or so staff sharing a 1mb internet connection)?
2) There is no drop down at the top of the home page to change the language of the Dashboard to Spanish or FrenchMTW: we will get back that functionality when we translate all the strings. We use a translation functionality called 'transifex', which we will share login details for. In that interface, you can simply provide spanish translations for the strings in the app. Could you please create a login here: https://www.transifex.com/
KK: I am registered and so is Mark Duey and we will get Claudia and Elias in Honduras to register (they are the ones who will actually do the translating).
3) Copying Surveys – I tried twice to copy a very large survey (2013 Survey Group, India Water Point 2013 copied first into Testing Copy India Water Point 2013 (copy), then into Testing India Water Point 2013 copy 2) the surveys appeared to copy and the saving bar disappeared but when I clicked on the copies they had only part of the surveys in them. I tried copying a smaller survey (Peru Household 2013 copied to Testing Peru Household 2013) and they survey copied over but the translations did not.MTW: This is a known issue. The problem we have seen is that it is a timeout: the copying is not completed within one minute. The solution that we have planned it so let the copying be done in a backend process on GAE, which does not have the 60 second restriction. However, to do this we need to make some changes to the dashboard user interface, for example to block editing of the survey while it is being copied. This will take some time. Please let us know if this really is a show-stopper right now, or that you can work around it for a week or two, after which we expect to have solved this.
KK: Thanks a week or two is fine on fixing this, by November I will need to copy all of our surveys to prepare for 2014 but until then I can wait on the really big ones. However, will this also fix the issue of the translations not copying over when a survey does copy?
4) Bulk Import of Surveys Failed – I did a bulk import of data into the Malawi 2013 surveys that I also imported into our 1.0 instance. On the 1.0 instance that resulted in a raw data report for the Malawi Water Point 2013 survey with 870 rows, I did the same bulk import on the 1.6 instance and the raw data report for the Malawi Water Point survey only has 583 rows. I waited until the next day and ran the report again on the 1.6 instance to see if the instance just needed time to process the data but there were still only 583 rows.MTW: This functionality has changed, because we needed to get rid of Java on the client computer. Where before you were able to select a folder, and the dashboard would read all the files inside, now you need to create a zip file of all the data you want to load (a zip of the surveyal folder), and upload that zip file. The reason we need to do this is that browsers can only handle only the selection of a single file - the folder-upload was handled by Java, which we now don't use any more. Perhaps you could provide us with a bit more info of what exactly you did to bulk upload the data. We can also walk you through the new way of doing that.
KK: If the process has changed I was doing this wrong, I did not zip the surveyal folder. If that is all I need to do I Can zip the files and retry. Do I need to do them one at a time (one phone at a time) or can I zip all the files and upload them all at once?
5) Inspect Data Tab – Can’t search for data by instance ID anymore (this is something that we could previously do) and is necessary to delete test data/duplicates out of data – field staff identify test data and find it by instance ID to delete – with sometimes thousands of records in a survey it is no plausible to scroll through the whole list and find each individual survey in the inspect data tab and delete them, there needs to be a way to find and delete individual records.MTW: Yes, that is a good point. We will bring back this functionality at some point, although we would like to tackle the other issues you have identified first, as those seem more urgent. One way of working around this might be to not do test data in a survey that is also used for real data collection, but make a copy of the survey, and use that for testing? In that way you can find the test data back in an easy way.
KK: I can try to get field staff to go through a new procedure for test data but between an established procedure, frequent data collections and mistakes and duplicates that happen from time to time this is something we actually use a lot, staff sends me a list and part of my Friday ritual is to go through and do my weekly list of deletes. So this isn't top priority but would be pretty high up there.
1) Reports are not Bi-lingual – It used to be reports could be run in the master language and the translations, now they can only be run in the master language. It doesn’t make much sense to have surveys that are multi-lingual but not reports that are as well – we need this functionality backMTW: As I understand it, only the headings of the colums were translated, right? At the moment these values are only available in spanish. What we can do is put a dropdown in the export popup, letting the user choose between english and spanish for the headings. Would that be useful?
KK: That would be extremely useful, I know it is just headings but for staff that doesn't speak English not having the headings in Spanish is really a killer for them. Longer term we have had the request from other programs too (Kinyerwandan for example: while all our staff speaks English there have been occasions where they wanted to share raw data with local government who doesn't speak English) so it would be ideal to be able to export the reports with any of the translations in longer term but for us Spanish is the critical one that we are used to having.
2) Comprehensive Report won’t run on some of our bigger surveys (India Waterpoint 2012, India PI 2012, Rwanda Household 2012)MTW: Actually, we have tried all the reports, and the reports are generated. It just takes a long time. When I tried to run the India PI 2012 report, it returned immediately, probaby because you had started it when you tried it. The system now caches reports, so that they are not created more then once if no additional data has come in. Please try again on these files, and let us know the result. I saw the India PI 2012 is 66 MB, so it will have taken a while.
KK: I will try again on these and give them more time.
Thanks for all your work on this!
mtwestra
commented
11 years ago 1) recover the language dropdown #376 - Iván
2) language translations - transifex done by WFP - need to check contributer agreement.
3) move the copy functionality to backend #377 - Iván
4) make recomputation of summaries more efficient in backend, and disregard duplicate s
5) on dashboard and backend in charts, only load question names. - Mark
6) make reports bilingual - dashboard reports, add a flag - Iván
7) http://flowhelp.akvo.org/discussions/problems/308-raw-data-import-on-flow-10
iperdomo
commented
11 years ago Adding to the list.
8) Improvements when loading questions in Reports tab #379
mtwestra
commented
11 years ago Adding to list: 9) copying translations when copying a survey
mtwestra
commented
11 years ago Tentative date for switchover: 8 November
iperdomo
commented
11 years ago Related to point 9) #357
iperdomo
commented
11 years ago 1) recover the language dropdown #376 - Iván
Done
2) language translations - transifex done by WFP - need to check contributer agreement.
Done https://github.com/akvo/akvo-flow/commit/c7d4a86aa99078fbf11cd4c9a655e606355782a2
3) move the copy functionality to backend #377 - Iván
Done
4) make recomputation of summaries more efficient in backend, and disregard duplicate s
Done #385
5) on dashboard and backend in charts, only load question names. - Mark
In progress
6) make reports bilingual - dashboard reports, add a flag - Iván
Done #381
7) http://flowhelp.akvo.org/discussions/problems/308-raw-data-import-on-flow-10
Done
iperdomo
commented
11 years ago 9) copying translations when copying a survey
Fixed in issue #357
mtwestra
commented
11 years ago Test of APK 1) install 1.9.36, and manually download survey 14439637 (akvo test - test apk 1.10.0). ... DONE 2) submit data and check on dashboard ... PASS 3) create translations in survey and republish ...DONE 4) redownload survey and check translations behave as expected ... PASS 5) install 1.12.1 on top of 1.9.36 without removing the old one first ... PASS 6) Open 1.12.1 app .. FAIL (https://dpaste.de/TLYa) error: "Duplicate column name"
Reason was that the db of an earlier install was not deleted. After removing the app + data, we try again: 1..5) PASS 6) Open 1.12.1 app and check if translations behave as expected ... PASS 7) submit data with 1.12.1 and check on dashboard ... PASS
KeriKugler
commented
11 years ago Hi Mark,
I was just going through the list of fixes to test that you sent over this morning.I seem to have found a new issue with Chart Builder. It loads the question names much faster now but it is not loading the actual chart. I choose a survey I know has data in it. For example Survey Group 2013 surveys, Rwanda Water Point 2013, Question: Why Was the Water Point Down for More than One Day? and it gives me the error message "No Data is Available for This Question:Please Select another one" I get the same message no matter what question or survey I choose.
mtwestra
commented
11 years ago Hi Keri,
I'll investigate, thanks, mark
On Oct 15, 2013, at 16:02, Keri notifications@github.com wrote:
Hi Mark,
I was just going through the list of fixes to test that you sent over this morning.I seem to have found a new issue with Chart Builder. It loads the question names much faster now but it is not loading the actual chart. I choose a survey I know has data in it. For example Survey Group 2013 surveys, Rwanda Water Point 2013, Question: Why Was the Water Point Down for More than One Day? and it gives me the error message "No Data is Available for This Question:Please Select another one" I get the same message no matter what question or survey I choose.
— Reply to this email directly or view it on GitHub.
mtwestra
commented
11 years ago Hi Keri,
the reason was that on the testing instance, we deleted all the summaries in order to test the new summary creation code. We then didn't rebuild all the summaries of all the data in the system, as this is an expensive operation. When we move WFP over, we will of course do this. In the mean time, I have rebuild the summaries of only the survey you mentioned, the Rwanda Water Point 2013 one. If you try again, you should be able to see the graph. There is a small remaining issue - the 'no data' warning does not dissapear.
cheers, mark
On Oct 15, 2013, at 16:02, Keri notifications@github.com wrote:
Hi Mark,
I was just going through the list of fixes to test that you sent over this morning.I seem to have found a new issue with Chart Builder. It loads the question names much faster now but it is not loading the actual chart. I choose a survey I know has data in it. For example Survey Group 2013 surveys, Rwanda Water Point 2013, Question: Why Was the Water Point Down for More than One Day? and it gives me the error message "No Data is Available for This Question:Please Select another one" I get the same message no matter what question or survey I choose.
— Reply to this email directly or view it on GitHub.
KeriKugler
commented
11 years ago Hi Mark,
That works for now. Thanks,
Keri
KeriKugler
commented
11 years ago Hi Mark,
One more issue, this may be me still doing something wrong but I still seem to be having trouble with bulk upload. I am taking a zipped surveyal folder from a phone and dropping it into the bulk upload -- it says it uploads and gives me the message that once it is processed there will be a message in the messages tab but after substantial time no message appears. I can email you a copy of a file I am trying to upload they are too big to upload here.
mtwestra
commented
11 years ago Hi Keri,
yes, please send me the file, and the zipped version that you tried to upload, so I can try to diagnose what is going wrong.
thanks mark
On Oct 15, 2013, at 17:12, Keri notifications@github.com wrote:
Hi Mark,
One more issue, this may be me still doing something wrong but I still seem to be having trouble with bulk upload. I am taking a zipped surveyal folder from a phone and dropping it into the bulk upload -- it says it uploads and gives me the message that once it is processed there will be a message in the messages tab but after substantial time no message appears. I can email you a copy of a file I am trying to upload they are too big to upload here.
— Reply to this email directly or view it on GitHub.
KeriKugler
commented
11 years ago Hi Mark,
I found a few more glitches yesterday when doing a demo:
1) Data Tab -- Data Cleaning Page isn't working. If I try to run a raw data report there I choose a survey group, then a survey then a language (I tried both English and Spanish) and then give it time to load and click Raw Data Report and I get an Error Message that I must choose a survey (even though I already did). I tried with multiple surveys. Same thing happens with Import data.
2) If I create a new survey with a Master Language other than English (Spanish). Then attempt to put in translations two things happen. 1) In the translations the master language shows as English even if you initially set it as something else. 2) the Title and Description of the survey do not populate in the Master language field for translation.
Thanks, Keri
mtwestra
commented
11 years ago Hi Keri,
you are our best tester by far! I am investigating them.
thanks! mark
On Oct 16, 2013, at 14:40, Keri notifications@github.com wrote:
Hi Mark,
I found a few more glitches yesterday when doing a demo:
1) Data Tab -- Data Cleaning Page isn't working. If I try to run a raw data report there I choose a survey group, then a survey then a language (I tried both English and Spanish) and then give it time to load and click Raw Data Report and I get an Error Message that I must choose a survey (even though I already did). I tried with multiple surveys. Same thing happens with Import data.
2) If I create a new survey with a Master Language other than English (Spanish). Then attempt to put in translations two things happen. 1) In the translations the master language shows as English even if you initially set it as something else. 2) the Title and Description of the survey do not populate in the Master language field for translation.
Thanks, Keri
— Reply to this email directly or view it on GitHub.
mtwestra
commented
11 years ago Hi Keri,
on 1) good point, our fault. We fixed the loading of questions in the 'export reports tab', but forgot this tab. It is fixed now.
on 2) I think this was caused by setting a text, description and master language on a survey and then going to translations, without saving the changes first. This is a logical mistake, and I have added a check to check for unsaved changes, and requires the user to save the changes to the survey first, before going to translations. Also fixed.
I haven't looked into the bulk upload issue yet, will do so tomorrow morning.
thanks as always for your sharp eyes!
cheers, mark
On Oct 16, 2013, at 14:40, Keri notifications@github.com wrote:
Hi Mark,
I found a few more glitches yesterday when doing a demo:
1) Data Tab -- Data Cleaning Page isn't working. If I try to run a raw data report there I choose a survey group, then a survey then a language (I tried both English and Spanish) and then give it time to load and click Raw Data Report and I get an Error Message that I must choose a survey (even though I already did). I tried with multiple surveys. Same thing happens with Import data.
2) If I create a new survey with a Master Language other than English (Spanish). Then attempt to put in translations two things happen. 1) In the translations the master language shows as English even if you initially set it as something else. 2) the Title and Description of the survey do not populate in the Master language field for translation.
Thanks, Keri
— Reply to this email directly or view it on GitHub.
KeriKugler
commented
11 years ago Thanks Mark!
From: Mark Westra [mailto:notifications@github.com] Sent: Wednesday, October 16, 2013 10:03 AM To: akvo/akvo-flow Cc: Keri Kugler Subject: Re: [akvo-flow] Upgrade Water For People to latest version (#330)
Hi Keri,
on 1) good point, our fault. We fixed the loading of questions in the 'export reports tab', but forgot this tab. It is fixed now.
on 2) I think this was caused by setting a text, description and master language on a survey and then going to translations, without saving the changes first. This is a logical mistake, and I have added a check to check for unsaved changes, and requires the user to save the changes to the survey first, before going to translations. Also fixed.
I haven't looked into the bulk upload issue yet, will do so tomorrow morning.
thanks as always for your sharp eyes!
cheers, mark
On Oct 16, 2013, at 14:40, Keri notifications@github.com<mailto:notifications@github.com> wrote:
Hi Mark,
I found a few more glitches yesterday when doing a demo:
1) Data Tab -- Data Cleaning Page isn't working. If I try to run a raw data report there I choose a survey group, then a survey then a language (I tried both English and Spanish) and then give it time to load and click Raw Data Report and I get an Error Message that I must choose a survey (even though I already did). I tried with multiple surveys. Same thing happens with Import data.
2) If I create a new survey with a Master Language other than English (Spanish). Then attempt to put in translations two things happen. 1) In the translations the master language shows as English even if you initially set it as something else. 2) the Title and Description of the survey do not populate in the Master language field for translation.
Thanks, Keri
— Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHubhttps://github.com/akvo/akvo-flow/issues/330#issuecomment-26431674.
mtwestra
commented
11 years ago Hi Keri,
We found an issue in the upload mechanism, that we fixed. We uploaded the 20MB file without problems after that.
cheers, mark
On Oct 15, 2013, at 17:12, Keri notifications@github.com wrote:
Hi Mark,
One more issue, this may be me still doing something wrong but I still seem to be having trouble with bulk upload. I am taking a zipped surveyal folder from a phone and dropping it into the bulk upload -- it says it uploads and gives me the message that once it is processed there will be a message in the messages tab but after substantial time no message appears. I can email you a copy of a file I am trying to upload they are too big to upload here.
— Reply to this email directly or view it on GitHub.
iperdomo
commented
11 years ago @KeriKugler it would be nice if you can try again, with the original file. It won't hurt and you'll get familiar with the new approach for uploading.
KeriKugler
commented
11 years ago Thanks for looking into this Mark and Ivan, I will retry today.
KeriKugler
commented
11 years ago Hi Mark, one of our staff did a demo in Peru last week of the new dashboard for our Latin America country directors. They noticed a few places where on the Spanish version of the dash the English is not translated to Spanish.
Survey Tab: Edit Survey or Create New Survey: In the Boxes Under Title and Description there is still English Instructions "Type in the name of your survey" and "Type in a description of your survey"
When attempting to add translations to an unsaved survey the notification that pops up is still in English "Before editing translations, you have to save your changes to the survey by clicking the "save" button"
Data Tab: On the Data Cleaning tab the button for "Browse" under the import is still in English and so is the message "No File Selected"
Reports Tab: They types of graphs are still in English "Doughnut Chart" "Vertical Bar Chart" "Horizontal Bar Chart" so is "Choose a Question from above selectors"
Maps: I realize the maps are imported in from Google maps so there is likely no way to change this but the buttons on the map "Terrain" "Map" "Satellite" and "Show" are all in English.
I got more feedback from the staff if it is helpful to hear but nothing else was a technical issue that had a quick fix, they were suggestions that would be longer term features or fixes or just opinions that may or may not be useful. If you want me to go ahead and let you know about the feedback let me know if not I will hold onto it for now.
iperdomo
commented
10 years ago v1.6.7locale instead of accessPoint for domain settingsdefaultLanguage to en, and pointType to PointUser level, except for @KeriKugler to Admincc/ @KeriKugler @mtwestra @henryjewell @bjelkeman
mtwestra
commented
10 years ago This must have been the longest issue so far!
On Nov 11, 2013, at 08:54, Iván Perdomo notifications@github.com wrote:
Closed #330.
— Reply to this email directly or view it on GitHub.
Notes in prep for migration:
We need to upgrade Water For People's dashboard from 1.0 to 1.6.1. We'll start this process by creating a clone of their datastore and laying the 1.6.1 app over it, and doing an evaluation of what issues, if any, arise in the transition. Then turn it over to them with some notes on our findings, for their more detailed review.
Some preliminary things to do / look out for:
Data model and database
Maps
Surveys
Survey group view:
Devices
Data
Reports
Users
Messages
Functionality that is no longer accessible in 1.5.0 and higher