ODK Authentication stops working on ODK Collect for no apparent reason.

[GoogleCodeExporter]

We are running ODK Aggregate (v1.2.0) on our own server and it all seems to 
work fine. We have set up ODK authenticated Data Collector users in Aggregate 
and matched those credentials as stored username/password settings on the ODK 
Collect (v1.2.1) app on our Android devices. Doing a "Get Blank Form" works 
fine with those credentials and we get the form uploaded onto the Android 
device. "Fill Blank Form" works fine and we can finish and finalize forms with 
no problems.

However, when it comes time to "Send Finalized Form", the Android phone reports 
an authentication error (despite the credentials being correct) against the 
Aggregate server and asks for a username and password to be entered. I can type 
in any valid username and password and it still will not send the form. All 
sets of credentials are in fact valid and work fine when we use a web browser 
to log in to Aggregate itself.

The only way we seem to be able to fix it is to do another "Get Blank Form" and 
re-upload the same form. Magically the "Send Finalized Form" starts working 
again with no change to the stored username/password settings.

Can anyone think of a reason why this might occur? (..and how we might stop it 
happening?)

Thanks

Evan Yates

Original issue reported on code.google.com by wbrune...@gmail.com on 26 Sep 2012 at 6:13

[GoogleCodeExporter]

Are you using the latest ODK Collect 1.2.1 (1015) build for ODK Collect? 
(shown on the main screen).

(thank you for the detailed reproduction/recovery description)

Original comment by mitchellsundt@gmail.com on 4 Oct 2012 at 9:04

[GoogleCodeExporter]

Yes, I am using v1.2.1 (1015) of ODK Collect.

Thanks for following this up.

Evan Yates
Technology Manager
AsureQuality Ltd.

Original comment by kotuku...@gmail.com on 4 Oct 2012 at 10:55

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

This is a link to a video that shows the issue in action.

http://www.youtube.com/watch?v=goAtrkh456U

Hope this helps.

Original comment by kotuku...@gmail.com on 1 Nov 2012 at 11:14

[GoogleCodeExporter]

As the video shows, we are also using the latest ODK Collect 1.2.1 (1020) and 
the issue is still occurring.

Original comment by kotuku...@gmail.com on 2 Nov 2012 at 12:54

[GoogleCodeExporter]

I have created a generic login to our system to allow a development team member 
to try our form and see why this is happening.

Server = https://odktrial.asurequality.com/ODKAggregate/
Username = DataCollector1
Password = collectdata
Form = FruitFlyV6

Is it something about our Aggregate installation that causes the authentication 
to pop-up on a "Send Finalized Form"?

Once someone from the dev team has tried the issue and seen the error, let me 
know and I will turn off the generic login.

Thanks

Original comment by kotuku...@gmail.com on 19 Nov 2012 at 11:09

[GoogleCodeExporter]

Any progress with this issue?

Original comment by kotuku...@gmail.com on 4 Dec 2012 at 1:16

[GoogleCodeExporter]

I haven't found a clear cause for this.  
Testing from here to your server on a Droid (Android 2.2.3) works.

What phone/device are you using? What Android OS version?

Can you install CatLog and attach a log capture to this issue taken during your 
reproduction of this issue on your device? 
http://code.google.com/p/opendatakit/wiki/CollectTroubleshooting

Original comment by mitchellsundt@gmail.com on 5 Dec 2012 at 7:26

[GoogleCodeExporter]

We are using Samsung Galaxy Ace Plus phones (GT-S7500) with Android 2.3.6

I will get a CatLog capture to you ASAP. I'm not sure if it will help as ODK 
Collect doesn't crash, it just refuses to acknowledge valid authentication 
until after a new "Get Blank Form" is performed.

Thanks

Original comment by kotuku...@gmail.com on 10 Dec 2012 at 12:24

[GoogleCodeExporter]

LogCat file attached

Original comment by kotuku...@gmail.com on 10 Dec 2012 at 12:38

Attachments:

• logcat_and_device_info.zip

[GoogleCodeExporter]

Hmm. I'm seeing these errors:

12-10 13:29:48.809 W/System.err(989): java.net.UnknownHostException: 
odktrial.asurequality.com
12-10 13:29:48.819 W/System.err(989):   at 
java.net.InetAddress.lookupHostByName(InetAddress.java:508)
12-10 13:29:48.819 W/System.err(989):   at 
java.net.InetAddress.getAllByNameImpl(InetAddress.java:296)
12-10 13:29:48.819 W/System.err(989):   at 
java.net.InetAddress.getAllByName(InetAddress.java:258)
12-10 13:29:48.819 W/System.err(989):   at 
org.opendatakit.httpclientandroidlib.impl.conn.SystemDefaultDnsResolver.resolve(
SystemDefaultDnsResolver.java:45)

--------
This likely indicates a problem with your network gateway(s). Often networks 
have 2 gateways that supply name look-ups (i.e., from odktrial.asurequality.com 
to an IP address). 

It looks like one of these 2 gateways does not have an entry for 
odktrial.asurequality.com.  Note that the gateway(s) are specified in your WiFi 
router (or the cellular network, if using that for connectivity).

Per other reported Android issues on this, "Addresses are cached for 600 
seconds (10 minutes) by default. Failed lookups are cached for 10 seconds."

So it is likely if you just waited 10+ seconds, and attempted to re-send the 
submission, that, too, should work.  If two gateways are available, the phone 
will alternate which it uses when requesting a lookup. Hence every-other 
connection seems to go OK, or every interaction within 10 minutes of a 
successful lookup goes OK, then the cache expires (e.g., while filling out a 
form), and the next attempt fails.

I haven't tried it, but this app might provide some insight into what the phone 
is seeing for your server address: 
https://play.google.com/store/apps/details?id=com.ulfdittmer.android.ping&hl=en

Original comment by mitchellsundt@gmail.com on 10 Dec 2012 at 6:24

[GoogleCodeExporter]

I think those Network error are a red-herring. As you say, I just waited and 
tried again to solve the network connection when the cache expired.

The real issue actually shows up at about line 7188 of the original log file. 
I've edited the log file (and attached the shortened modified one) to start 
from the ODK form submission line.

You can see a /submission (line 1) is called and very soon after a 
"clearHostCredentials" is called (line 23) when ODK splashes up the credentials 
screen. You can see I try this a couple of times (lines 57, 124)
I then do a "Get  Blank Form" (line 330)
After that the next "Send Finalized Form" succeeds (line 435)

Does this change anything? I'm hoping this is just an ODK Aggregate 
configuration thing rather than a bug in ODK Collect, but I'm not sure how to 
go about diagnosing it beyond observing behaviour.

Thanks

Original comment by kotuku...@gmail.com on 11 Dec 2012 at 8:24

Attachments:

• logcat_edited.txt

[GoogleCodeExporter]

Download the "ODK Collect v1.2.2 uploader 1022.apk" test build of Collect here: 

http://homes.cs.washington.edu/~msundt/collect/

If you allow (have checked) : Settings / Applications / Unknown Sources

Then you can use the browser on the phone to go to the above URL, and click on 
the apk file, and install ODK Collect.

You will likely need to uninstall your existing ODK Collect to be able to 
install this one.

With this build, try to reproduce the problem and attach the logs if it is 
still happening.  I made some changes to how the activity is saved and restored 
which might fix this issue.

Mitch

Original comment by mitchellsundt@gmail.com on 12 Dec 2012 at 2:05

[GoogleCodeExporter]

I just updated the APK on the above page. Try the REVISED apk.  I think the 
uploads likely did succeed, but that a variable was not being re-initialized, 
causing the erroneous re-asking of the username/password.

Original comment by mitchellsundt@gmail.com on 12 Dec 2012 at 5:39

[GoogleCodeExporter]

I have downloaded the v1.2.2 REVISED apk from your website and installed it. 
I'm still getting the authentication dialog on Submit Finalized Form. I have 
noticed that it can be made to happen by re-booting the phone, going in to ODK, 
Filling in a Blank Form and trying to Send Finalized Form straight away. 
Once you have re-loaded the Get Blank Form, the Send Finalized starts to work 
again and keeps working until the app is closed and the phone goes to sleep. 
Something to do with submission authentication seems to get lost in the "ODK 
Collect shutdown" process.
Logfile attached. The action starts on line 5638

Original comment by kotuku...@gmail.com on 13 Dec 2012 at 8:33

Attachments:

• logcat_and_device_info.zip

[GoogleCodeExporter]

I'm still unable to replicate the problem on 2.2.3 or 2.3.5 even with a reboot 
and initial launch.

I made some minor changes and added logging statements in the recently released 
1.2.2 (rev 1023), but doubt they will have much effect on your problem.

I suspect it is related to the specific OS version and hardware on your phone.  
if there are any ROM updates available from your telecom provider or from 
Samsung, you might try applying them.

Original comment by mitchellsundt@gmail.com on 19 Dec 2012 at 12:35

[GoogleCodeExporter]

Ok. Looks like it is related to the model of phone we use. I will investigate 
OS updates from Samsung and will report back. I'm happy for you to close this 
issue.

Thanks

Original comment by kotuku...@gmail.com on 19 Dec 2012 at 9:12

[GoogleCodeExporter]

I have made another change to the code. Can you download the latest update on 
the above link and try it?

My supposition is that the first attempt to upload fails with a connection 
timeout, and that this poisons all subsequent upload attempts.

Original comment by mitchellsundt@gmail.com on 4 Jan 2013 at 6:54

[GoogleCodeExporter]

It may be a Samsung issue. We started getting this error on the Samsung Galaxy 
Pocket running 2.3.6 when we started capturing images on the forms. It seems to 
timeout related as we have very slow, intermittent Edge networks where we are 
collecting. I was able to upload the files over WIFI. However we just switched 
to the source with admin settings (another variable). None of the tests failed 
on the earlier versions of collect. No issues so far on rev 1024 but I wont 
really know until the end of next week.

Original comment by i...@tgharusha.com on 12 Jan 2013 at 6:26

[GoogleCodeExporter]

OK. Let me know later this/next week if 1024 fixes the issue. I now think it 
was due to the initial connection failing with a timeout, and then not fully 
resetting.  The form download had the reset logic, but not the submission code 
(I discovered this as I was working on ODK Survey release)

Original comment by mitchellsundt@gmail.com on 14 Jan 2013 at 5:27

[GoogleCodeExporter]

Thanks, Mitch.

I have also just downloaded build 1024 of Collect and I am still getting the 
same issue on our Samsung Galaxy Ace Plus phones.

Original comment by kotuku...@gmail.com on 21 Jan 2013 at 1:37

[GoogleCodeExporter]

Frustrating; I'm stumped.

Original comment by mitchellsundt@gmail.com on 22 Jan 2013 at 6:52

[GoogleCodeExporter]

Hi there,

I'm sorry if this seems like a basic question, but I am very new to ODK and am 
trying to set up aggregate on my mac laptop, and collect on a nexus 7, to test 
out before I go into the field in two weeks.

I seem to have set up aggregate ok, and collect ok, and i can upload forms to 
aggregate and then access them in collect and fill them in, but when I try to 
re-upload actually completed surveys from my nexus 7 to odk aggregate, I get 
the following message, and I really am not sure what this means/ what I need to 
do. I have tried to look for others with similar issues but so far have not 
come across any answers that seem comprehensible to a computer lay person.

Error: Not found (404) at 
https://example-odk-aggregate.appspot.com/submission?deviceID=mac%3A30%3A85%3Aa9
%3A66%3Ad4%3A37.

My appspot is ve-evaluation.appspot.com, so I get that my nexus is not 
recognizing the server (I think) but I'm not sure how to fix it.

Any help would be tremendously appreciated.

Many thanks,

Andrea

Original comment by dr.a.a.a...@gmail.com on 25 Feb 2013 at 5:00

[GoogleCodeExporter]

This is a separate issue. Please don't add a comment to an existing issue, but, 
instead, open a new one. If the issues are, in fact, the same, we will mark 
them as duplicates.

In your case:
(1) open the XLS file that you are using to generate the form.
(2) click on the 'settings' sheet in the XLS file
(3) notice that there is a 'submission_url' column on this page, with a value 
of https://example-odk-aggregate.appspot.com/submission

You will either want to revise this column to identify the server the data 
should be submitted to, or, if you delete this column entirely, the submission 
would be sent to whatever the individual phone has configured as the default 
server location.

Original comment by mitchellsundt@gmail.com on 25 Feb 2013 at 6:24

[GoogleCodeExporter]

Thanks, this resolved it.

Have a great day!

Original comment by dr.a.a.a...@gmail.com on 28 Feb 2013 at 1:08

[GoogleCodeExporter]

I have been asked to further investigate the Authentication failure issue as we 
are still keen on further deployment of our ODK forms setup, but our field 
staff are being frustrated by the error screen (even though we have the 
workaround of "re-Getting" the form.
Are we looking at this from the wrong perspective? Perhaps the problem is being 
caused by something on the ODK Aggregate server rather than in the ODK Collect 
client?
Are there any ODK Aggregate server configuration settings we could look at 
which might affect the authentication process during Form Submission?

Original comment by kotuku...@gmail.com on 4 Mar 2013 at 9:57

[GoogleCodeExporter]

I welcome your help.  

Do any of your team have Java programming skills?
Or network or system admin skills?

One clarification: The build I made earlier will generally not eliminate the 
first failure pop-up, but it should succeed on the 2nd try.  If it continues to 
always fail, that is what we need to fix. If it does work on the 2nd try, then 
we are much closer to a solution.

----
I don't believe it is a server-side software issue. The same authentication 
path is used for getting forms and for uploading submissions, so having one 
fail and the other work is very odd. 

But the requests to the server are different: form download starts with a GET 
request, and instance upload is a HEAD request followed by a POST request.  
These could be processed differently on Google AppEngine.  If you have network 
admin experience, you could set up a network sniffer to see exactly what gets 
sent between the server and the cell phone (via wifi connection), and that 
could help diagnose the problem and detect any strange behavior w.r.t. the 
server.  If you have those skills, let me know and I can walk you through how 
to configure a Google Appspot with http connections (vs. https), so that we can 
more easily understand what transpires between the phone and server.

There are very different code paths on the client side, between the 
DownloadFormListTask and InstanceUploaderTask ( 
http://code.google.com/p/opendatakit/source/browse?repo=collect#hg%2Fsrc%2Forg%2
Fodk%2Fcollect%2Fandroid%2Ftasks )
and the UI activities paired with these: FormDownloadList and 
InstanceUploaderList ( 
http://code.google.com/p/opendatakit/source/browse?repo=collect#hg%2Fsrc%2Forg%2
Fodk%2Fcollect%2Fandroid%2Factivities ).

My current hypothesis is that the initial connection times out and fails during 
upload, and that this somehow either corrupts the socket or corrupts the 
context, and we are not cleaning those up properly before the retry attempt.

Original comment by mitchellsundt@gmail.com on 4 Mar 2013 at 10:30

[GoogleCodeExporter]

Issue 545 has been merged into this issue.

Original comment by mitchellsundt@gmail.com on 18 Mar 2013 at 9:03

[GoogleCodeExporter]

What Android OS does your phone have? Is it running 2.2 perhaps?

Original comment by mitchellsundt@gmail.com on 28 Mar 2013 at 5:27

[GoogleCodeExporter]

The Samsung Galaxy Ace we use runs Android 2.3.6
I'm marshaling some internal network and server resources to do further 
sniffing and tracing, but it will be a week or so before we get organized.

Original comment by kotuku...@gmail.com on 29 Mar 2013 at 5:06

[GoogleCodeExporter]

I believe I found the issue. Hopefully this if fixed in the ODK Collect 1.3 
release. Please let me know if this is still an issue in that release (now 
pushing out through Google Play and on the website for download).

Original comment by mitchellsundt@gmail.com on 12 Apr 2013 at 9:02

[GoogleCodeExporter]

I have just loaded ODK Collect 1.3 onto a Samsung Galaxy Ace Plus running 
Android 2.3.6.

When I go to send a finalized form, instead of continually asking for 
Authentication details as it used to, the app now consistently crashes with a 
"Sorry, ODK Collect has stopped unexpectedly" error. All I can do is tap "Force 
Close" and then exit.

I can still do a "Get Blank Form" ok and then send the finalized forms 
successfully. It just seems that now, instead of failing the Authentication as 
it used to do, the app is now actually crashing.

On another note, I really appreciate the "Toggle All" option now available in 
the "Delete Forms" screen. This will enable our people to clear out their 
phones a lot more easily. (Though I won't get them to upgrade until the new 
crash is fixed)

Original comment by kotuku...@gmail.com on 14 Apr 2013 at 11:45

[GoogleCodeExporter]

Ok. Can you send me the logcat of the crash. 

http://code.google.com/p/opendatakit/wiki/CollectTroubleshooting

This is actually a good sign...

Original comment by mitchellsundt@gmail.com on 15 Apr 2013 at 2:13

[GoogleCodeExporter]

I just pushed a new version (ODK Collect 1.3 rev 1027). 

Please install/download that. The Force Close should go away and hopefully the 
authentication issue will also be resolved.

Original comment by mitchellsundt@gmail.com on 15 Apr 2013 at 2:45

[GoogleCodeExporter]

I have downloaded and installed the v1.3 rev 1027 of the app. The Force Close 
has gone away but the authentication failure issue has returned same as before.

I assume that you no longer need the logcat output for the "Force Close" 
issue...?

Thanks for your continued efforts.

Original comment by kotuku...@gmail.com on 15 Apr 2013 at 10:56

[GoogleCodeExporter]

Ok. Yes, I don't need the logcat. 

It is frustrating that the issue is still unresolved.

Original comment by mitchellsundt@gmail.com on 15 Apr 2013 at 11:57