409 error when using headless API

[MartijnR]

See email on Aug 2nd.

We are testing headless form usage and we encountered the following error infrequently:

2021-07-31 06:53:29,207 ERROR [pool-15-thread-1] c.o.a.o.s.c.x.EnketoAPI [EnketoAPI.java:953] Error calling bash
HeadlessMode api: 409 {
    "code": 409,
    "message": "Conflict. Busy handling pending request for same survey"
}

This occurred twice while running logic for 800 different instances of FormX in a batch. We were passing the calls in continuously to keep the server cores fully utilized, but each record we passed in was unique so there shouldn't have been any cases of trying to process the same record multiple times concurrently.

Could be similar to https://github.com/enketo/enketo/issues/1069

[MartijnR]

tried reproducing with:

curl --user enketorules: -d "server_url=https://enketo-aggregate.appspot.com&form_id=vitals&instance=<demo-vitals-grid-v9><page1><wh_y>1</wh_y></page1></demo-vitals-grid-v9>&instance_id=b&load_warning=Hey you&ecid=a" http://localhost:8005/oc/api/v1/instance/headless & curl --user enketorules: -d "server_url=https://enketo-aggregate.appspot.com&form_id=vitals&instance=<demo-vitals-grid-v9><page1><wh_y>1</wh_y></page1></demo-vitals-grid-v9>&instance_id=c&load_warning=Hey you&ecid=a" http://localhost:8005/oc/api/v1/instance/headless

Not successful yet.

[MartijnR]

I misunderstood. This protection is for the case in which a survey db entry is being created and another request in a different thread for the same survey comes in while doing this. It avoids creating 2 different IDs for the same survey. This was expected to be a very unlikely event.

In this case, I think the survey db entry was already created and there is nothing to update. But in order to check this you'd have to make a db query which still takes time. Not sure if we can completely eliminate this from happening.

However, this code seems to be flawed:

 _getEnketoId( openRosaKey )
                .then( id => {
                    if ( id ) {
                        survey.active = true;
                        delete pending[ openRosaKey ];
                        resolve( _updateProperties( id, survey ) );
                    } else {
                        resolve( _addSurvey( openRosaKey, survey ) );
                    }
                } )
                .catch( error => {
                    delete pending[ openRosaKey ];
                    reject( error );
                } );

• The else code should also clear the pending object.
• In both cases (if and else) it seems to me that this should happen after _updateProperties/_addSurvey.
• [ ] reproduce, seems difficult
• [ ] move to enketo/enketo-express (and first merge some of the outstanding PRs that may touch this code)

[MartijnR]

I think this protection may very limited it won't work across multiple threads since each thread will have its own pending object.

We should probably invent a solution that works across threads but this seems quite a challenge.

However, to address the current issue, the best would be to retry when receiving a 409 response.

[MartijnR]

• [ ] Martijn to check if there is any reason why set account would be called more often than expected (because that is is what is causing this 409)

[pbowen-oc]

We will retest this now that the form engine has been updated to the new evaluator to see if we still see this problem.

[pbowen-oc]

408 "message": "waiting for selector #headless-result failed: timeout 120000ms exceeded" 409 "message": "Conflict. Busy handling pending request for same survey"