Fogl 3648:: handling race conditions that produce the message SQLITE_BUSY

[sstefano69]

@YashTatkondawar @MarkRiddoch

Hi Yash, I think (already talked with Mark) we can approve and close this PR as I have done more tests and after a while (1 hour and half) I saw the error but it is a different situation respect this one it is a 'database is locked' that has impacted other tables for example the 'foglamp.tasks'

btw: I tested on 2 RPs:

• one having a degraded sd card where the problem happened
• and one having a good sd card that worked without problems at all

I'm going to create a specific jira about that.

Feb  4 05:40:31 raspberrypi FogLAMP Storage[728]: ERROR: SQLite3 storage plugin raising error: database is locked
Feb  4 05:40:31 raspberrypi FogLAMP Storage[728]: ERROR: SQL statement: BEGIN TRANSACTION;UPDATE foglamp.tasks SET exit_code = 1, state = 2, end_time = '2020-02-04 05:37:07.246885+00:00' WHERE id = '692df3fd-7b7b-4e1b-a1d4-b3132f7aaa31';COMMIT TRANSACTION;
Feb  4 05:40:31 raspberrypi FogLAMP[714] ERROR: storage_client: foglamp.common.storage_client.storage_client: Error code: 400, reason: Bad Request, details: {'entryPoint': 'update', 'message': 'database is locked', 'retryable': False}
Feb  4 05:40:31 raspberrypi FogLAMP[714] ERROR: scheduler: foglamp.services.core.scheduler.scheduler: Update failed: {"values": {"exit_code": 1, "state": 2, "end_time": "2020-02-04 05:37:07.246885+00:00"}, "where": {"column": "id", "condition": "=", "value": "692df3fd-7b7b-4e1b-a1d4-b3132f7aaa31"}}#012Traceback (most recent call last):#012  File "/usr/local/foglamp/python/foglamp/services/core/scheduler/scheduler.py", line 273, in _wait_for_task_completion#012    res = await self._storage_async.update_tbl("tasks", update_payload)#012  File "/usr/local/foglamp/python/foglamp/common/storage_client/storage_client.py", line 204, in update_tbl#012    raise StorageServerError(code=resp.status, reason=resp.reason, error=jdoc)#012foglamp.common.storage_client.exceptions.StorageServerError: code: 400, reason:Bad Request, error:{'entryPoint': 'update', 'message': 'database is locked', 'retryable': False}