search

onaio / mspray

Mspray
https://mspray.onalabs.org/
Apache License 2.0
3 stars 1 forks source link

Look into why new structures from Reveal are not showing up #513

Closed moshthepitt closed 5 years ago

moshthepitt commented 5 years ago

Ensure that new points from Reveal via Nifi are received and processed correctly.

The data comes in this format:

{
    "id": "71e2d011-a498-46d6-8986-cd15e5cf1d6f",
    "parent_id": "3537",
    "status": "Pending Review",
    "geometry": "{\"type\":\"Point\",\"coordinates\":[32.6446266608928,-14.1567959784487]}",
    "server_version": 1547719286599,
    "task_id": "d0fd193d-bf88-4cdd-abbe-8683da12031b",
    "task_spray_operator": "demomti",
    "task_status": "Ready",
    "task_business_status": "Not Visited",
    "task_execution_start_date": "",
    "task_execution_end_date": "",
    "task_server_version": 1548159600347
}
moshthepitt commented 5 years ago

It seems that this works as expected.

Some notes:

moshthepitt commented 5 years ago

mspray reveal new point

moshthepitt commented 5 years ago

@craigappl @githengi and also @ukanga

I've investigated how to make new structures show up even when they have no spray_status (task_business_status other than Not Visited). i.e. show a yellow circle on a map where a new structure that has not yet been visited is located.

mSpray currently does not support it - it makes the assumption that when you add a new structure to the map then at the same time you mark it as Sprayed/Not Sprayed/Not Sprayable. So, technically, all the new points in the mSpray system currently do have to have a spray_status.

I can modify mSpray to make these new structures show up even when they have no spray_status but it a significant amount of work, in my view - because it changes a core assumption of the mSpray system.

I am on the fence about whether we should actually do this because:

  1. I do agree with the assumption that when you add a new structure you have in essence visited it and should at that time set some status
  2. I view using mSpray in reveal as a temporary thing and I am not sure whether this investment in time and effort should be done right now

With that said, I would love to have your views on this. Specifically whether you think we should go ahead and do this bit of work.

craigappl commented 5 years ago

@moshthepitt , how do we currently get yellow structures in mSpray? Are those loaded in to the location instead of spray days?

moshthepitt commented 5 years ago

@craigappl yes, the yellow structures are loaded as locations at the beginning of the season

craigappl commented 5 years ago

Is the LOE of loading them in as locations what you're describing here?

craigappl commented 5 years ago

So, can we just load them all in as locations instead of loading them in as sprayed items?

moshthepitt commented 5 years ago

That's the only way we would end up doing it in my view @craigappl. The complexity I mentioned arises in how to tell if a structure is new or not as that currently is tied to spray events.

The level of effort is 2 or so days of me working on this, but there may be unexpected changes/errors in the indicator calculations that aren't easy to anticipate. But if everything goes well, its two or so days of me working on this.

moshthepitt commented 5 years ago

I'm going to close this issue.

Some final notes:

  1. Nifi uses the "task server version" field from OpenSRP tasks to determine if a task has been updated or not
  2. Nifi periodically queries OpenSRP for updated tasks, and syncs them if found
  3. Nifi periodically sends synced tasks to mSpray

There were some problems with the above, which have since been solved (or so we hope):

  1. The reveal app was not setting a new task server version when the spray status (aka task business status) field on a task changed. Nifi was thus unable to know that a task status had been updated. This was fixed
  2. The reveal Nifi deployment was unstable and couldn't reliable sync everything all the time. This has hopefully been fixed now