Mexico Data not displaying?

[tonysecurityforcemonitor]

@sam-ffctn @evz - Not sure what the issue is here, when I go to the Map it is mostly blank - http://ffctn.com/clients/on/sfm/M19/#country/mx!date=2014-01-01&focus=map

However, when I search multiple units that aren't displayed are in the system: http://ffctn.com/clients/on/sfm/M19/#country/mx/o/33c5c094-fb41-4d45-9d9c-074ce428acf2!date=2014-01-01&focus=map&overlay=organization-dossier

[sam-ffctn]

@tonysecurityforcemonitor @evz Currently I'm only getting 4 organizations for this endpoint (the ones displayed on the map): https://sfm.datamade.us/api/countries/mx/geometries

The API should be listing all the organizations for that endpoint.

[sam-ffctn]

@tonysecurityforcemonitor One thing to note is that the organization may not be visible at due to the date selected on the timeline. (which wasn't working for mexico until now)

Selecting different dates will reveal other organizations.

[tonysecurityforcemonitor]

@evz It looks like the relevant data is in the CMS so there should be Areas and Sites displaying on the Map from at least 2004 onwards...

[evz]

OK, I'll check this out

[evz]

So, it looks like I am inadvertently filtering out quite a few OSM nodes that are used in the Mexico data. This cascades into the organizations not showing up on the map because they don't have any geospatial data associated with them. I'm looking into that now.

[evz]

OK, this is getting a bit strange. I was able to get all the site data to match up but now I've only got 12 organizations that have sites and also have a division identifier for Mexico. Still trying to figure out where the problem lies.

[evz]

Alright, I figured this out. I'll need to re-import the mexico data from OSM which will take a while.

[tonysecurityforcemonitor]

Thanks for the update @evz glad its being sorted! Would it cause a problem for me to update the Master Import Sheet? There have been a number of updates and I'd be interested to see how they look in the UI.

[evz]

Nope. Have at it.

[tonysecurityforcemonitor]

@evz - I've updated the Master Import Sheet

[jsmoak]

@evz @tonysecurityforcemonitor how is the data re-import coming along?

[tonysecurityforcemonitor]

@jsmoak - I see that the updated Mexico and Nigeria data have gotten into the system - there is still an issue of Areas not displaying, but I think this is due to the re-import of the OSM data, right @evz?

[evz]

Yeah, I had make sure that we were not filtering out some of the OSM nodes that we are relying on for those orgs when importing the OSM data. It looks like we are getting quite a few more to display now:

http://ffctn.com/clients/on/sfm/M20/#country/mx!date=2014-01-01&focus=map

[tonysecurityforcemonitor]

@evz Yep I see a lot more Nodes loading - as far as the Areas I suspect when we add that OR conditionality as noted in #43 we should be seeing more areas as well, right?

[evz]

@tonysecurityforcemonitor I think once we get #43 sorted out, this will be looking a lot better.

[sam-ffctn]

@tonysecurityforcemonitor mexico does seem to be looking a lot better now. with the fix from #43

http://ffctn.com/clients/on/sfm/M22/#country/mx!date=2014-01-01&focus=map

[tonysecurityforcemonitor]

@evz - there should be Area data to cover the entire map of Mexico... my understanding is the map will display an area even if an organization does not have a Site. @sam-ffctn is that true?

[sam-ffctn]

@tonysecurityforcemonitor The map will show all the areas and sites/installations of the organizations in that country for a given point in time.

If an part of the map is not covered that means that no organization has an area over that part of the map.

[tonysecurityforcemonitor]

Thanks @sam-ffctn - @evz there seems to be an API issue then in both Nigeria and Mexico as the sum of all the Areas for organizations with the classification of Army should cover the entire map, but don't seem to be. The issue is also apparent in other classifications as well.

[tonysecurityforcemonitor]

Hi @evz just checked on this for the newest version - this is still and issue. For Nigeria for instance the 2 Mechanised Division should be displaying on the Map, but isn't, for Mexico many organizations are missing including 33 Zona Militar and 32 Zona Militar.

[evz]

@tonysecurityforcemonitor I'm trying to figure out what's going on here but I can't seem to be able to find 33 Zona Militar or 32 Zona Militar in the Master Import sheet.

[evz]

Oooohhhh, I see what's happening. We have people that are associated with those orgs so they get created when we create the people. But, since they are not in the org sheet, we don't have any geospatial data about them.

[tonysecurityforcemonitor]

@evz - sorry ignore about 33 Zona Militar or 32 Zona Militar - but the 2 Mechanised Division is in there.

[evz]

Huh, OK, now I'm seeing the 2 Mechanised division on the map:

http://ffctn.com/clients/on/sfm/M22/#country/ng!date=2014-01-01&focus=map

Maybe @sam-ffctn made an update?

[tonysecurityforcemonitor]

@evz Hmmm - but then the 82 Division should be showing up when I filter down to Army over Taraba, Benue, and Ebonyi states - and the 81 Division should be showing up over Lagos and Ogun states...

@sam-ffctn Is this due a filtering issue? Does the Map reload based on the Filter selection? If so, I'd expect those organizations to be displayed on the map as those Areas are the "smallest" and thus would be on the "top" layer, covering the Area of the Nigerian Army Headquarters which is "larger" as it is all of Nigeria (at least that's how I understand the Map to work)

[tonysecurityforcemonitor]

@sam-ffctn - let me know if there are any questions on this - especially if there are issues on the data side of things.

[evz]

@tonysecurityforcemonitor The reason 81 Division is not showing up on the map is because the two OSM IDs for those areas that are in the spreadsheet are not accurate. The reason why only Enugu is showing up for 82 division is because that's the only area that organization has for that date (2014-01-01). These are the other areas we have for 82 division along with first and last cited dates:

    name     |  osm_name   |  osmid  | first_cited | last_cited 
-------------+-------------+---------+-------------+------------
 Abia        | Abia        | 3713501 | 2000-03-02  | 2012-02-16
 Akwa Ibom   | Akwa Ibom   | 3715359 | 2007-10-17  | 2012-02-16
 Anambra     | Anambra     | 3715505 | 2000-03-02  | 2012-02-16
 Bayelsa     | Bayelsa     | 3715844 | 2011-01-31  | 2012-02-16
 Benue       | Benue       | 3716076 | 2011-01-31  | 2012-02-16
 Cross River | Cross River | 3716250 | 2009-10-02  | 2012-02-16
 Ebonyi      | Ebonyi      | 3717071 | 2000-03-02  | 2012-02-16
 Enugu       | Enugu       | 3717212 | 2000-03-02  | 2015-03-24
 Imo         | Imo         | 3717825 | 2000-03-02  | 2012-02-16
 Rivers      | Rivers      | 3720743 | 2011-01-31  | 2012-02-16
 Taraba      | Taraba      | 3720850 | 2011-01-31  | 2012-02-16

Does that seem accurate?

[sam-ffctn]

@tonysecurityforcemonitor @evz The filter simply hides/shows the organizations based on if the organizations classification property is included one of the selected classifications. (I match the strings). I don't think the filter is the issue.

[tonysecurityforcemonitor]

@evz Ah ok I think I see what the issue is - for the 82 Division all of those Areas have Y in Column BD Assume Area of Operations to Current Date? (Y/N) --- so I'd expect them to continue to be displayed on any date after 2012-02-16. When there is N in that column then I'd expect the Area would not continue to be displayed after whatever the date is.

[evz]

@tonysecurityforcemonitor Oh, OK. The way it works currently is it only looks at the current date info if there is no last cited date. I'll make that adjustment.

[evz]

@tonysecurityforcemonitor If a last cited date is null, how should that be treated? Should we ignore these areas or should we always include them? The way the current logic is setup it checks where (the first cited date is greater than the date given) AND (the last cited date is less than the given date OR assume operations to current date is true). (things in parentheses are evaluated together first kinda like the way math works). I could adjust that to be (first cited date is greater than the date given) AND (the last cited date is less than the given date OR assume operations to current date is true OR end date is null). Does that make sense?

[tonysecurityforcemonitor]

@evz Sorry dumb question - the logic string is for when an Area should be displayed right? The greater than/less than is throwing me for a loop am thinking of dates in reverse (ie 1 June 2016 is less than 3 June 2016, so if user selects 3 June 2016 and assume operations to current date is true then the Area should display)...

I think your fix makes sense. But two questions:

1) If date first cited is 1 June 2016 and the user selects 1 June 2016 the Area would display right? As it is I'd expect it to be (first cited date is greater than or equal to the date given) 2) In cases where (assume operations to current date is false) the Area should not display past the date last cited or the date first cited if there is no date last cited. Is that accounted for?

[tonysecurityforcemonitor]

@evz - do we need to worry about if the end date is null? In my mind the true/false of the assume operations to current date is the more important factor...

I'm just going to write the logic as it makes sense to me (which doesn't mean it necessarily does make sense).

Logic is assuming: 1 June 2016 is LESS THAN 3 June 2016 3 June 2016 is GREATER THAN 1 June 2016

(first cited date is less than OR equal to the date given) AND (the last cited date is greater than OR equal to the given date OR assume operations to current date is true)

So if Area has date first cited 1 June 2016, null last cited and assume operations is true and user selects:

• 3 June 2016 and Area should display

If Area has date first cited 1 June 2016, null last cited, and assume operations is false and the user selects:

• 1 June 2016 - Area should display
• 31 May 2016 - Area should not display
• 2 June 2016 - Area should not display

If Area has date first cited 1 June 2016, 4 June 2016 last cited and assume operations is false, and user selects:

• 1 June 2016 - Area should display
• 3 June 2016 - Area should display
• 4 June 2016 - Area should display
• 5 June 2016 - Area should not display

If Area has date first cited 1 June 2016, 7 June 2016 last cited and assume operations is true, and user selects:

• 10 December 2016 - Area should display

[evz]

OK, this all makes sense. I'll make the adjustments.

[evz]

OK, I made the adjustments to the various queries here to make sure the logic should work. Let me know if it's returning what you'd expect.

[tonysecurityforcemonitor]

@evz - great thanks - this is working as expected