supply IBLI_UNIT map + CSV including Kenya & Ethiopia

[niryariv]

@vrieling @apurbashee we need a shapefile of IBLI units - along with the corresponding CSV, containing the index data - which will be the final output from the data processing system

please make sure the output CSV have an "IBLI UNIT ID" column (which match the IDs in the shapefile) so we know with unit the index corresponds to

[vrieling]

@niryariv @apurbashee The shapefile that we used has been previously put on my ftp. It is called "KenyaEthiopia_IBLIunits_May2014.shp" and Apurba certainly has a copy. This file relates directly to the "division" results (CSV-files).

However, it seems wise to make an update now, if the clustering of the kebeles for Ethiopia is final. If so, we could probably put the clustered kebeles instead of the woredas in that same file. Those clustered kebeles would need to get ID-codes between 201 and 224 (and then create a new corresponding raster-file as well).

I was tempted to do this tonight (organizing the new shapefile with correct ID-codes), but am having troubles with ArcGIS. I will be off-line now for at least 3 weeks. Perhaps Apurba can do it, otherwise leave this as a task for me early August.

[bnedavies]

@vrieling I think we have figured out an interim solution, but as soon as you get back, please try to make time to do these changes.

[vrieling]

@bnedavies @niryariv I have created a new shapefile that I have put on my ftp. I just sent it by e-mail as well. Let me know if you have questions.

[niryariv]

Thanks, I put the shapefiles here: https://github.com/Gizra/IBLI/tree/master/dataProcessing/shapefiles (you can see the actual map here)

Still have a question regarding the CSV, for either @vrieling or @apurbashee: I understand we need to use this file - what should we do in order to produce a current index out of the data there?

[vrieling]

@niryariv @bnedavies @apurbashee For that file it relates to Bryn's earlier request (#143): "@apurbashee @vrieling It was mentioned during Nir's presentation yesterday that we wanted to be able to generate the czndvi graphs on the webplatform. Can you forward @niryariv what he needs to build this into the platform?"

It is the intermediate data that shows the development of the index, or rather of the zNDVI (so without cumulating). We go from this file to the czNDVI by cumulating for the LRLD or SRSD season... I am personally not sure how you want to incorporate this, but perhaps @apurbashee knows? In any case, it seems to me that the data can help to show users how the season is developing...

[apurbashee]

Hi Nir, Among the CSVs our focus will be on CZNDVIs which are cumulation for LRLD and SRSD. The CZNDVI values are actually our index. We will do two things with this index; 1) The time series seasonal CZNDVIs will be used as input in the Matlab where we will get the premiums and payouts as outputs 2) We have to generate color index maps on the web platform using these CZNDVI indexes. To generate the color index maps we have to write simple codes which will say; if a specific seasonal CZNDVI falls below 15 percentile (of the historical time series) put black color if CZNDVI is between 15 and 30 percentile put red color if CZNDVI is between 30 and 45 percentile put orange color if CZNDVI is between 45 and 65 percentile put yellow color if CZNDVI is above 65 precentile put green color... As Anton mentioned they will capture how a season is developing. With index update in every 10 days.. these color maps should be generated automatically.

On Mon, Jul 28, 2014 at 11:10 PM, vrieling notifications@github.com wrote:

@niryariv https://github.com/niryariv @bnedavies https://github.com/bnedavies @apurbashee https://github.com/apurbashee For that file it relates to Bryn's earlier request (#143 https://github.com/Gizra/IBLI/issues/143): "@apurbashee https://github.com/apurbashee @vrieling https://github.com/vrieling It was mentioned during Nir's presentation yesterday that we wanted to be able to generate the czndvi graphs on the webplatform. Can you forward @niryariv https://github.com/niryariv what he needs to build this into the platform?"

It is the intermediate data that shows the development of the index, or rather of the zNDVI (so without cumulating). We go from this file to the czNDVI by cumulating for the LRLD or SRSD season... I am personally not sure how you want to incorporate this, but perhaps @apurbashee https://github.com/apurbashee knows? In any case, it seems to me that the data can help to show users how the season is developing...

— Reply to this email directly or view it on GitHub https://github.com/Gizra/IBLI/issues/151#issuecomment-50392061.

[niryariv]

@apurbashee let me try to create a "for dummies" version of this (math is hard!):

1. I should use this file to calculate the percentiles: https://github.com/Gizra/IBLI/blob/master/dataProcessing/zscore_results/eMODIS/zCumNDVI_aggregated_eMODIS_ibli.csv
2. Then get the values from the latest dekade on this file: https://github.com/Gizra/IBLI/blob/master/dataProcessing/zscore_results/eMODIS/zNDVI_aggregated_eMODIS_ibli.csv - and according to the value's percentile, color the homepage map.

Right?

[vrieling]

@niryariv @apurbashee @mbrv I think that there is some confusion here. Also the zCumNDVI is updated at every step (using CUMULATE_Z_PER_DIVISION). Only in that case, we need to update the number of time periods that are available for the current season: I do that manually for each update, so perhaps this still needs to be implemented in the automatic chain? In any case, the zCumNDVI is the file that you want to use for making your maps. It is basically the accumulation of all aggregated z-scores for the (available) dekads of the season. The zNDVI is just an intermediate product (from the IDL-program: AGGREGATE_Z.pro). In our meeting in Nairobi I thought that there was a request to show online besides the maps, also a temporal version of the index, when clicking on a division. For map making, I think that you can forget about zNDVI (unless if this updating for available dekads still needs implementation)! But for making temporal graphs, zNDVI may be useful...

[niryariv]

OK, so zNDVI is for the temporal graphs, zCumNDVI for the map.

But I notice zCumNDVI only lists values until 2013?

[vrieling]

That is where the updating comes in. Manually I adapt this in CUMULATE_Z_PER_DIVISION by changing the "nPeriodsLong" or "nPeriodsShort" to the amount of periods that we have available. If I leave them at the full-season values (21 and 15 respectively), the cumulation is only done for seasons that are fully finalized (this explains that 2013S is now the latest). The within-season updates are basically the cumulation of zNDVI for that season up until the most recent dekad. Maybe I have overlooked this updating issue, or @mbrv did you implement it somehow? Basically you can also start yourself from zNDVI, and do the cumulation... (it is a simple procedure)

[niryariv]

Got it. @mbrv is online only sporadically these days, will reply when possible.

Regarding the percentiles calculation, I understand we should use the data in zCumNDVI. Should we use all the data, or one row at a time? When data is updated, do we recalculate? Do we use long/short season together or separately?

@apurbashee - do you want to separate the index calculation into its own script, considering this is going to be a pretty important calculation which you might want to tweak later?

[niryariv]

@vrieling do we need to implement the 2014 data for calculating the percentiles, or can we just use seasons where we have the full data (ie up to 2013)?

[apurbashee]

Hi Nir, When we are to provide index update for present ongoing season in 2014 we have to implement 2014 data for calculating the percentiles, and then determine color for that season according to the color scheme. Apurba

On Sun, Aug 24, 2014 at 12:14 PM, Nir Yariv notifications@github.com wrote:

@vrieling https://github.com/vrieling do we need to implement the 2014 data for calculating the percentiles, or can we just use seasons where we have the full data (ie up to 2013)?

— Reply to this email directly or view it on GitHub https://github.com/Gizra/IBLI/issues/151#issuecomment-53185105.

[niryariv]

OK. @mbrv - in order not to delay, please do the percentiles CSV first (so we'll have a sample to work with) and then work on integrating all the historical data.