mstefanro
commented
10 years ago @chingchia
Cut-off rate 0.2 means picking only those for which at least 80% of the series is known prior to interpolation?
Important: This may be confusing, but the "region" column in the daily and weekly datasets really means city, NOT region. To get the region, you need to join the datasets with the /data/india/csv_daily/agmarknet.nic.in/regions.csv file. I suggest we replace the column name now to avoid future confusion. Since our prediction model is per-region rather than per-city, maybe you should base your stats on per-region instead. When you are saying "35-40 cities" it is not very informative, because they may all be from the same region. And we are going to merge them using PCA (or averages?) in the end so we would really have one city, if it is indeed the case that they are all in the same region.
Besides, it seems that we do not have to worry about the subproduct dimension except for rice.
Our most important data-set is the daily one, not the weekly one. On the daily one, rice has 100 subproducts, onion has 26 subproducts, wheat has 68 subproducts etc. So we do have to worry about both city and subproducts. What we would like to do is the following:
let D be a mapping from all (R, P) to a time series
for each region R:
| for each product P:
| | let M be a matrix.
| | for each subproduct SP (of product P):
| | | for each city C (of region R):
| | | | let T be the time-series corresponding to (R,P,SP,C)
| | | | interpolate T to obtain a full time-series
| | | | add the vector T as a column to matrix M
| | let T = PCA(M, 1)
| | store a mapping from (R, P) to T into D
Try to make your code in such a way that it works both on the daily and weekly datasets. The only differences between the datasets are the date-range you have to pick and the gaps between dates for interpolation (1 week vs. 1 day). I can provide help with implementing this after we meet. We first need to go over your code.
One extra difficulty for the weekly dataset is that you might have to account for prices reported on the same week, but different day (I don't know if this occurs in the data, you should check). If that is the case, then you should really interpolate on week-of-the-year index rather than date index.
halccw
f4bD3v
For the India weekly dataset, we may select our target products based on previous statistics and this table:
https://github.com/fabbrix/humanitas/blob/master/analysis/ts/na_table_org.csv