ycroissant / plm

Panel Data Econometrics with R
GNU General Public License v2.0
49 stars 13 forks source link

Empty model problem and how to see the time variation in plm package? #29

Closed GokuuQ closed 2 years ago

GokuuQ commented 2 years ago

Hi,

In my input data, I have 32 time periods and thousands of individual study areas. But the independent variables are 32-times repeated same data. So only the dependent variable varies over time. In my case, I have two questions when I apply the plm package:

  1. When I set model="random" or model="within", I got this error. But when I set model="between", it runs well. Why it's like this? Error in plm.fit(data, model = models[1L], effect = effect) : empty model
  2. Is there any way to see the time variation through plm package, or should I use some other methods? I noticed the pvar() function to check if time variation exist, but is there any possibility to see it in more detail?

Thanks!

tappek commented 2 years ago

ad 1) As you describe your data, they does not vary over time for the dependent variables. Computationally, this results in 0 values for the within transformation, hence model = "within" does not have anything to work with and stops rightfully with error message "empty model". The random effect estimator as per Swamy-Arora uses the variation of the associated within model and the associated between model. For a plm-based exposition see one of the package's vignettes https://cran.rstudio.com/web/packages/plm/vignettes/B_plmFunction.html, section "Unbalanced panels" (but this is not specific to unbalances panels). Any good text book about panel models will cover this, e.g., Wooldridge or Baltagi. Other random effect estimators like Wallace-Hussain use slightly other "base models" (but Amemiya's estimator uses the within model twice), see Baltagi's text book for an overview.

So, setting for the RE case, e.g., model = "random", random.method = "walhus" should work as this does not entail estimating a within model.

ad 2) It is unclear to me what more there could be to see if there is no time variation. Could you explain further what you expect to see? You could run the within transformation on the affected variables and see that they will be zero after transformation.

GokuuQ commented 1 year ago

Thank you so much for your explanation in detail! They are very helpful for our case. Now I have 2 additional questions:

  1. When I get the model by simply setting model="between", and model = "random", random.method = "walhus" with different settings of effects ("individual", "time", "twoways"), I got 4 models and used "predict" function to predict, and they all get the same results. Why will this happen and is there any better way to predict from panel data?
  2. I'm wondering, is it common to use models from "plm" package for predicting? Or it will mostly be used in explanatory research rather than predictions?
tappek commented 1 year ago

ad 2) I guess using plm for prediction is as common as it is for lm (or in other words: as muss as you want to trust estimates of a sample applied to another data set).

ad 1) I cannot follow, please see below example showing different output. Can you supply a reproducible example? Note that effect is an argument to plm() (i.e., to model estimation), not to predict(). Also, make sure you really estimated different models, e.g., by looking at the model estimates.

library(plm)
data("Grunfeld", package = "plm")

Grunfeld.p <- pdata.frame(Grunfeld)

fit.plm.be.ind   <- plm(inv ~ value + capital, data = Grunfeld, model = "between", effect = "individual")
fit.plm.be.time  <- plm(inv ~ value + capital, data = Grunfeld, model = "between", effect = "time")

predict(fit.plm.be.ind)
#>          1          2          3          4          5          6          7 
#> 595.778486 266.416047 265.682416  88.694556  38.231217  51.346468  21.729676 
#>          8          9         10 
#>  84.551468  45.939729   1.212436
predict(fit.plm.be.time)
#>      1935      1936      1937      1938      1939      1940      1941      1942 
#>  53.20750  93.23382 127.76342  87.13161 114.46630 120.65669 120.77093 107.23899 
#>      1943      1944      1945      1946      1947      1948      1949      1950 
#> 121.13411 123.94717 136.98304 147.07920 138.93421 146.61683 157.85766 169.00788 
#>      1951      1952      1953      1954 
#> 197.68956 217.66233 260.43763 277.34613

fit.plm.re.ind   <- plm(inv ~ value + capital, data = Grunfeld, model = "random", effect = "individual", random.method = "walhus")
fit.plm.re.time  <- plm(inv ~ value + capital, data = Grunfeld, model = "random", effect = "time", random.method = "walhus")
fit.plm.re.tw    <- plm(inv ~ value + capital, data = Grunfeld, model = "random", effect = "twoways", random.method = "walhus")

predict(fit.plm.re.ind)
#>            1            2            3            4            5            6 
#>  281.0501699  470.0508556  581.6938615  313.0820684  478.1687785  515.6180201 
#>            7            8            9           10           11           12 
#>  520.2018170  391.7070226  468.3565339  484.8673612  554.9971768  603.7515021 
#>           13           14           15           16           17           18 
#>  563.4050163  583.0422016  661.9486060  692.2783636  843.8918665  922.4571609 
#>           19           20           21           22           23           24 
#> 1173.5210595 1240.4286596  108.4522673  156.2261367  272.3648713  220.1119554 
#>           25           26           27           28           29           30 
#>  253.2980787  262.2615718  283.9592265  272.1766457  252.9976513  227.2378471 
#>           31           32           33           34           35           36 
#>  211.1488162  210.0520596  220.9553815  215.1463209  233.2523159  236.4527091 
#>           37           38           39           40           41           42 
#>  298.7806584  315.8902168  356.9792005  380.3867520  100.9342704  195.6901464 
#>           43           44           45           46           47           48 
#>  286.2673514  214.2341938  243.0274222  233.7265708  211.5648341  205.1284268 
#>           49           50           51           52           53           54 
#>  232.7023842  226.3087224  260.9483616  291.0709344  264.4887737  285.4924528 
#>           55           56           57           58           59           60 
#>  289.5787494  318.1285746  348.3933086  393.7950102  448.6266138  518.4604820 
#>           61           62           63           64           65           66 
#>   -8.5223561   37.4967019   50.0850113    6.4102787   36.7804212   42.9181372 
#>           67           68           69           70           71           72 
#>   36.1702525    9.4726276   27.6245111   37.6639843   52.0876135   66.5705778 
#>           73           74           75           76           77           78 
#>   35.7222392   52.5235691   52.5150872   68.6937058   93.7524233  111.5284417 
#>           79           80           81           82           83           84 
#>  158.7031924  147.1239141   16.0583660   23.5707895   36.1495145   49.2987268 
#>           85           86           87           88           89           90 
#>   62.7572181   68.5340420   77.3657203   88.3989157   90.1600626  107.5045197 
#>           91           92           93           94           95           96 
#>  119.4420184  132.3197843  143.9220870  150.0558146  162.5563061  172.3613495 
#>           97           98           99          100          101          102 
#>  188.5673710  206.0438503  224.1543816  229.9724946  -33.9429893  -29.6252638 
#>          103          104          105          106          107          108 
#>  -24.5632213  -21.7305675  -11.1944395   -8.7230409   -8.2715644   -2.9032144 
#>          109          110          111          112          113          114 
#>    2.9929967    7.1462987    6.4067953   15.4933598   21.3835984   26.4990599 
#>          115          116          117          118          119          120 
#>   41.2394062   66.9012602   71.1757486   80.9376046   94.5109039  117.5507228 
#>          121          122          123          124          125          126 
#>  -11.6149644    2.8209233    9.2663333   10.8788842   20.0627451   26.8787602 
#>          127          128          129          130          131          132 
#>   32.0690710   38.1361404   46.6870156   53.2202865   59.6636133   67.5874755 
#>          133          134          135          136          137          138 
#>   70.3804545   76.3011532   82.2440713   89.9104591  100.5869852  105.4984409 
#>          139          140          141          142          143          144 
#>  112.3852380  120.7476147  -35.9910538   -0.6974114   24.6995662    9.4912982 
#>          145          146          147          148          149          150 
#>    6.7078472   19.5445156   12.4985145   22.7039332   36.1017388   39.2341301 
#>          151          152          153          154          155          156 
#>   51.7259751   52.3150337   40.3809913   55.2503521   50.0806758   54.1521819 
#>          157          158          159          160          161          162 
#>   61.7209036   81.9697771  127.1144304  138.5051337   24.1225474   27.8658898 
#>          163          164          165          166          167          168 
#>   35.4485405   30.2949261   45.4587486   42.7845887   43.7959578   42.0760933 
#>          169          170          171          172          173          174 
#>   64.8677449   70.1792060   82.0561544   73.3864465   76.8261125   85.3992996 
#>          175          176          177          178          179          180 
#>   86.5532923   88.1452020   99.9198983  121.7894281  133.0291319  138.3547071 
#>          181          182          183          184          185          186 
#>  -48.3911182  -46.4582020  -47.1309719  -49.7100453  -47.3714922  -46.7731633 
#>          187          188          189          190          191          192 
#>  -47.7651811  -49.5570245  -49.6250912  -49.7234231  -49.3120277  -48.9317863 
#>          193          194          195          196          197          198 
#>  -49.2272222  -48.4398864  -47.4666032  -47.2919141  -45.6556326  -46.3369944 
#>          199          200 
#>  -46.9960302  -46.7728282
predict(fit.plm.re.time)
#>           1           2           3           4           5           6 
#>  313.689629  508.135423  616.023978  328.216223  502.648328  541.741311 
#>           7           8           9          10          11          12 
#>  542.101267  402.237879  486.662133  509.871765  577.840273  616.423091 
#>          13          14          15          16          17          18 
#>  540.477244  546.183619  620.203848  644.806524  794.387943  856.403273 
#>          19          20          21          22          23          24 
#> 1088.574820 1117.253628  127.138015  177.767267  293.807759  225.539623 
#>          25          26          27          28          29          30 
#>  255.608603  270.495977  292.680701  276.797387  256.306835  231.286192 
#>          31          32          33          34          35          36 
#>  220.417793  226.821469  225.999821  215.961813  230.918963  233.666355 
#>          37          38          39          40          41          42 
#>  300.780299  309.297936  335.878144  356.242756  115.123047  214.318660 
#>          43          44          45          46          47          48 
#>  308.461085  229.029741  257.832075  246.731866  220.195060  207.187604 
#>          49          50          51          52          53          54 
#>  233.244115  226.379099  263.024617  292.296298  254.019117  268.044245 
#>          55          56          57          58          59          60 
#>  265.376036  292.739737  322.393887  365.115898  415.964835  481.275735 
#>          61          62          63          64          65          66 
#>    7.954955   56.456526   67.435564   19.839445   50.665855   56.870295 
#>          67          68          69          70          71          72 
#>   49.008457   21.240565   40.760930   51.950289   67.692485   80.136622 
#>          73          74          75          76          77          78 
#>   46.525797   62.975874   59.503981   75.074715   97.718488  108.334487 
#>          79          80          81          82          83          84 
#>  152.858955  134.257444   17.770082   23.746928   34.017694   42.683136 
#>          85          86          87          88          89          90 
#>   53.936447   58.075811   65.172317   73.116940   73.166072   87.395537 
#>          91          92          93          94          95          96 
#>   97.251527  107.520777  116.763039  120.552239  130.358928  137.810467 
#>          97          98          99         100         101         102 
#>  152.870143  167.052491  181.939947  185.219827  -18.449214  -14.766928 
#>         103         104         105         106         107         108 
#>  -10.542658   -8.629453    1.593607    3.833774    3.471519    7.357493 
#>         109         110         111         112         113         114 
#>   12.274244   16.816639   16.065619   25.459975   30.063237   33.962104 
#>         115         116         117         118         119         120 
#>   47.450001   73.074955   76.385882   84.545962   97.772702  119.509373 
#>         121         122         123         124         125         126 
#>   -3.652807    9.244429   14.413856   13.999269   20.908201   25.471935 
#>         127         128         129         130         131         132 
#>   28.220852   32.358745   39.131543   44.346973   49.458351   56.408506 
#>         133         134         135         136         137         138 
#>   57.265235   62.030908   66.680351   72.633193   81.914984   85.571037 
#>         139         140         141         142         143         144 
#>   91.028523   97.500369  -20.168995   17.100246   43.237463   26.221944 
#>         145         146         147         148         149         150 
#>   22.787340   36.029426   27.704626   36.164365   48.079858   50.746312 
#>         151         152         153         154         155         156 
#>   63.792745   65.009001   50.101848   63.949057   57.461027   62.224462 
#>         157         158         159         160         161         162 
#>   70.848519   90.706610  135.531664  143.927336   28.237908   31.063831 
#>         163         164         165         166         167         168 
#>   38.213116   31.388262   46.429996   42.216846   42.258167   39.143304 
#>         169         170         171         172         173         174 
#>   60.730279   65.580323   74.925700   67.096827   67.834161   74.444281 
#>         175         176         177         178         179         180 
#>   74.635832   75.754110   86.760497  107.655437  116.795031  120.077406 
#>         181         182         183         184         185         186 
#>  -33.481804  -31.465338  -32.160959  -34.876666  -32.396622  -31.750553 
#>         187         188         189         190         191         192 
#>  -32.787106  -34.647527  -34.695952  -34.783703  -34.341056  -33.933086 
#>         193         194         195         196         197         198 
#>  -34.306650  -33.615093  -32.777007  -32.718673  -31.025834  -31.823596 
#>         199         200 
#>  -32.680692  -32.692274
predict(fit.plm.re.tw)
#>            1            2            3            4            5            6 
#>  281.0602706  470.0456397  581.6744941  313.0760804  478.1527720  515.5993923 
#>            7            8            9           10           11           12 
#>  520.1796284  391.6896338  468.3370091  484.8510548  554.9721253  603.7140237 
#>           13           14           15           16           17           18 
#>  563.3455931  582.9705728  661.8653405  692.1878070  843.7843371  922.3294896 
#>           19           20           21           22           23           24 
#> 1173.3539131 1240.2266889  108.4697795  156.2408602  272.3676611  220.1083558 
#>           25           26           27           28           29           30 
#>  253.2888073  262.2557223  283.9515176  272.1671377  252.9891418  227.2325098 
#>           31           32           33           34           35           36 
#>  211.1489446  210.0577918  220.9514152  215.1398515  233.2416917  236.4414264 
#>           37           38           39           40           41           42 
#>  298.7665141  315.8680340  356.9421980  380.3451308  100.9492578  195.6987067 
#>           43           44           45           46           47           48 
#>  286.2692710  214.2380537  243.0283479  233.7271290  211.5644523  205.1238912 
#>           49           50           51           52           53           54 
#>  232.6939213  226.3005674  260.9381374  291.0570105  264.4690018  285.4654262 
#>           55           56           57           58           59           60 
#>  289.5463598  318.0924006  348.3535964  393.7486989  448.5717863  518.3952103 
#>           61           62           63           64           65           66 
#>   -8.4945138   37.5216617   50.1075072    6.4343641   36.8017388   42.9388768 
#>           67           68           69           70           71           72 
#>   36.1908657    9.4951838   27.6462154   37.6855052   52.1086266   66.5886190 
#>           73           74           75           76           77           78 
#>   35.7414084   52.5407651   52.5297481   68.7062693   93.7606592  111.5296196 
#>           79           80           81           82           83           84 
#>  158.6976116  147.1143743   16.0728866   23.5834185   36.1591689   49.3037552 
#>           85           86           87           88           89           90 
#>   62.7592573   68.5342922   77.3637981   88.3936052   90.1533186  107.4937235 
#>           91           92           93           94           95           96 
#>  119.4284790  132.3030197  143.9024094  150.0337938  162.5310363  172.3333551 
#>           97           98           99          100          101          102 
#>  188.5368820  206.0091644  224.1154861  229.9311464  -33.9132708  -29.5964516 
#>          103          104          105          106          107          108 
#>  -24.5355395  -21.7038482  -11.1690255   -8.6980486   -8.2472141   -2.8804977 
#>          109          110          111          112          113          114 
#>    3.0143940    7.1675567    6.4281204   15.5139822   21.4026769   26.5167252 
#>          115          116          117          118          119          120 
#>   41.2546490   66.9138549   71.1872013   80.9468874   94.5185467  117.5550583 
#>          121          122          123          124          125          126 
#>  -11.5930409    2.8402457    9.2840633   10.8949652   20.0762224   26.8898923 
#>          127          128          129          130          131          132 
#>   32.0778854   38.1429226   46.6916226   53.2232612   59.6649547   67.5872947 
#>          133          134          135          136          137          138 
#>   70.3785708   76.2978190   82.2391831   89.9035332  100.5779477  105.4879826 
#>          139          140          141          142          143          144 
#>  112.3730297  120.7331682  -35.9608858   -0.6694015   24.7255241    9.5174861 
#>          145          146          147          148          149          150 
#>    6.7338427   19.5694969   12.5232791   22.7263773   36.1217292   39.2534594 
#>          151          152          153          154          155          156 
#>   51.7444346   52.3338923   40.3988919   55.2659856   50.0958720   54.1674690 
#>          157          158          159          160          161          162 
#>   61.7361903   81.9827096  127.1225180  138.5098650   24.1380043   27.8802926 
#>          163          164          165          166          167          168 
#>   35.4618516   30.3075399   45.4697242   42.7947106   43.8052662   42.0845559 
#>          169          170          171          172          173          174 
#>   64.8729976   70.1835784   82.0574602   73.3892534   76.8265895   85.3974637 
#>          175          176          177          178          179          180 
#>   86.5506337   88.1420340   99.9149652  121.7815478  133.0185660  138.3421012 
#>          181          182          183          184          185          186 
#>  -48.3603520  -46.4275721  -47.1002900  -49.6791999  -47.3407821  -46.7424793 
#>          187          188          189          190          191          192 
#>  -47.7344285  -49.5261391  -49.5941844  -49.6924986  -49.2811223  -48.9008994 
#>          193          194          195          196          197          198 
#>  -49.1963623  -48.4091770  -47.4360922  -47.2615062  -45.6253504  -46.3067279 
#>          199          200 
#>  -46.9658414  -46.7428341
GokuuQ commented 1 year ago

Thank you for your answer and example. In my case, the independent variables are time-invariant, only the dependent variable is time variant. Could this be the reason that I got the same estimates and prediction results for "effect=individual", "effect=individual", and "effect=twoways"? But the standard error, p values, and standard deviations are different from each other.