Applying Functions Across a List

Hi all,

I've written a function that fits a model to a dataframe and outputs a summary of the model. It works fine when applied to a single dataframe or when applied to a list of dataframes when using either lapply or plyr::llply so long as the function only needs to act on the whole dataframe. However I'm now wanting to apply it to subsets within each dataframe in a list and am having issues with this.

A structure of one of the dataframes in the list (called tdHRSData) is as follows, there are 192 dataframes within the list:

> str(tdHRSData$`Baccha elongata`)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   3364 obs. of  8 variables:
 $ species: Factor w/ 192 levels "Baccha elongata",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ Year   : Factor w/ 38 levels "1981","1982",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ Month  : chr  "04" "04" "04" "05" ...
 $ Day    : chr  "17" "18" "28" "01" ...
 $ count  : num  1 1 1 1 1 1 1 1 1 1 ...
 $ JDay   : num  107 108 118 121 124 125 127 130 131 133 ...
 $ cumN   : num  1 2 3 4 5 6 7 8 9 10 ...
 $ propMax: num  0.00952 0.01905 0.02857 0.0381 0.04762 ...

The list of dataframes is already separated by the factor 'species'. I am wanting to apply the function to each level of 'Year' within each dataframe, rather than to each dataframe as whole. Therefore I am ultimately wanting to fit 7,296 (192*38) models.

I have tried doing this by nesting the function within the by() function and then using llply to apply it across the dataframe list. My hope was that it would then subset each dataframe in the list by Year and then apply the function to each subset of Year.

Example code is as follows:

modelfit<-function(x){
  by(x, x$Year, FUN = modelfunction)
}

nlsmodel<-plyr::llply(tdHRSData, .fun=modelfit)

I have also attempted to do this using tapply. The code appears to run and return something, I don't get any error messages (or, at least, only errors relating to the model fit). However the output is not what I was expecting.

I would anticipate this returning nlsmodel as a list of lists. First a list of species, then within each species a list of Year, with each Year being a dataframe of the model summary. Instead it outputs a list of species, then within each species a single model summary and the empty character "Year".

Is this a result of my not specifying what I want the function to return or an underlying issue with my whole approach?

Any help much appreciated, I can provide more code/context if needed

Hey John,

There are a few things I am unclear on here, but I'll attempt to offer some ideas, until someone with a better idea comes along.

It sounds like you have used the split function to split by species? As a quick suggestion, do you know you can subset by more than one factor in split, by etc? Ie you can split on both species and year at the same time. Eg split(data, list(data$species, data$year)) should give you a list of 7296 dataframes, split as you describe. You might then be able to *apply your function to all the dataframes.

If you specifically want to get a nested list, it should be easy to rearrange into a nested structure (or using by should (I think) produce the nesting you describe, if given two factors). Otherwise you might have to check out rapply, or if you are not specifically sticking to base R, the tidyverse has some options too. (Actually I guess plyr has this kind of functionality too? - I'm not so familiar)

If your problem is specifically about the modelfit function, I would say it depends on what modelfunction in your modelfit object is. If you are trying to pass model parameters through modelfit into a model function, it might be that you are running into issues with non-standard evaluation, though I would probably expect an error message for that.

If that doesn't help, I would say it's difficult to try anything with only a description of your data and approach. Could you post a subset of your data, maybe 3 obs of 3 species over 3 years or something? Also maybe post some of your actual code, (specifically, modelfunction) then we could maybe see if there is any problem there.

Thanks, apologies if my question was unclear, think the confusion probably reflected my own lack of clarity over the problem!

My issue was with the structure of the dataset I was trying to apply the function to rather than the function itself. I had indeed used the split function and hadn't thought to go back to that point to further divide the data, however I've tried that now and it works well.

Thanks!

StirlingCodingClub / studyGroup

Applying Functions Across a List #18