Open stephens999 opened 9 years ago
Two further thoughts: i) some scoring functions might be computationally intensive, and this is a naively parallel task so it would make sense to allow it to be submitted as jobs through BatchJobs. ii) I guess you might want to allow the score to depend on design variables as well. Does it help to think of a job as consisting of (static, dynamic, res, prob_designvars, algo_designvars) and the goal is to compute a score for each job, which is a function of these variables?
@stephens999 : Are you wanting to have access to static
and dynamic
in reduceResultsExperiments
?
I would second this request, as computing assessment statistics often require accessing to the problem data used by each job. Although these data could be stored within the result object, it saves space not to do so and re-generate it on the fly only if needed in the reduce step.
This could be provided as optional arguments to the reduce function fun
, e.g., .static
and .dynamic
.
For backward compatibility we need to add the argument as trailing arguments after the ...
.
So fun
could have either one of the following signatures:
I can send a pull request for this.
Yes! That looks very helpful,and I will try it out when I get a chance. Thanks @renozao!
As requested by Berndt, I'm following up here on my post to the google group requesting a feature enhancement. I've edited the prostate cancer example from the wiki to show what I am suggestion. Here is the .rmd code:
Set-up
this code is taken directly from the example on the wiki
Set up problems and algorithms
Modified Code
Now I have modified the algorithm functions to return not the score on the test set, but the predictor function. Note that I want to require predictor functions to have the same signature, so the returned function (pred) in each case is a wrapper to the relevant method (predict.lm or predict.glmnet).
In addition, to allow for scoring via the reduce function below, these algorithms return the "dynamic" parameter that they were provided. This is the behaviour I want to change! I'm just doing this here to give a working example.
Now back to the standard code (repl reduced to 5 in final line for convenience)
Submit jobs to the cluster
still the original code....
Analysis
Here again I have modified the code. I score the output using reduceResultsExperiments to run the reduce_score function. Note I can do this here easily because the res parameter returned from the methods includes the dynamic parameter (which includes the test data needed for scoring)
This all seems to work. So what do I want to do? I want to be able to do the same thing, but without the algorithms passing their input (dynamic) as output. I want the passing of the problem outputs (static,dynamic) and the algorithm output (res) to the scoring function to be automatically handled by a new function, scoreResultsExperiments(reg, ids=findDone(reg), fun=eg_score) say. [Just as passing of res to reduce_score is handled by reduceResultsExperiments]
To be explicit, here would be the new algorithm definitions under my proposal- note that these no longer return dynamic as part of the result.
And here is the modified scoring function - note it is now supplied the static and dynamic values. (I don't use the static values in this example)
But of course this does not work :)
Thanks for any help! I'd be happy to speak by skype if it would help.