Question about hit_urls / prep

[yamins81]

@ardila @hahong

@ardila, in the experiment objects you create, there is a method called hit_urls which basically does three things: (1) chunks the trial list (you call this object ExperimentData, which is usually mostly created in the constructor cal into smaller chunks) (2) templates and writes out the final HTML files (3) uploads them to S3

@hahong similarly, you have a function called prep which: (1) creates the trial list (you call this object 'trials') (2) chunks the trial list (3) templates and writes out the final HTML files

In both of your cases, you allow there to be the possibility of multiple ways this prep method could be done. In @ardila 's case, this is by sometimes overriding the hit_url method. In @hahong 's case, this is by having a different prep function in each driver script.

In other words, @hahong has factored the code so that trial list creation is combined with chunking/templating/writing-out but uploading is separate. While @ardila has factored the code so that trial list creation is (mostly) separate but chunking and chunking/templating/writing-out is combined with uploading.

My question is: is the any objection to factoring the code as follows:

(1) a "create_trials" method, which creates a trial_list object, and which various between experiments (and is probably not even implemented in the base class) (2) a single standard "prep" method which does chunking/templating/writing-out of final HTML files (both "sandbox" and "production") (3) a single standard "uploadHTML" method, which actually uploads the HTMLs.

[yamins81]

I expect one issue with having a single standard "prep" method as I described above is that sometimes you want chunks to have additional structure, e.g. when segregating training and testing trials. (As in @ardila 's ReactionTimeTunedExperiment.)

Do you think it's reasonable to create a default "prep" method that basically follows the typical chunking structure (e.g. the one MatchToSampleExperiment.hit_urls or @hahong 's prep_web_simple function + last lines of driver.prep), and then allow people to override as necessary? Or there a better suggestion?

[ardila]

Yes. Talking with Arash has convinced me that scientifically there is not much value in train/test splits, since this would introduce even more subject-subject variability, so I'm not that worried about it.

On Sat, May 24, 2014 at 6:30 PM, Dan Yamins notifications@github.comwrote:

I expect one issue with having a single standard "prep" method as I described above is that sometimes you want chunks to have additional structure, e.g. when segregating training and testing trials. (As in @ardila https://github.com/ardila 's ReactionTimeTunedExperiment.)

Do you think it's reasonable to create a default "prep" method that basically follows the typical chunking structure (e.g. the one MatchToSampleExperiment.hit_urls or @hahong https://github.com/hahong's prep_web_simple function + last lines of driver.prep), and then allow people to override as necessary? Or there a better suggestion?

— Reply to this email directly or view it on GitHubhttps://github.com/dicarlolab/mturkutils/issues/7#issuecomment-44101520 .

[ardila]

I have no objection to this, and I would be happy to switch to it once it's done. What's the best way to make all old experiments in my experiments.py easily replicable, as they are now? Perhaps just keeping them in the imagenet_psychophysics repository, whereas future experiments can be done in some folder of mturkutils?

On Mon, May 26, 2014 at 1:54 PM, Diego Ardila ardila@mit.edu wrote:

Yes. Talking with Arash has convinced me that scientifically there is not much value in train/test splits, since this would introduce even more subject-subject variability, so I'm not that worried about it.

On Sat, May 24, 2014 at 6:30 PM, Dan Yamins notifications@github.comwrote:

I expect one issue with having a single standard "prep" method as I described above is that sometimes you want chunks to have additional structure, e.g. when segregating training and testing trials. (As in @ardila https://github.com/ardila 's ReactionTimeTunedExperiment.)

Do you think it's reasonable to create a default "prep" method that basically follows the typical chunking structure (e.g. the one MatchToSampleExperiment.hit_urls or @hahong https://github.com/hahong's prep_web_simple function + last lines of driver.prep), and then allow people to override as necessary? Or there a better suggestion?

— Reply to this email directly or view it on GitHubhttps://github.com/dicarlolab/mturkutils/issues/7#issuecomment-44101520 .

[yamins81]

I think that sounds reasonable.

[hahong]

No objection to the proposal. Looks good to me.