04/08 Eytan Bakshy

[ehuppert]

Comment below with questions or thoughts about the reading for this week's workshop.

Please make your comments by Wednesday 11:59 PM, and upvote at least five of your peers' comments on Thursday prior to the workshop. You need to use 'thumbs-up' for your reactions to count towards 'top comments,' but you can use other emojis on top of the thumbs up.

[j2401]

Hi Dr. Bakshy. Thank you for visiting out workshop! I find your discussion on Pareto Frontier of different metrics particularly interesting! I wonder if the technique you mentioned produce an explicit, closed form trade-off relationships between metrics of interest or not. If so, it seems that researchers still have to manually determine the direction from which they incorporate other theories into the A/B test results (but knowing more is always better than none). Could you please elaborate more on this? Thank you!

[william-wei-zhu]

Thank you very much for sharing with us your work. I look forward to your presentation tomorrow.

[JuneZzj]

Thank you for presenting in advance. Could you explain more specifically why applying BayesOpt on the top of casual results can drive the actual personalized policy as you mentioned in the policy implementation for Heterogenous Treatment Effects? Thank you.

[Raychanan]

I have no specific questions. Looking forward to your presentation!

[YijingZhang-98]

Thanks for your sharing. It just reminds me of one paper that discussed in Autumn 2020 implies that A/B testing may discourage innovation in the company. I was wondering what kind of specific decision would be more insightful with AB testing, and without? How do you think about it? Thank you!

[Jasmine97Huang]

Thank you for sharing your work! It's really interesting to hear how bayesian optimization is useful in the context of A/B testing. Looking forward to your presentation!

[boyafu]

Thank you for your presentation! I am also curious about the potential limits of A/B testing in the context of big data.

[luyingjiang]

Thank you for sharing your work. Could you elaborate more on how the A/B test could be applied in a social science field like economics?

[Qlei23]

Thank you for sharing you work with us! It's a very interesting paper to read. It reminds me of the Dr. Berk's talk in the fall quarter as well, and I hope you can answer @afchao's question. Thank you! Looking forward to your presentation!

[luckycindyyx]

Thank you for sharing such interesting work! I was wondering how A/B test could be applied to economic research. Thank you!

[FrederickZhengHe]

Thanks for this interesting work! Look forward to your presentation tomorrow!

[Leahjl]

Thank you so much for visiting our workshop, Dr. Bakshy! I'm interested in learning more about the application of Bayes Optimization to A/B testing.

[YileC928]

Hi Dr. Bakshy, thanks for coming to our workshop! I enjoyed reading your work, the papers are really clear and concise. I have a small question about noise - can you expand a little bit more on how to appropriately handle noise when applying BeyesOpt in A/B tests. Thanks!

[bjcliang-uchi]

Thanks for the presentation! Irrelevant as it may be, I am very curious about the experience of doing research in the industry, regarding publication, data privacy, etc.

[yongfeilu]

Thank you very much for your presentation! I enjoyed reading your work so much! I am very curious how you come up with the idea? How do you get the data and apply methods without invading people's privacy. Thank you !

[luxin-tian]

Thank you very much for your presentation. This paper presents interesting case studies on A/B Test iterations with BayesOpt. I explored a little bit into the practical side and was able to find a tutorial here.

[Dxu1]

Thank you for sharing. I am curious the effect of the restriction of parameters on BayesOPT (you mentioned about 20) on the scope of causal inference from AB testing.

[goldengua]

Thanks for your sharing. I was wondering what is the benefit of A/B test with respect to the research of human cognition?

[romanticmonkey]

Thank you for your presentation Dr. Bakshy! Could you please share your thoughts on how the A/B testing and the Bayesian model implementation changes the industry?

[zixu12]

Thanks for sharing your work! I am wondering is A/B test the only type of test that is used in the internet-based experiment? What are the trade-offs that may occur during the A/B test? Thanks!

[xzmerry]

Thanks for sharing Dr. Bakshy! It's really interesting to see how BayesOpt could optimize black-box functions with time- or resource-consuming evaluations, and be applied in internet-based field experiments.

I wonder how the setting of experiments is related to the application/specific setting of BayesOpt here. Besides, how BayesOpt could help the specification in A/B test settings specifically. Could you explain more about that? Thanks

[chun-hu]

Thanks for sharing! Do you think A/B testing reinforces the bias in experiments?

[YanjieZhou]

Thanks very much for your presentation! A/B testing is really a good form of testing and I expect to see a more flexible extension of it to help research!

[k-partha]

Thanks for presenting! Do you see alternate forms of analysis than iterative A/B testing as a future framework for R&D?

[ziwnchen]

Thanks a lot for the presentation! It's really interesting to learn about the potential of BayesOpt. What kind of potential problems do you foresee if this methodological framework is to be applied in other systems?

[fyzh-git]

Thank you Dr. Bakshy for bringing us the immediately useful experience from industry practice! The example experiments, in which the BayesOpt techniques were applied to the A/B tests, provide a nice guidance for us when applying these methods in our own work. Look forward to your presentation in our workshop!

[heathercchen]

Thank you for your presentation! I am wondering do you think bootstrapping within the given sample help alleviate this issue? Thanks!

[hihowme]

Thank you for your presentation! I am wondering what is the key takeaway that the business settings could take from this? Thanks a lot!

[yiq029]

Thank you for your presentation! I also tried to use A/B testing when designing a study about advertisement. I tried to find the most impressive image to viewers via A/B testing. Could you talk more about the use of A/B testing? Thank you.

[wu-yt]

Thanks for presenting this very interesting paper! Could you talk more about personalized policy search?

[siruizhou]

Thank you for sharing! I am interested in learning more about dealing with the duration problem in A/B testing.

[timqzhang]

Thank you for the paper. I have the similar question on how A/B test could be applied to social science fields?

[chuqingzhao]

Thanks for sharing your work and I look forward to hearing more about the A/B testing.

[cytwill]

Thanks for this presentation. A/B testing is now quite popular in SE, I am wondering what are some typical ways to do the A/B testing among online users, and are they notified of being tested or in a testing session? Also, the Bayesian approach you mentioned provides the potentials to analyze continuous treatment, how this is compared to some traditional causal inference methods or if there are any possibilities to combine it with traditional causal inference methods like propensity scores, etc?

[chentian418]

Thanks for sharing with us your research, Dr. Bakshy! I was wondering how realiable for the assumption and underlying model to work for Bayesian Optimization in industries, and how would you account for the bias?

Thank you and look forward to hearing you tomorrow!

[97seshu]

Thank you for sharing. I would like to learn more about A/B testing and its applications in social sciences. Looking forward to your presentation.

[Yiqing-Zh]

Thank you so much for your presentation. I have a similar question about the practical examples of implementing the A/B testing.

[minminfly68]

Thanks for the presentation.We are looking forwaed to see its application.

[wanxii]

Thanks for your sharing! Look forward to your presentation!

[rkcatipon]

Thank you for coming to our workshop! Many others have adressed the questions I had about this paper, so I'd like to ask more general questions. For example, what is it like to do research at Facebook? As a research director do you get to see the research agenda for your team or is it a bottom up approach with team members pitching ideas? Finally, I'd like to also plus one @FranciscoRMendes questions around Bandit to BayesOpt. Always interested in learning more about the selection process.

[Panyw97]

Thanks for sharing! A/B test is a useful tool in application development. Could you elaborate on how could we apply Bayesian Optimization to the application development?

[lyl010]

Thanks for coming! Nice to hear about the powerful tool and I am curious about the interpretability of BayesOpt. How does it deal with general interpretability and personalization? Thank you!

[egemenpamukcu]

Thanks for coming to the workshop, looking forward to your talk. I would like to hear more about concerns related to novelty since the proliferation of A/B testing in the industry and it's applications beyond UX design.

[bakerwho]

Thanks for speaking at our workshop! How do experiments like these fit into the UI/UX design process at the company? Can you share any experiences communicating findings to designers or developers?

[Tanzi11]

I look forward to your presentation! Since I am not familiar with this area, I would love to learn more about the limits and advantages of using BayesOpt on A/B testing. Thanks!

[skanthan95]

Thanks for presenting! I'm interested in learning more about the overlap between your work and psychology research -- A/B testing is referenced through other names, and I'm wondering how this 'language' gap is bridged in collaboration.