djxjacob
commented
6 years ago I think this would be great to send this survey to the geos-chem email list, telling them that we invite them to use the cloud facility, that it should be very easy to use and if it's not we want to hear about it. You can also invite others, e.g. Harvard classes, with no prior GC experience. Expect self-selected responses though. From the standpoint of the BAMS paper, I think what we should do is just ask the students and postdocs in the group with prior experience of GEOS-Chem. Four reasons: (1) no credit card issue, (2) homogeneity of background, (3) we can ask them all to participate, (4) that's a large enough sample. Yes, they will be proficient in GEOS-Chem, but with no experience in running on the cloud and that's what what we want to test.
JiaweiZhuang
Daniel suggested a survey & testing among users to see how quickly they can get started with the AWS cloud. I think this is a great idea and will add a significant value to the paper.
Here's my preliminary plan:
Test specification
Users/volunteers will be asked to go through one or two pre-defined workflows, and report the time spent on each stage.
Test 1 (required). The absolutely minimum beginner demo, following the Quick start guide.
The entire process should only take 10~20 minutes if everything goes smoothly. Factors that could slow down the process include:
Test 2 (optional). A more complete, customizable workflow, resembling a realistic project. This requires the user to read the all beginner tutorials for additional stuff like S3 and spot.
s3://gcdatabucket.Those steps will be documented more explicitly, to replace the current research workflow section. The entire process should take 1\~2 hours, if the users have already read most of beginner tutorials like AWS-CLI, S3, and spot. Would it be necessary to also record the time spent on those individual chapters? They should be finished in 1\~2 hours in total, depending on how focused a user is.
Information to gather
Besides the time spent on each stage of the workflow, additional information includes:
How do users report results
We could in principle use Google Forms, but I want to users to directly reply on GitHub issue tracker. This makes the "raw data" transparently visible and ensures information credibility. All participants wound need to sign up their own GitHub accounts.
Remaining issues to discuss
Where to obtain AWS accounts? I would like to let users sign up their own accounts using their own credit cards. But not everyone might be willing to do so. We can also send out temporary user account (IAM users derived from our own root account) for testing. I should be able to get some credits for this.
Collaborate with existing classes & conferences? Students from EPS200/EPS236 seem good candidates. I also really like to collaborate with CS205 because that class teaches a lot about scientific computing on AWS, but it is not offered in the coming semester. A great chance to gather information is the workshop at ICC9, but it is two semesters ahead. We can get some initial results by inviting users online, and gradually add more data if there's a chance to perform offline user testing.
How many samples should we get? 20 doesn't seem very compelling? Maybe 50?
CC @yantosca @lizziel @msulprizio @sdeastham @ayshaw for any comments. Let's make all our discussions transparent on GitHub. This is the approach adopted by Pangeo (https://github.com/pangeo-data/pangeo/issues) and I think this transparency is quite valuable for the community.