udacity / robotics-beta-test-issues

1 stars 0 forks source link

arek 867sec@gmail.com "need to define metric" in writeup template #304

Closed 867sec closed 7 years ago

867sec commented 7 years ago

arek 867sec@gmail.com "need to define metric" in writeup template There is "need to define metric" in writeup template in sentence: "Iterate on your perception and decision function until your rover does a reasonable (need to define metric) job of navigating and mapping." under "Autonomous Navigation / Mapping".

My suggestion is to make the project 1: Search and Sample Return into 2 projects: the first the way it is now and the second to collect all the yellow rocks in the autonomous mode. Clear metrics give tremendous motivation to students and are deeply satisfying fostering good learning habits and addiction to Udacity. The way the Project 1 Search and Sample Return is structured now it doesn't provide enough motivation for student to put effort into creating artificial intelligence algorithm (decision tree) to achieve anything in autonomous mode because nothing is required in the rubric. Adding it to the rubric may not work either because the project is already big therefore the idea of splitting it into 2 parts. I'm so excited about the prospect of having a project with clearly defined goal: pick up all rocks using autonomous mode! Sounds so exciting that I'm sure the students will greatly appreciate it because it will teach how hard it is to hard code all the decision, will train in thinking about all the edge cases etc, etc.

ryan-keenan commented 7 years ago

Thank you for this feedback! I agree. Originally I set out to build a simple first week project but it just kept getting more and more complicated as we made it cooler! The current plan is to have mapping of 40% of the terrain at 60% fidelity and finding at least 1 rock be the minimal passing submission. But I've now created a page called Requirements & Challenges listing tips and tricks for taking on the optional challenge of finding and picking up all the rocks and making a high quality map. Still trying to figure out what kind of prize we can offer to students who solve the challenge or come up with the best result.

ryan-keenan commented 7 years ago

and modified the writeup template accordingly.

867sec commented 7 years ago

In order to make sure that students have mapped 30% of the terrain at 60% fidelity and finding at least 1 rock being the minimal passing submission requires reviewing students code and running the simulator with students code for extended period of time. This is not a $13 price tag. It is more like 30 to 40 dollars price tag per review for the reviewer. Is it possible to increase the price? Please advise.

On Sun, May 21, 2017 at 8:06 PM, ryan-keenan notifications@github.com wrote:

Thank you for this feedback! I agree. Originally I set out to build a simple first week project but it just kept getting more and more complicated as we made it cooler! The current plan is to have mapping of 30% of the terrain at 60% fidelity and finding at least 1 rock be the minimal passing submission. But I've now created a page called Requirements & Challenges https://classroom.udacity.com/nanodegrees/nd209-beta/parts/f491093d-8e67-4eab-8102-87bae095ca79/modules/0573f9ff-2e23-48cd-87d0-af24d758348a/lessons/ef841d31-8c53-49b3-8da3-a9d1523adef0/concepts/ed33ab6f-2b67-4991-b372-ab915538b734 listing tips and tricks for taking on the optional challenge of finding and picking up all the rocks and making a high quality map. Still trying to figure out what kind of prize we can offer to students who solve the challenge or come up with the best result.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/udacity/robotics-beta-test-issues/issues/304#issuecomment-302972599, or mute the thread https://github.com/notifications/unsubscribe-auth/AaxC7MFhZFfBXZc_2vEoHa8hOCoDmQQxks5r8NF5gaJpZM4NbqaW .

ryan-keenan commented 7 years ago

How long do you think it would take you to review?

867sec commented 7 years ago

Here's my estimation (40+ minutes) of how long it will take me to review the first project based on beta testing and peer reviewing we did last week. In order to make it easy to understand my summary has 6 sections and I summarized each section with estimated time underneath each of the 6 sections:

1 of 6: Setting up and running the notebook to see if student implemented rock_thresh function and obstacle_tresh function as well as understanding how the student implemented all the analysis in the process_image function will take at least 8-10 minutes. Step 1 of 6 estimated 8-10 minutes.

2 of 6: Watching the output video 1-2 minutes. Step 2 of 6 estimated 1-2 minutes.

3 of 6: Setting up the simulator and running autonomous mode until 40% is mapped can take 2 to 4 minutes with additional 2 to 4 minutes to find the first rock therefore 4-8 minutes here. This is optimistic assuming that the rover is not stuck. If the rover is stuck then I would restart the simulator hoping that the second time the rover would get to 40% map coverage without getting stuck. So the 4-8 minutes here can easily be 15 minutes but let's count 4-8 minutes for the purposes of this estimation. Step 3 of 6 estimated 4-8 minutes.

4 of 6: Reviewing code in decision.py and perception.py in order to mentor the student and arrive at possible fixes, tips and areas to improve at least 5 minutes if the submission is perfect. If there are errors but python runs, or if python doesn't run then additional 5 minutes to find the exact place and reason and a suggestion fix so 10 minutes. (Side note: If it gets stuck every time how many times should I restart simulator to decide that the student's submission doesn't do a reasonable job mapping?) Step 4 of 6 estimated 10 minutes.

5 of 6: Write-up of 5 sections rubric review taking into consideration all the steps above assuming average 35 words per minute typing speed - estimated 6-10 minutes. Step 5 of 6 estimated 6-10 minutes.

6 of 6: In case there is a "Student Notes" section populated with a student's custom- tailored question that student has regarding their submission or the project in general - this could require additional research - estimated time varies. Step 6 of 6 estimated time 0 to varies.

Total 29-40 minutes plus time spent researching and answering questions in the section called "Student Notes" in the submission.

On Sun, May 21, 2017 at 11:56 PM, ryan-keenan notifications@github.com wrote:

How long do you think it would take you to review?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/udacity/robotics-beta-test-issues/issues/304#issuecomment-302993544, or mute the thread https://github.com/notifications/unsubscribe-auth/AaxC7AT7G9qClIj4VQ534ynxn9F1mmlLks5r8QdlgaJpZM4NbqaW .

ryan-keenan commented 7 years ago

Thanks! That helps a lot. I don't set the prices but I can communicate this to the people in the reviews department and see if we can get an adjustment

867sec commented 7 years ago

Forgot to add one more section about reading the write-up worth 2 minutes. So the total jumps from 6 to 7 parts of the review process to perform and therefore from 29-40 minutes to 31-42 minutes.

On Mon, May 22, 2017 at 12:50 AM, ryan-keenan notifications@github.com wrote:

Thanks! That helps a lot. I don't set the prices but I can communicate this to the people in the reviews department and see if we can get an adjustment

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/udacity/robotics-beta-test-issues/issues/304#issuecomment-302998700, or mute the thread https://github.com/notifications/unsubscribe-auth/AaxC7LOSpVxOoB3O7rTHZRhi2XcztWIzks5r8RQNgaJpZM4NbqaW .