Student gets no feedback on number of specs passed but others do

[mtc2013]

[zhangaaron]

Can you clarify what the issue here is? The number of specs here passed is 13/13, number of cucumber scenarios passed is 3/3.

[tansaku]

@zhangaaron: @rosewcs345 is saying some other students (in her SUNY Binghamton class) don't get this information. (@mtc2013 filed this on her behalf)

Ideally we need a screen shot of the other kind of output they are seeing in order to understand this better ...

[mtc2013]

@tansaku could you post the picture of the results returned from the grader when submitting this student's hw through @rosewcs345 spoc? I don't have access.

[tansaku]

I think this is it:

[rosewcs345]

rspec passes locally, but not accepted by autograder. Students with this issue used FactoryGirl

[tansaku]

so I am essentially replicating this in #93 - I get exactly the same message in the local acceptance test as I do in the grader:

Submission message after grade: Your submission was recorded at 2010-06-19 18:52:28 +0100 : submission is on time.
Score: 40.0/100.0
----------------------------------------
Running student tests found in features/ spec/:
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -S bundle exec cucumber  --profile default
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -I/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed

----------------------------------------

----------------------------------------
Checking coverage for:
  controllers >= 90.00%
  models >= 90.00%
----------------------------------------
  all files: 84.21%% coverage
  controllers: 80.43%% coverage
  models: 100.00%% coverage
  mailers: 100.00%% coverage
  helpers: 100.00%% coverage
  libraries: 100.00%% coverage

Failed coverage test (controllers coverage too low).
  Score: 0/20.0
----------------------------------------

----------------------------------------
Running reference Cucumber scenarios:
  3 out of 3 scenarios passed
  Score: 40/40
----------------------------------------
Total score: 40.0 / 100.0
Completed in 19.305187 seconds.

[tansaku]

and if I open the zipped file from the student all the specs pass and the coverage is good:

[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/features/support/hw4 (92_factorygirl_in_hw4)]$ 
→ be rspec
...................

Finished in 0.20686 seconds (files took 2.21 seconds to load)
19 examples, 0 failures

Coverage report generated for RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/features/support/hw4/coverage. 0.0 / 0.0 LOC (100.0%) covered.

[tansaku]

however if I go to the submission folder where the grader has unpackaged the student code (and done various things with it) then we get the following:

[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42 (92_factorygirl_in_hw4)]$ 
→ be rspec
...................

Finished in 0.20125 seconds (files took 1.51 seconds to load)
19 examples, 0 failures

Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42/coverage. 57 / 57 LOC (100.0%) covered.

which is very similar ...

[tansaku]

now the grader doesn't run bundle exec rspec, it runs a special rake task like so ....

[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42 (92_factorygirl_in_hw4)]$ 
→ bundle exec rake saas:run_student_tests
----BEGIN CUCUMBER----
--------------------------------------------------------------------------------
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -S bundle exec cucumber  --profile default
Using the default profile...
Feature: search for movies by director

  As a movie buff
  So that I can find movies with my favorite director
  I want to include and serach on director information in movies I enter

  Background: movies in database      # features/add_director.feature:7
    Given the following movies exist: # features/step_definitions/director_steps.rb:1
      | title        | rating | director     | release_date |
      | Star Wars    | PG     | George Lucas | 1977-05-25   |
      | Blade Runner | PG     | Ridley Scott | 1982-06-25   |
      | Alien        | R      |              | 1979-05-25   |
      | THX-1138     | R      | George Lucas | 1971-03-11   |

  Scenario: add director to existing movie                # features/add_director.feature:16
    When I go to the edit page for "Alien"                # features/step_definitions/web_steps.rb:48
    And I fill in "Director" with "Ridley Scott"          # features/step_definitions/web_steps.rb:60
    And I press "Update Movie Info"                       # features/step_definitions/web_steps.rb:52
DEPRECATION: Using `should` from rspec-expectations' old `:should` syntax without explicitly enabling the syntax is deprecated. Use the new `:expect` syntax or explicitly enable `:should` with `config.expect_with(:rspec) { |c| c.syntax = :should }` instead. Called from /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42/features/step_definitions/web_steps.rb:107:in `block in <top (required)>'.
    Then the director of "Alien" should be "Ridley Scott" # features/step_definitions/director_steps.rb:7

  Scenario: find movie with same director                       # features/add_director.feature:22
    Given I am on the details page for "Star Wars"              # features/step_definitions/web_steps.rb:44
    When I follow "Find Movies With Same Director"              # features/step_definitions/web_steps.rb:56
    Then I should be on the Similar Movies page for "Star Wars" # features/step_definitions/web_steps.rb:230
    And I should see "THX-1138"                                 # features/step_definitions/web_steps.rb:105
    But I should not see "Blade Runner"                         # features/step_definitions/web_steps.rb:123

  Scenario: can't find similar movies if we don't know director (sad path) # features/add_director.feature:29
    Given I am on the details page for "Alien"                             # features/step_definitions/web_steps.rb:44
    Then I should not see "Ridley Scott"                                   # features/step_definitions/web_steps.rb:123
    When I follow "Find Movies With Same Director"                         # features/step_definitions/web_steps.rb:56
    Then I should be on the home page                                      # features/step_definitions/web_steps.rb:230
    And I should see "'Alien' has no director info"                        # features/step_definitions/web_steps.rb:105

3 scenarios (3 passed)
17 steps (17 passed)
0m0.414s
Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42/coverage. 57 / 57 LOC (100.0%) covered.
--------------------------------------------------------------------------------
----END CUCUMBER----
----BEGIN RSPEC----
--------------------------------------------------------------------------------
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -I/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...................

Finished in 0.19556 seconds (files took 1.46 seconds to load)
19 examples, 0 failures

Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-10-42/coverage. 57 / 57 LOC (100.0%) covered.
--------------------------------------------------------------------------------
----END RSPEC----

but running that in the submission directory above also generates a 100% test coverage ...

[tansaku]

so stepping through the HW4 grader in the debugger is not helping me very much, but if I look in the coverage directory in the submission directory I can see that a <100% coverage is being reported there:

[tansaku]

showing gaps in the movies controller like so:

[tansaku]

and if I run bundle exec rake saas:run_student_tests again in that submission folder then the coverage updates to 100%, indicating that somehow on the first run of that task, that perhaps not everything is in place ...?

[tansaku]

so I hooked the grader with the debugger to stop just before it starts parsing the test coverage files. The coverage files then show another different story

Is there some race condition here where the parsing of the coverage files starts before they have finished being generated? That doesn't make sense ...

[tansaku]

so with the grader hooked on the debugger I ran the rake task and got the failure in the terminal:

Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-39-08/coverage. 48 / 57 LOC (84.21%) covered.
--------------------------------------------------------------------------------
----END CUCUMBER----
----BEGIN RSPEC----
--------------------------------------------------------------------------------
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -I/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
....FF.............

Failures:

  1) MoviesController GET #index should sort movies by title
     Failure/Error: expect(assigns(:movies)).to match_array(Movie.order(:title))
       expected collection contained:  [#<Movie id: 11, title: "Star Wars", rating: "PG", description: "Star Wars is a movie about...", release_date: "1977-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 12, title: "Blade Runner", rating: "PG", description: "Blade Runner is a movie about...", release_date: "1982-06-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 13, title: "Alien", rating: "R", description: ".........", release_date: "1979-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 14, title: "THX-1138", rating: "R", description: "THX is a movie about", release_date: "1971-03-11 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 15, title: "Superman", rating: "PG", description: "Superman is a movie about", release_date: "1973-03-21 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "">]
       actual collection contained:    [#<Movie id: 1, title: "Aladdin", rating: "G", description: nil, release_date: "1992-11-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 2, title: "The Terminator", rating: "R", description: nil, release_date: "1984-10-26 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 3, title: "When Harry Met Sally", rating: "R", description: nil, release_date: "1989-07-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 4, title: "The Help", rating: "PG-13", description: nil, release_date: "2011-08-10 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 5, title: "Chocolat", rating: "PG-13", description: nil, release_date: "2001-01-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 6, title: "Amelie", rating: "R", description: nil, release_date: "2001-04-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 7, title: "2001: A Space Odyssey", rating: "G", description: nil, release_date: "1968-04-06 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 8, title: "The Incredibles", rating: "PG", description: nil, release_date: "2004-11-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 9, title: "Raiders of the Lost Ark", rating: "PG", description: nil, release_date: "1981-06-12 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 10, title: "Chicken Run", rating: "G", description: nil, release_date: "2000-06-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 11, title: "Star Wars", rating: "PG", description: "Star Wars is a movie about...", release_date: "1977-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 12, title: "Blade Runner", rating: "PG", description: "Blade Runner is a movie about...", release_date: "1982-06-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 13, title: "Alien", rating: "R", description: ".........", release_date: "1979-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 14, title: "THX-1138", rating: "R", description: "THX is a movie about", release_date: "1971-03-11 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 15, title: "Superman", rating: "PG", description: "Superman is a movie about", release_date: "1973-03-21 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "">]
       the extra elements were:        [#<Movie id: 1, title: "Aladdin", rating: "G", description: nil, release_date: "1992-11-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 2, title: "The Terminator", rating: "R", description: nil, release_date: "1984-10-26 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 3, title: "When Harry Met Sally", rating: "R", description: nil, release_date: "1989-07-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 4, title: "The Help", rating: "PG-13", description: nil, release_date: "2011-08-10 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 5, title: "Chocolat", rating: "PG-13", description: nil, release_date: "2001-01-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 6, title: "Amelie", rating: "R", description: nil, release_date: "2001-04-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 7, title: "2001: A Space Odyssey", rating: "G", description: nil, release_date: "1968-04-06 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 8, title: "The Incredibles", rating: "PG", description: nil, release_date: "2004-11-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 9, title: "Raiders of the Lost Ark", rating: "PG", description: nil, release_date: "1981-06-12 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 10, title: "Chicken Run", rating: "G", description: nil, release_date: "2000-06-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>]
     # ./spec/controllers/movies_controller_spec.rb:108:in `block (3 levels) in <top (required)>'

  2) MoviesController GET #index should sort movies by rating
     Failure/Error: expect(assigns(:movies)).to match_array(Movie.order(:release_date))
       expected collection contained:  [#<Movie id: 11, title: "Star Wars", rating: "PG", description: "Star Wars is a movie about...", release_date: "1977-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 12, title: "Blade Runner", rating: "PG", description: "Blade Runner is a movie about...", release_date: "1982-06-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 13, title: "Alien", rating: "R", description: ".........", release_date: "1979-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 14, title: "THX-1138", rating: "R", description: "THX is a movie about", release_date: "1971-03-11 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 15, title: "Superman", rating: "PG", description: "Superman is a movie about", release_date: "1973-03-21 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "">]
       actual collection contained:    [#<Movie id: 1, title: "Aladdin", rating: "G", description: nil, release_date: "1992-11-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 2, title: "The Terminator", rating: "R", description: nil, release_date: "1984-10-26 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 3, title: "When Harry Met Sally", rating: "R", description: nil, release_date: "1989-07-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 4, title: "The Help", rating: "PG-13", description: nil, release_date: "2011-08-10 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 5, title: "Chocolat", rating: "PG-13", description: nil, release_date: "2001-01-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 6, title: "Amelie", rating: "R", description: nil, release_date: "2001-04-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 7, title: "2001: A Space Odyssey", rating: "G", description: nil, release_date: "1968-04-06 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 8, title: "The Incredibles", rating: "PG", description: nil, release_date: "2004-11-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 9, title: "Raiders of the Lost Ark", rating: "PG", description: nil, release_date: "1981-06-12 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 10, title: "Chicken Run", rating: "G", description: nil, release_date: "2000-06-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 11, title: "Star Wars", rating: "PG", description: "Star Wars is a movie about...", release_date: "1977-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 12, title: "Blade Runner", rating: "PG", description: "Blade Runner is a movie about...", release_date: "1982-06-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 13, title: "Alien", rating: "R", description: ".........", release_date: "1979-05-25 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "Ridley Scott">, #<Movie id: 14, title: "THX-1138", rating: "R", description: "THX is a movie about", release_date: "1971-03-11 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "George Lucas">, #<Movie id: 15, title: "Superman", rating: "PG", description: "Superman is a movie about", release_date: "1973-03-21 00:00:00", created_at: "2016-03-24 12:46:32", updated_at: "2016-03-24 12:46:32", director: "">]
       the extra elements were:        [#<Movie id: 1, title: "Aladdin", rating: "G", description: nil, release_date: "1992-11-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 2, title: "The Terminator", rating: "R", description: nil, release_date: "1984-10-26 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 3, title: "When Harry Met Sally", rating: "R", description: nil, release_date: "1989-07-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 4, title: "The Help", rating: "PG-13", description: nil, release_date: "2011-08-10 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 5, title: "Chocolat", rating: "PG-13", description: nil, release_date: "2001-01-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 6, title: "Amelie", rating: "R", description: nil, release_date: "2001-04-25 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 7, title: "2001: A Space Odyssey", rating: "G", description: nil, release_date: "1968-04-06 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 8, title: "The Incredibles", rating: "PG", description: nil, release_date: "2004-11-05 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 9, title: "Raiders of the Lost Ark", rating: "PG", description: nil, release_date: "1981-06-12 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>, #<Movie id: 10, title: "Chicken Run", rating: "G", description: nil, release_date: "2000-06-21 00:00:00", created_at: "2016-03-24 12:39:11", updated_at: "2016-03-24 12:39:11", director: nil>]
     # ./spec/controllers/movies_controller_spec.rb:115:in `block (3 levels) in <top (required)>'

Finished in 0.21762 seconds (files took 1.59 seconds to load)
19 examples, 2 failures

Failed examples:

rspec ./spec/controllers/movies_controller_spec.rb:104 # MoviesController GET #index should sort movies by title
rspec ./spec/controllers/movies_controller_spec.rb:112 # MoviesController GET #index should sort movies by rating

Coverage report generated for Cucumber Features, RSpec to /Users/tansaku/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-12-39-08/coverage. 57 / 57 LOC (100.0%) covered.
/Users/tansaku/.rvm/rubies/ruby-2.2.2/bin/ruby -I/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/lib:/Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-support-3.3.0/lib /Users/tansaku/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed

[tansaku]

what I would like to do now is leave the grader to finish running, and then compare what, if any, files are changed over the completion of the grader activity - I would do that with git, but the submissions directory is nested inside the rag git space ...

[tansaku]

so I tarred the directory up (to take a snapshot of it) and then let the grader carry on - the acceptance test now failed because this time around the different coverage report gives us 60% instead of 40% (go figure), but when I test in the submission directory again I get all tests passing and 100% coverage implying that something has changed through the completion of the grader run ...

[tansaku]

so the only differences that appear are in the timestamps in the coverage directory and the contents of the test.log

[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4 (92_factorygirl_in_hw4)]$ 
→ diff -bur 2016-03-24-12-39-08 ~/tmp/
.ssh/                2016-03-24-12-39-08/ ssh.tar              
[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4 (92_factorygirl_in_hw4)]$ 
→ diff -bur 2016-03-24-12-39-08 ~/tmp/2016-03-24-12-39-08/
diff -bur 2016-03-24-12-39-08/coverage/.resultset.json /Users/tansaku/tmp/2016-03-24-12-39-08/coverage/.resultset.json
--- 2016-03-24-12-39-08/coverage/.resultset.json    2016-03-24 12:52:50.000000000 +0000
+++ /Users/tansaku/tmp/2016-03-24-12-39-08/coverage/.resultset.json 2016-03-24 12:50:05.000000000 +0000
@@ -227,6 +227,6 @@
         null
       ]
     },
-    "timestamp": 1458823970
+    "timestamp": 1458823805
   }
 }
diff -bur 2016-03-24-12-39-08/coverage/index.html /Users/tansaku/tmp/2016-03-24-12-39-08/coverage/index.html
--- 2016-03-24-12-39-08/coverage/index.html 2016-03-24 12:52:50.000000000 +0000
+++ /Users/tansaku/tmp/2016-03-24-12-39-08/coverage/index.html  2016-03-24 12:50:05.000000000 +0000
@@ -14,7 +14,7 @@
       <img src="./assets/0.10.0/loading.gif" alt="loading"/>
     </div>
     <div id="wrapper" style="display:none;">
-      <div class="timestamp">Generated <abbr class="timeago" title="2016-03-24T12:52:50+00:00">2016-03-24T12:52:50+00:00</abbr></div>
+      <div class="timestamp">Generated <abbr class="timeago" title="2016-03-24T12:50:05+00:00">2016-03-24T12:50:05+00:00</abbr></div>
       <ul class="group_tabs"></ul>

       <div id="content">
Binary files 2016-03-24-12-39-08/db/test.sqlite3 and /Users/tansaku/tmp/2016-03-24-12-39-08/db/test.sqlite3 differ
diff -bur 2016-03-24-12-39-08/log/test.log /Users/tansaku/tmp/2016-03-24-12-39-08/log/test.log
--- 2016-03-24-12-39-08/log/test.log    2016-03-24 12:52:50.000000000 +0000
+++ /Users/tansaku/tmp/2016-03-24-12-39-08/log/test.log 2016-03-24 12:50:05.000000000 +0000
@@ -759,355 +759,3 @@
    (0.5ms)  rollback transaction
    (0.1ms)  begin transaction
    (0.0ms)  rollback transaction
-   (0.8ms)  CREATE TABLE "movies" ("id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "title" varchar, "rating" varchar, "description" text, "release_date" datetime, "created_at" datetime, "updated_at" datetime, "director" varchar) 
-   (0.8ms)  CREATE TABLE "schema_migrations" ("version" varchar NOT NULL) 
-   (0.1ms)  select sqlite_version(*)
-   (0.8ms)  CREATE UNIQUE INDEX "unique_schema_migrations" ON "schema_migrations" ("version")
-   (0.1ms)  SELECT version FROM "schema_migrations"
-   (0.6ms)  INSERT INTO "schema_migrations" (version) VALUES ('20160310215150')
-   (0.6ms)  INSERT INTO "schema_migrations" (version) VALUES ('20111119180638')
-   (0.5ms)  INSERT INTO "schema_migrations" (version) VALUES ('20120130161449')

[tansaku]

which makes me think this must be some sort of database issue - that something to do with FactoryGirl is running in a thread or something? That some process isn't completing before the coverage is being assessed ... or something ... or am I totally on the wrong track?

[tansaku]

is the problem here that the tests are being run more than once and that the test database is not being cleared between runs ...?

[tansaku]

running repeatedly in the submission directory does not cause a problem, but looking at the failure we got, it appears to be that the movies table is holding more movies that we expect ...

[tansaku]

ah, is it that the cucumber and specs are being run together, and elements from the cucumber test are hanging around in the database and not being cleaned before the specs are run or something ...?

[tansaku]

hmm, looks like we have the movie Aladdin being added in both a migration and also the seeds file ...

[tansaku]

and that this is a migration added by the grader?

[tansaku]

so the grader gets the test db set up with some data in the movies table:

[tansaku@Samuels-MBP:~/Documents/GitHub/saasbook/rag/submissions/506af89a6181960fc69f47f1fbc8d708/assignment4/2016-03-24-13-32-11 (92_factorygirl_in_hw4)]$ 
→ sqlite3 db/test.sqlite3 
SQLite version 3.8.10.2 2015-05-20 18:17:19
Enter ".help" for usage hints.
sqlite> .tables
movies             schema_migrations
sqlite> select * from movies;
1|Aladdin|G||1992-11-25 00:00:00.000000|2016-03-24 13:32:14.898606|2016-03-24 13:32:14.898606|
2|The Terminator|R||1984-10-26 00:00:00.000000|2016-03-24 13:32:14.901964|2016-03-24 13:32:14.901964|
3|When Harry Met Sally|R||1989-07-21 00:00:00.000000|2016-03-24 13:32:14.902735|2016-03-24 13:32:14.902735|
4|The Help|PG-13||2011-08-10 00:00:00.000000|2016-03-24 13:32:14.903478|2016-03-24 13:32:14.903478|
5|Chocolat|PG-13||2001-01-05 00:00:00.000000|2016-03-24 13:32:14.904258|2016-03-24 13:32:14.904258|
6|Amelie|R||2001-04-25 00:00:00.000000|2016-03-24 13:32:14.905083|2016-03-24 13:32:14.905083|
7|2001: A Space Odyssey|G||1968-04-06 00:00:00.000000|2016-03-24 13:32:14.906002|2016-03-24 13:32:14.906002|
8|The Incredibles|PG||2004-11-05 00:00:00.000000|2016-03-24 13:32:14.907292|2016-03-24 13:32:14.907292|
9|Raiders of the Lost Ark|PG||1981-06-12 00:00:00.000000|2016-03-24 13:32:14.908173|2016-03-24 13:32:14.908173|
10|Chicken Run|G||2000-06-21 00:00:00.000000|2016-03-24 13:32:14.908977|2016-03-24 13:32:14.908977|

[tansaku]

which I tried deleting, but that has no impact ...

[tansaku]

however if I drop the db after and then run the tests straight after running the migrations then I get the failures ...

[tansaku]

hmm, but if I remove that extra migration from the hw4 grader template - I still get the same overall failures, and checking the submission directory that migration is not present ... what?

I can no longer replicate the error with by running turning over the db in the submission directory ... which makes me think there might be some caching complexity - gonna push this into the branch and see what the CI thinks ...

[mtc2013]

@tansaku placed factory_girls_rails into the root autograder gem file as a quick fix for some of the issues relating to FactoryGirl that were silently failing. It appears that the way the AG launches the rake task does not make use of the bundle installed gems from the student submission.

[tansaku]

Thanks @mtc2013 - just to follow up on that with some more details for @zhangaaron and @rosewcs345 it looks like the hw4grader runs the two commands bundle install --without production and bundle exec rake saas:run_student_tests in two separate processes.

The grader runs the tests in the correct directory (a generated submission directory) where the Gemfile and Gemfile.lock are correctly set up (and now have factory_girl because I added it to the hw4 skeleton). However when the hw4 grader runs the following code:

Open3.popen3(env, "bundle exec rake saas:run_student_tests")

the gem environment that's available appears to be that of the grader itself and not that of the merged student/hw4-template directory (this might be due to the bundle install having been run in a separate process, but then again ...).

We've rectified this temporarily by adding the factory_girl_rails gem to the grader Gemfile, and set that up on staging (which @rosewcs345 's students are using), which seems to make factory_girl_rails available at the point when the grader runs the test on the student assignment.

It's possible that we are still mis-interpreting things. The hw4 grader template also includes an additional migration that dumps extra movies in the database that can interfere with tests that pass on student machines. I'm not clear if this extra movie migration is actually necessary for the hw4 grader (I've removed it in the PR), however it can definitely cause failures that are then not reported back to the student (as well as making it difficult for developers to debug).

And of course anytime any a student uses any new gem in their submission things will break silently. It seems odd actually that we are even using Open3.popen3 since we are already running the hw4grader in a fork, but I guess that's just a way of running for and waiting for a series of threads to finish. I had thought that running a bundle exec prefixed process in a directory with a particular Gemfile and Gemfile.lock would guarantee the gems in that file to be available, but perhaps it is different for threads ...? (ruby generated threads somehow inherit the parent thread's gemset?) That's something we should carefully experiment with, although there's the lower hanging fruit of adjusting the grader to make these errors visible rather than failing silently ...

Anyway, I'm off for the next 10 days. I've added the factory_girl_rails gem to the grader in the pull request, and at some point before the next run of the MOOC we will have to get all this stuff out onto the master branch and into production, but things are a bit jury-rigged at the moment so I'd be cautious about merging a lot of stuff in ...