Github access failure during cucumber tests

[tansaku]

Running the tests on fresh OSX 10.11.2 (El Capitan) I got the following trying to run the cucumber tests

The authenticity of host 'github.com (192.30.252.129)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.252.129' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
    Then I should receive a grade of "100" for my assignment                                             # features/step_definitions/adapter_x_queue_steps.rb:49
      expected PutResultException, got #<IOError: Fatal error: Retrieving spec files from git@github.com:saasbook/hw-ruby-intro-ci.git repository failed.> with backtrace:
        # ./lib/assignment/xqueue.rb:58:in `fetch_spec_file'
        # ./lib/assignment/xqueue.rb:34:in `initialize'
        # ./lib/submission/xqueue.rb:21:in `new'
        # ./lib/submission/xqueue.rb:21:in `next_submission_with_assignment'
        # ./lib/submission/polling.rb:18:in `block in run'
        # ./lib/submission/polling.rb:17:in `loop'
        # ./lib/submission/polling.rb:17:in `run'
        # ./features/step_definitions/adapter_x_queue_steps.rb:53:in `block (3 levels) in <top (required)>' (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/adapter_x_queue_steps.rb:50:in `/^I should receive a grade of "(.*?)" for my assignment$/'
      features/autograder_xqueue_stubbed.feature:20:in `Then I should receive a grade of "100" for my assignment'

cucumber features/autograder_xqueue_stubbed.feature:16 # Scenario: student submits RSpecGrader assignment containing multiple files with specs hosted on Github

[tansaku]

I notice that this seems to be trying to reach one of our private repositories:

git@github.com:saasbook/hw-ruby-intro-ci.git

[zhangaaron]

We assume that the machine running rag has the correct ssh keys set up in .ssh/ folder. It is put in the set up instructions I believe. We have the ssh key configured in the EC2 rag as well.

[tansaku]

@zhangaaron right, but this means that for a 3rd party developer they can't run the tests, and also that the tests are not sandboxed, i.e. they are hitting github every time we run them ...

[tansaku]

@zhangaaron how is the Travis CI not hitting this error? I think you described it before, but I can't see anything in the travis settings to support it. Is travis running as a particular user or something? Ah, no I see it now, we are not running any of the @require-net-connect features on the CI ... hmmm .... seems like we could if we followed the advice here:

https://docs.travis-ci.com/user/private-dependencies

[tansaku]

had a go with https://docs.travis-ci.com/user/private-dependencies/#API-Token but no luck so far ...

[zhangaaron]

I think I tried the option for configuring a SSH key/deploy key onto the RAG instance, but that requires a private repository and we didn't want that. And that's true, the tests are sandboxed but I think part of the tests are checking that it can correctly establish an ssh connection with the GitHub server and download the files appropriately. I suppose we could stub the SSH connection but that's a big part of the test.

Do you think this is high priority? I do think we should eventually distinguish Cucumber tags for @requires-ssh-key and @require-net-connect, since the second one is more general and includes Heroku tests.

[tansaku]

@zhangaaron I think the high priority is to remove all the solutions to the homeworks that we currently expose through this repo, which this connects to.

Ideally we need a VCR for ssh to create a proper sandbox:

http://stackoverflow.com/questions/18632270/is-there-something-like-vcr-http-interaction-stubbing-replay-for-commandline-c

However in principle if we can get the API token working we can both re-hide the solutions and ensure that we are keeping everything in step with the other dependent repos ...

[tansaku]

opened an issue with travis-ci: https://github.com/travis-ci/travis-ci/issues/5774

[tansaku]

@zhangaaron the other issue is that any developer who wants to contribute needs the ssh keys in order to check for green tests as they are working on the codebase

[zhangaaron]

Yeah I see why this is an issue but I guess the purpose of integration tests is to see if the autograder does what it is intended to do, which is grade assignments. So not sure how we should modify these tests to allow other developers to work on this without access to the solutions or ssh keys.

[tansaku]

@zhangaaron sure, but at the moment we have the worst of both worlds right? The solutions are exposed AND other developers can't work on this without the ssh keys.

I'm starting to get input from the travis folks on the API_TOKEN issue: https://github.com/travis-ci/travis-ci/issues/5774

If we can get that working then in principle the solutions can all be concealed in private repos, and other developers can run the complete tests via CI using the token, but will not have full access to the solutions ...

[tansaku]

Hey @zhangaaron - lots of great input from the travis folks here: https://github.com/travis-ci/travis-ci/issues/5774 - seems like we can get things to run with encrypted API_TOKENS (securing our private content and allowing us to run all our tests), although it doesn't go quite as far as allowing 3rd party devs to do the same, but at least it addresses one of our two issues.

Funny thing now is that I have everything passing locally, but I get this one failure on CI that seems unrelated to network access, see: https://github.com/saasbook/rag/pull/84

any ideas?