Faststart should allow for upload of logs when it fails

[gregdek]

Yeah, this really does need to happen -- probably the most valuable thing here.

The key question is really, "where should it go?" What's easiest and makes the most headway the quickest? S3 bucket? And what should go there -- just faststart logs? Full sosreport?

[eucacolby]

@gregdek with help from @hspencer77 we have a mechanism to store them. At least, for our initial experiment which will grab a useful set of logs and command outputs. We can later move to a full SOSreport model.

Worth keeping in mind that many EUCA_INSTALL_FAILURE conditions have been due to network timeout. Thus, we cannot always rely on SOSReport. Better detection and handling of slow network connections would, I think, necessitate another ticket.

[kushaldas]

Copy pasting the questions and responses to keep things in one place. Questions are from @viglesiasce and answered by @hspencer77

A few questions before we get too far in the implementation:

    Can we host conversations like this in a Github issue in the eucalyptus-cookbook project?
Sure, don’t see a problem with it.  Makes sense.

    Having just the chef-output log is not enough to triage a failure. As such I had opened the following issue
        https://github.com/eucalyptus/eucalyptus-cookbook/issues/80

SOSReport will be a great place to get to. However, for this experiment we'll keep it very simple - also we've seen enough failures due to slow network and shouldn't immediately count on user being able to install SOSReport. Vic shared a list of commands whose output we should add to the log prior to uploading. https://github.com/eucalyptus/eutester/blob/master/testcases/cloud_admin/get_debug.py#L6 The key commands are in the basic_commands and network_commands. We should also grab /var/log/eucalyptus files if any are present. ... so at this point we probably need to tar/compress all these and just upload that file.

    How do we identify a user by their log?
Colby can clarify more here, but from my understanding, there would be unique ID generated using the user’s email address and MAC address of the machine they are using.

Yep, there's a uuid for each install. When a user a) permits us to upload the log(s) and b) gives us their email address we'll know exactly which file goes with which user. If we get a and not b, then obviously we won't know; but at least we'll have gathered more info for analysis.

    We should not downplay the misuse of the server if we put our private key out there and allow people to upload anything they want. (Im sure Vika would agree). How did we do this for the black-hole/worm hole project that was started up a year or so ago?
On the SFTP server, I set up a quota for the user that has the ability to upload to the server.  The user is in a chroot environment, and only has access to write to the uploads directory.  Additionally, I added a 2 gig quota (with a 1.5 gig soft quota), so the user can’t fill up the filesystem.   

With Black Hole, just as we do on the SFTP server, accounts were generated manually.  Someone from Support had to create the account, then provide the credentials to the customer/user.  

From my understanding, once the user confirms that they would like to upload their files to help with triaging, they would have to provide a name and email address - which is then used in Marketo and SalesForce. 

At a high-level,  maybe instead of providing the private key in the github repo, we could get the user to confirm that their email address is valid.  Once they confirm, then we will allow them to upload by giving them the key?

    Is this automatically done for the user or does it require manual intervention? Can someone explain the lifecycle?

Colby can provide more clarity here, but from my understanding, manual intervention is required by the user. Here is the flow:

[kushaldas]

Colby can provide more clarity here, but from my understanding, manual intervention is required by the user.

The code I wrote still now asks for user's permission and then renames and uploads the log file to the server.

[eucacolby]

1) User launches FastStart installer 2) Install fails while running Euca installer chef recipe 3) Install script catches the EUCA_INSTALL_FAILURE case 4) Install prompts user to upload logs for analysis [y/n] default to y 5) Installer runs series of commands suggested by @viglesiasce (see above). Output captured into a file 6) Install gathers basic log files and the file from step 5 into a tar/compressed file 7) Install names the tar/compressed file with the install's uuid 8) Install uploads file to sftp server 9) Install prompts user with option to get free support 10) User does / does not enter their email address (hey, people say "no" to help all the time) 11) Install pings existing form with tar/compressed file name and, if provided, the user's email address 12) Install thanks the user, gives any additional guidance (as it does today) and exits.

On the backend, the Eucalyptus support team is notified when a log bundle is uploaded. Team reviews the bundle on best-effort priority. If user provided a valid email address, support contacts user to assist or for further debugging.

Our goal throughout all of this is to improve the installation experience for all users. To that end, we will kindly ask to gather logs for debugging and product improvement. Log/debug info collected is used ONLY for that purpose. The more logs we're given the better we can make the install experience for future users but, again, we will not collect any of this without the user's consent.

We also sincerely want to help each user successfully install Eucalyptus. For this reason we will offer users the opportunity to get free installation support. And for obvious reasons we ask for their email address. Users are of course not required to provide this and may, instead, use IRC, send mail to the euca-users list, or not do anything at all.

[kushaldas]

@viglesiasce Would you prefer one single commit with both code change and the private key? or else I can do one commit for each.

[viglesiasce]

@kushaldas Doesnt matter as long as its the same pull request.

[kushaldas]

Added PR https://github.com/eucalyptus/eucalyptus-cookbook/pull/89