2019-04-01

[limnh]

Where I left off on Friday:

• Confirmed that the process worked (once) from start to finish without error. There's still some cleaning up that should be done, but the process will run as it currently stands. Testing the process more today to verify that the first success wasn't an outlier.
• Had some time to mull over feedback I've received, still brainstorming ways to make high productivity days more consistent. Andrew has been invited to a daily code review between Barry and I. First meeting is today.

Daily To-Dos:

• Create a new user in AWS, create a policy for said user that allows access only to specified s3 bucket. Use these updated credentials to create these images.
• Utilize aws s3 ls to list files in the s3 bucket to confirm that the image was uploaded successfully. be kind and allow people to avoid the aws console.
• Go through the existing files and do some housecleaning because that shit is uggo.

Issues/Questions:

1. I accidentally added the iso file to git add, which github did not like.
2. 1st pomodoro (11:15a-11:45a)
   • how do I set a policy for AWS to ensure the created user only has access to the specified s3 bucket?
   • how do I print aws ls s3 to the CircleCI console?
3. how do I make sure that my first run isn't an outlier?
4. There are some wild inconsistencies when it comes to build time, build on friday took ~20 minutes and the one from today took ~2 hours. Is there any way to make this more consistent?

Solutions:

1. I spent some time during my earlier meeting to sort through getting the commit reverted and fixed.
2. 1st pomodoro (11:15a-11:45a)
   • I originally looked through the AWS documentation about how to grant access to an s3 bucket. however, I found just creating a policy from scratch was pretty straightforward, so I created the tbw-kvm-guest-s3-access policy and added the tbw-kvm-guest-images-s3 user to it.
   • ~added an "inline": "echo 'aws s3 ls s3://$S3_BUCKET --human-readable --recursive --summarize'" to the packer.json file.~
   • decided it was better to let packer solely build the image, so moved this command to the shell script.
3. I currently have two successful builds, one from Friday and one from earlier today. I am currently readying my third successful run with updated AWS credentials for the specifically-created user in AWS & policy for the s3 bucket.
   • 2pm PDT: currently have six successful builds with one building currently
4. Unfortunately, I feel like we're at the mercy of the components needed to build this image (circle, packet, etc.) for now.
   • looking at build times from today, we've seen between 35m to 1.5h
   • 3:30p Packet is having a time right now, so all attempts to spin up a server are timing out

[limnh]

I would like to learn more efficient practices of testing locally.

[limnh]

Questions to continue to mull over:

• Should we be keeping a running history of images within AWS, or should each one replace the one prior?
  • Discussed with barry.

    • I think in the short term, we're fine filling that bucket up..but at some point, we're gonna find ourselves not changing that image for a while, so it wouldn't be a bad thing to no let old ones linger. That might be some value in a promotion process or something...e.g "I like this one" then rename it to something and have your script match on some string that isn't what it's renamed to. There are a few ways to solve this problem, just need to think about all the edge cases. Because this is all in source control (git), should we need to build a specific image, we could in the event that one got deleted...but should we find ourselves in a situation where we need to revert quickly, we're at the mercy of the amount of time it takes to build an image and the availability of that pipeline...So having the images (some at least) readily available is more advantageous than having to push a specific commit through the pipeline again

• Should I keep the PACKER_LOG=1 in there? On one hand, it ensures that circle doesn’t timeout due to no output in 10 minutes. On the other, the output to the circleci console is so long that you have to download the full file to see the whole logs.
  • I'm going to keep the logging in for now, so we can confirm that we know what's going on with these builds. It keeps Circle from timing out, and there's no harm in downloading the full log file.