9/24 -- Conduct User Interviews

[iamb55]

Talk with users about their requirements for the product.

[jessepollak]

TEAM TASK: @brennenbyrne @dunvi @jessepollak

[dunvi]

Risk:

• One of the most important tasks in the process! If mishandled, will damage/corrupt all following tasks

Avoidance:

• Careful planning of the interview process
• Consider practice interviews, both for us as the interviewers, and to test our planned process

Dependencies:

• 10, identify users to approach

• 11, create questions

[jessepollak]

Notes from Tommy Vertanen hangout

 Twisted guys set up a buildbot from scratch
-part of those conversations
-at work, using GitBuilder,
    -does build part and runs unit tests
    -builds all the branches and all the commits
-people near him use Jenkins

-problems
    -trivial projects, build is quick
        -one environment you care about
        -unit tests are sufficient for everything
        -easy--precommit hook and everyone would run the tests before you push them
    -simple thing isn't good enough because
        -if your builds take a long time
            -worker runs that cache results or splits up the builds
    -make has thing theing where you build a target on dependencies
        -looks at dependencies and compares them with preexisting destination
        -doesn't look at change in build system, doesn't give you comfort in knowing things are done right
        -SOLUTION
            -build a clean tree everytime and hook the compilationg into clean cache
            -if contents are exactly identical, then spits out old

    have to look at what our target audience is
        -web developers vs non-web developers

    don't think there is a fundamental solution to all of the problems for multiple categories

    typical web project, division of work for build system is completely different
        -don't have compilation time
    automated testing of something like django project is on a compeltely different level

    if you don't have a SQL database in the picture the nyou don't have to test against a SQL database
        -if you can avoid the problem then you don't need to solve it

    in his space, essentially no testing besides build time set up
        -nightly build across 100 computers
        -test suite takes up to 20 minutes for stress oriented tests

unit tests happen at build time, the rest is to complicated to integrate
    -GitBuilder
        -understands git perfectly
            -tags, commits, which have been tested etc
            -can do things like
                -if you have a history of a branch that used to be green, it will try the latest and then if that fails it will start bisecting the commit space and find out which commit caused it is
                    -REALLY HELPFUL

if you were to solve the full problem, it would become a question of distributing resources
    if you have a huge set of systems dedicated to testing, how do you use it to its maximum potential

if you push it enough, in build or tests taking a long time, it becomes a question of scheduling
    -jenkins is a FIFO queue of work to do
    -gitbuilder will skip building intermediate commits if it goes from green to green

BUILD SOMETHING THAT CAN PLUGIN TO SOMETHING ELSE
    -replacability

Think about "scalability"
    -you will *overwhelm* a single machine
    -should look at cloud API to launch other machines

want to have lots of control over isolation and reproducibility
    -it's fairly easy to ignore when you have a simple problem
    -easy way to create new environments

try to build a system that has minimum trust from one component to another

are we building this so everything is located next to each other or across the country
    -push or pull?
        -some of the components might be behind a NAT where communication is one way
        -PICK ONE!

say GOAL is this NON-GOAL is this

 USES GO!

[dunvi]

Notes from Randall Hunt

Works at company that does MongoDB, python evangelist

uses:

• Buildbot - for MongoDB; SLOW to run testing
  • good but you have to write it yourself
  • most people don't do it right but remake wheel, stuff is provided but people don't realize it's there
  • like a day to get setup
  • a week to a year to get it set up the way you want it
• Jenkins
  • provides packages, can just install it
  • if running on the jenkins master you can basically run it right away
  • about a week to get it right (external services to start up, distributed etc)

Opinion: small teams should be using CI more than big teams!

• Wishlist of stuff wishes CI would do:
  • security - builds should be open source, but intermediate stuff (buildbot doesn't allow you to do with unless you write your own custom auth)
  • perfect integration with a bugchecker! <-- I get that :P make it modular so we can use something besides github issues
  • force a common set of slaves within a group (linux 64 should be matched linux 64)
  • run builds across platforms
  • run CLEANLY, don't keep old stuff from the previous build lying around! makes it harder to find where bugs are introduced
  • mark certain builds, logs, test logs etc to keep into the future (stick into a new branch?)
  • prioritize most frequently failing tests first
  • things that take a long time to test should fail faster - short circuiting
  • track performance (see if your changes are affecting performance)
  • track a certain subset of slaves (tag slaves)
  • ordering tests (sniffing, then build, then unit, then functional, then non-functional, then acceptance/performance)
  • BLAME LIST <--- MOST requested feature. preferably the line, not just the commit, or at least the more specific the better

Many CI servers have some of these, none of them have all of these

best case scenario would be sensible defaults and easy customizability

A lot of CI were made years ago, and they would have to refactor to improve dramatically. instead they've been doing incremental upgrades, so they're sorta stuck

Jesse would know > Daniel Something

[dunvi]

Notes from Michael Stock-Matthews

Michael Stock-Matthews

• Do you ever build tests into small projects? (unit tests, etc)
  • rarely
  • size and complexity is pretty related
  • code base side, team members might also be related
  • integration tests that his code had to pass, but no unit tests
  • written, no regression tests to pass
  • for team member size, there's no threshold, tests are always
  • helpful and a good idea
  • "Honestly, you should probably do it pretty much all the time"
  • who you're building for affects it a lot
  • building for general public > more expectation that it will work
  • less expectation of your availability > needs to be more robust
  • what keeps people from building tests?
  • what keeps people from doing it: LAZINESS
  • what's actually hard: coming up with edge cases, covering all options, etc.
• Do you use GitHub? Features?
  • YES
  • Issue tracking
  • Pull requests is very nice
  • wiki
  • social abilities is pretty cool, but doesn't use it right now
  • service hooks? never used it through github interface, but has used hooks
  • requires some googling to use it the first time :P
  • didn't take too long, but had been using git for a while
  • estimates 10 minutes first time set up
  • accessible? if you know they exist and you have background knowledge etc.
• How do you build your small projects? Any project?
  • Usually hack stuff away without any version control until it
  • works or it becomes too big
  • Development on some repo? "one way is..." response, traditional branching setup
  • At Yammer, had an internal service called diploymency, web interface that you
  • used to specify details and it just deployed it for you
  • was doing internal stuff, was given the ability to push to production "with a click" as a result *Yammer repos were on github, but weren't using it necessarily
  • Doesn't really use any service
  • How would you go about putting a personal project to the public? Heroku
  • if it would exceed the free tier of heroku, probably shop around
  • At Yammer, were tests integrated? You had to run your own tests
  • DEPENDS ON WHICH REPO
  • Higher priority repos had tests integrated into deployment
  • For his, he was allowed to break live stuff so no real testing beyond his own
  • "If you have to pass tests, it's pretty much enforced"
  • Might encourage less testing if it's required because you would rely on it just breaking and not deploying instead of checking
  • If there was a blame list, though... :P
    • (At Yammer, did a lot of "shaming" - statistics for the deploy
    • if it was public that you were breaking a lot, well...)
    • Might be an effective deterrant
  • How much testing goes into a build? If you're fixing a bug, you might do it
  • in one commit, a feature might take a lot, so you might not test until after a lot of commits. However, testing on every commit is probably preferable.
  • If you're pushing to production, it should pass the tests
  • Reminder here that people are lazy :P
  • How much do you commit code that is broken that a test would have caught?
  • (Not just you, other people too)
  • Depends on what you're working on, but something that's not statically compiled
  • Worth enforcing tests? probably, since it would catch the bugs.
    • if you had to write all the tests, it'd be annoying but a good thing
    • (depends on how comprehensive the tests are, perhaps, at some point
    • the amount of testing you're writing exceeds code, might be frustrating)
  • Breaking production would lose him his job, but it was still frowned upon
  • Not something you want to be doing
  • Able to fix it immediately
    • on the main page of the service, 10 minutes max
    • quick but non-trivial
• How long do these processes take?
  • A commit takes, like, half a second
  • "git commit -m "blaaaah"" is pretty much trivial
  • Running tests? (I want to run tests --> tests have been run)
  • a couple minutes if there's a lot of them, it varies
  • you "run tests" and it runs all the tests
    • have to go get aquainted with them again when one of them fails
  • Deploying - super easy, but various stuff had to happen depending on what you were deploying
  • a "if it takes a minute go get coffee" sort of thing
  • never really been in a long-deploy-time situation
• Ever set up a continuous integration server?
  • No, not personally, not from scratch
• Testing process: consistancy would be a good thing, integrating it would be useful, though a possible threshold at which it becomes frustrating
• What has kept him from integrating tests? It's already been set up, or someone's already done it, OR it's too small to be worth it
• Feels he could make something that meets Jesse's threshold, would probably take some time and cause some frustration