Issues with getting up-and-running

[stevepeak]

I'm very interested in Noms but found it very difficult to get started as a modest yet experienced developer who is new to Go and Noms.

I'm using Docker to start and operate Noms.

Start Noms

docker run --name=noms -d -v noms:/data -p 8000:8000 scjalliance/noms

That was easy. Now noms is running.

Let's interact with it by starting a new container and linking it to noms.

docker run -it --rm --link noms:noms scjalliance/noms /bin/bash

At this point we access bash within the new container. Our noms db is located at http://noms:8000

⚠️ Issue 1

I wanted to try the csv importer. But quickly got errors that I could not resolve (remember, Go newb).

go install github.com/attic-labs/noms/samples/go/csv/import

As stated here https://github.com/attic-labs/noms#run

can't load package: package github.com/attic-labs/noms/samples/go/csv/import: cannot find package "github.com/attic-labs/noms/samples/go/csv/import" in any of:
    /usr/local/go/src/github.com/attic-labs/noms/samples/go/csv/import (from $GOROOT)
    /go/src/github.com/attic-labs/noms/samples/go/csv/import (from $GOPATH)

Continuing

I decided to install the csv importer from source. I needed to install some dependencies as well.

git clone https://github.com/attic-labs/noms.git --depth=1
cd noms/samples/go/csv
go get github.com/dustin/go-humanize
go get github.com/juju/gnuflag
go build 
apt-get update
apt-get install less

Now let's pull the data for the example.

curl 'https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv?accessType=DOWNLOAD' > rows.csv

Now let's use the CSV importer.

⚠️ Issue 2

Now I got some more errors when running go run csv-import/importer.go rows.csv http://noms:8000::foo

# command-line-arguments
csv-import/importer.go:53:30: cannot use "github.com/juju/gnuflag".CommandLine (type *"github.com/juju/gnuflag".FlagSet) as type *"github.com/attic-labs/noms/vendor/github.com/juju/gnuflag".FlagSet in argument to spec.RegisterCommitMetaFlags
csv-import/importer.go:54:30: cannot use "github.com/juju/gnuflag".CommandLine (type *"github.com/juju/gnuflag".FlagSet) as type *"github.com/attic-labs/noms/vendor/github.com/juju/gnuflag".FlagSet in argument to verbose.RegisterVerboseFlags
csv-import/importer.go:55:30: cannot use "github.com/juju/gnuflag".CommandLine (type *"github.com/juju/gnuflag".FlagSet) as type *"github.com/attic-labs/noms/vendor/github.com/juju/gnuflag".FlagSet in argument to profile.RegisterProfileFlags
root@c0090d3e851a:/go/noms/samples/go/csv#

I resolved that issue by commenting out the following lines. https://github.com/attic-labs/noms/blob/1ac2da2e2a38a7ca6c5712954cde8e7f1784b9f0/samples/go/csv/csv-import/importer.go#L24 https://github.com/attic-labs/noms/blob/1ac2da2e2a38a7ca6c5712954cde8e7f1784b9f0/samples/go/csv/csv-import/importer.go#L53-L55

Continuing

go run csv-import/importer.go rows.csv http://noms:8000::foo
noms show http://noms:8000::foo

Success! I now have data in Noms and able to view it. Now it's time to make commits and changes to the dataset. 🤞

---

TL;DR

Maybe I'm missing something obvious, but I wonder how many other developers are experiencing the same issues and abandoning using Noms because they cannot figure out how to interact with it. There should be more straightforward documentation to help onboard new developers. Noms has great potential but these gateway issues are restricting usage and not promoting trust in the project.

[aboodman]

Hi!

Issue (1) was a stupid typo in the setup readme that I recently introduced. Now fixed. Should have been csv/csv-import, not csv/import.

Issue (1.5) (the missing deps) -- fair I suppose. I'm not sure how much detail to go into in the setup, because typical linux and osx installs have less, and it's fairly self-explanatory when that is the problem.

Issue (2) doesn't reproduce for me:

root@dfa02b7a504c:/go/src/github.com/attic-labs/noms/samples/go/csv# go run csv-import/importer.go /tmp/data.csv http://noms:8000::foo

root@dfa02b7a504c:/go/src/github.com/attic-labs/noms/samples/go/csv# noms show http://noms:8000::foo | head -n10
struct Commit {
  meta: struct Meta {
    date: "2018-01-07T05:28:33Z",
    inputFile: "/tmp/data.csv",
  },
  parents: set {
    #a92poju0cq0ek9bpi84hkkj014unpno6,
  },
  value: [  // 236 items
    struct Row {

[aboodman]

OK, so looking at this a little closer, I think we had one bug -- the typo in the path of csv-import.

The rest, I think is really the result of "go-newb" and then hacking around trying to make things work.

• Starting from the beginning, using the docker container, it actually worked fine, but who knows. We don't maintain that, so it's not clear why to expect any of the instructions here to make sense against it.
• Missing Go package deps. They are there, e.g.: https://github.com/attic-labs/noms/tree/master/vendor/github.com/dustin/go-humanize. It looks like perhaps you just cloned the code into a directory and expected that to build. It won't, and that's a standard Go thing. That's why the setup instructions say to use go get, not git clone -- it would have put it in the right place.
• Missing binary deps. Arguable ... I feel like listing every single system dependency is counterproductive in a case like this. I think it is much more common for less to be present than not, and when it's not, it's self-explanatory.
• Build errors -- this is also a result of Go environment issues. If you are new to an environment and something doesn't build, is it more likely that it is broken and you should start hacking around in the code, or more likely your environment isn't configured correctly?
• I also noticed you use go run to run csv-import. That works in this case, but it's not what the instructions say to do and is just by luck that it worked here.

Generally speaking, there are of course bugs (like the typo you found), but we do run Noms on a continuous build and test. So if something just doesn't build, it's likely to be something local to you (particular in your case because you're new to Go), not something in the source.

As for the meta question of how much to cater to people totally new to the environment, it's hard to say. We used to make native binaries available, and that was easier, but it was harder for us to maintain. Perhaps we could have a separate "building Noms" page that assumes you know nothing. But in that case, how far back do we go -- do we teach about installing build tools on osx?

[aboodman]

Answering own question: I think just providing native binaries is probably better. And relying on a docker image like the one you did is probably a close second. Everything would have probably gone according to plan if it weren't for the typo in issue 1.

[stevepeak]

Thank you @aboodman for helping this go-newb 👍 😄

I certainly understand that using a non-supported Docker container could cause problems.

What is the hesitation to supporting Noms from a Docker container? IMO supporting a Docker build would help greatly with onboarding new developers and adoption rates.

[aboodman]

No real hesitation... just something we haven't thought about.

I wonder how the version of noms in the container would stay updated... I would almost like it if on startup it synced and re-built itself. Otherwise we'll have to maintain an up to date container (unless there's some service somewhere that does this?). I'm a bit of a docker newb.

[stevepeak]

You can automate this process. Once you build a Dockerfile you can have Docker Cloud rebuild it on every commit (or release). I'll be happy to produce this Dockerfile and help where I can.

[aboodman]

That sounds great. Is there a cost to have Docker Cloud do that? If not, I definitely support creating this image and referencing it as part of the setup instructions.

[stevepeak]

Nope, free 👍

[aboodman]

Sounds good, feel free to create a bug and get started :-) !