Standard Implementation Test Suite

jbenet commented 8 years ago

we should have a standard test suite in this repo, to be used across implementations.

Perhaps:

require an implementation can produce an executable we can call to exercise an implementation like the go-multihash bin. (options below)
setup a sharness test suite with clear examples that can test any bin
make it easy for this suite to be included in an implementation to test it.
setup a sharness suite to test all known implementations in this repo.

@chriscool maybe this is something you could help with? o/

Proposed options of the bin:

> multihash --help
usage: multihash [options] [FILE]
Print or check multihash checksums.
With no FILE, or when FILE is -, read standard input.

Options:
  -a, --algorithm string  one of: sha1, sha2-256, sha2-512, sha3 (default: sha2-256)
  -c, --check string      check checksum matches
  -e, --encoding string   one of: raw, hex, base58, base64 (default: base58)
  -l, --length int        checksum length in bits (truncate). -1 is default (default: -1)
  -q, --quiet             quiet output (no newline on checksum, no error text)

chriscool commented 8 years ago

Ok I will take a look at this soon.

jbenet commented 8 years ago

We need:

[ ] flesh out the sharness test suite to test all the algorithms, lengths, etc
- we can define the sharness test suite as the bin's "api spec"
[ ] make it easy to swap other binaries in the sharness test suite.

@chriscool possible to take a look at this sooner? @Kubuxu wants to be able to test his c-multihash impl with it.

jbenet commented 8 years ago

In js-land, we also need:

[ ] making the js-multihash bin to conform to this test suite
[ ] adding the test suite to js-multihash/js-multihashing

@RichardLitt possible for you to handle this one? or @dignifiedquire @diasdavid

dignifiedquire commented 8 years ago

I can help making the js cli compliant but I don't have time writing the tests.

dignifiedquire commented 8 years ago

nvm just saw the base setup will take a look at integrating Monday/Tuesday

chriscool commented 8 years ago

PR #39 starts to implement some tests.

@jbenet: to make it easy to swap other binaries, do you want the test suite to check if for example the MULTIHASH env variable is defined and, in this case test $MULTIHASH instead of 'multihash'?

Kubuxu commented 8 years ago

Also something we have discussed is separating multihash from multihashing. Meaning we should have two sets of tests, one for encoding and decoding and second for hashing.

jbenet commented 8 years ago

@chriscool: yeah, maybe MULTIHASH_BIN ?

@Kubuxu: maybe. Let's start with just different test files in the sharness tests and break it out only once needed. On Sat, Aug 6, 2016 at 16:09 Jakub Sztandera notifications@github.com wrote:

Also something we have discussed is separating multihash from multihashing. Meaning we should have two sets of tests, one for encoding and decoding and second for hashing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/multiformats/multihash/issues/32#issuecomment-238046401, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIcoW0hGl_xEOT4T0ZiUbcKzt9cUwEAks5qdOoBgaJpZM4I6eEy .

RichardLitt commented 7 years ago

@dignifiedquire What is the status of the test suite?

dignifiedquire commented 7 years ago

@RichardLitt same as before, I completely forgot about this :/

RichardLitt commented 7 years ago

@dignifiedquire Will you be able to get to it? Alternatively, I can take a look.

dignifiedquire commented 7 years ago

@RichardLitt please do so, but keep im mind js-multihasing-async is the main used module now

multiformats / multihash

Standard Implementation Test Suite #32