search

dhhoang / IonDotnet

Amazon Ion ( http://amzn.github.io/ion-docs/ ) library for dotnet
MIT License
12 stars 4 forks source link

Integrate with ion-tests. #2

Open tgregg opened 6 years ago

tgregg commented 6 years ago

@dhhoang Promising work so far. Nice job!

Before you get too far down the road of trying to develop your own collection of test data, please consider integrating with ion-tests, which is a repository of test Ion data cultivated over the years. The other known Ion implementations depend on ion-tests via a git submodule.

The ion-tests README enumerates the test semantics that should be applied to different subsets of files within the repo. Implementing all of these semantics provides excellent coverage over valid data, invalid data, and many edge cases. And if you find something missing, pull-requests are always appreciated.

Also take a look at the ion-test-driver package, which aims to measure consensus across Ion implementations. Integrating IonDotnet with the ion-test-driver would measure how consistent IonDotnet is with the other implementations when processing the ion-tests data.

Don't hesitate to reach out if you need help.

dhhoang commented 6 years ago

@tgregg Thanks for pointing me to these repos! ion-tests is now added as a submodule. In the coming weeks (months) I'll start writing comprehensive testings for the project now that we have some basic implementations.

dhhoang commented 6 years ago

@tgregg Looking at ion-tests there are a set of Ion files in both text (.ion) and binary(.10n) format. It should be easy to write tests for the text files and some self-describing binary files (such as nullBool.10n) but some binary files need a description of the data content to write test for (like item1.10n).

So far I have not seen any detailed description about the content of those files. Is there anything like that that you can point to? (Of course we can try using ion-java to load each file and see the content but that would be pretty tedious).

tgregg commented 6 years ago

In the bad/ subdirectory there is a README which describes the content of the binary files contained within. Unfortunately, there isn't a similar README for the good/ subdirectory. The file names are the best indication there, but as you've pointed out, not all of them are descriptive. For good/ binary files, re-writing them as text Ion is often the quickest way to see what they contain (I recommend using a tool like ion-c's CLI). In other cases, using hexdump (or an equivalent tool) to examine the bytes may the be best option. I would like to create a "manifest" for all of the files, which would contain descriptions of them for use in generating helpful error messages when tests fail. This has not yet been done.

I will point out, however, that implementations should not write individual tests for every test file in ion-tests. That would be way too tedious. Instead, the files should be treated as parameters to a small set of tests which exercise the test semantics described in the top-level ion-tests README. The other implementations have good examples of how to do this (the most concise being ion-python's).

rickardp commented 6 years ago

Wow, that’s what I like about Python! I wonder if it is possible to do something similar with MStest, just to get quickly up to speed. Otherwise a simple code generator (e.g. T4) could be an option.

dhhoang commented 5 years ago

As an update the test vector semantic is being implemented at Vector.cs .