Swift Implementation Status

[chrispomeroyhale]

Overview

I am writing an idiomatic Swift implementation of data package and table schema. I have a partial implementation of these libraries based on my own needs. A foundation is in place which provides a subset of features. There are still some architectural decisions to be revisited, additional tests to be written, and missing features to be implemented. I anticipate that this library will prove useful to someone and it could benefit from being more robust, too.

Context

In addition to providing implementation status it may be helpful to place it in context. The Swift language is a product of the industry to support Apple's ecosystem including their evolving mobile platform. Their recent attitude shift allows for greater flexibility in transferring files. This is where my own usage comes into play: My emphasis is on enabling anyone to access their small data so that it can be carried with someone beyond the lifetime of an app.

Swift is a young language which comes with its strengths and needs. One of the challenges with Swift its lack of a stable ABI. In practice, this makes the library a moving target for developer-users but it doesn't prevent writing or consuming an open source library. Although this scenario is not unfamiliar for projects in languages like Python and JS, keys distinctions are the yearly pace at which Swift releases new language versions and the lack of the language in Linux distribution package managers.

Another consideration is Linux support. While Swift is functional on Linux, the foundational library is a community effort that is still a work in progress. By way of example, ISO 8601 date parsing support is incomplete. I view Linux as a secondary objective while this effort is ongoing.

Notable Differences

Here are some derivations from the reference implementations. Any of these items may be reconsidered.

• Descriptor deserialization results in a class representation of the descriptor over an object that maintains a JSON descriptor. Leverages failable initializers (i.e. factory methods that can return a null reference). As a consequence, serialization may not produce the same JSON.
• Design choices revolving around the language being statically-typed over deferring type inference to the developer-user. Prefers types over ambiguity. Prefers maintaining an object graph over stringification where possible.
• Use of log levels for errors (e.g. MUST, REQUIRED, SHALL) and warnings (e.g. SHOULD, RECOMMENDED). Provides a log over throwing exceptions which allows for a command-line utility to display all warnings and errors for a package.
• No external dependencies.

Status

datapackage

Requirements not implemented or incomplete (strike-thru marks completion):

• read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas
• create and edit a data package descriptor, including methods to add and remove resources and validation after editing
• provide an API to interact with a data package descriptor
• streaming and reading of resources through a table schema with cast on iteration
• save a data package descriptor to a file path
• save a data package as a zip file on disc
• read a zip file containing a data package
• cover the latest changes in the specs such as string/array for path etc
• support for schema dereferencing before validation
• support for multipart resources
• support for csv dialects

tableschema

Requirements not implemented or incomplete (strike-thru marks completion):

• read and validate a table schema descriptor
• create and edit a table schema descriptor
• provide an API to interact with a table schema descriptor
• infer a table schema descriptor from a supplied sample of data
• validate a data source against the table schema descriptor, including in response to editing the descriptor
• streaming and reading of a data source through a table schema with cast on iteration
• save a table schema descriptor to a file path
• cover the latest changes in the spec such as bareNumber, true/falseValues, string PK/FK etc
• support true data streaming
• support all data constraints
• support all data types

[lwinfree]

Thanks for checking in via Gitter! I'm copying the text here for continuity: 'I wrote a partial Swift language implementation of [Tabular] Data Package, Table Schema for my own project (which is used in the shipped Quotemarks iOS app). It is unlikely I'll have the time/energy to complete the entire specification (e.g. zipping, data inference, every single data type). I haven't yet open sourced it since it's incomplete. If someone on here wants to complete the Swift work, reach out to me.... I did write a dialectal CSV parser in Swift since one did not exist already. That's open sourced here: https://github.com/slythfox/csv-dialect-swift'

[roll]

@slythfox @lwinfree WDYT of publishing and moving the Swift implementation to the FrictionlessData organization to create momentum in finishing it?

[lwinfree]

@roll & @slythfox that works for me. I think @slythfox was saying that they hadn't publicly released the code yet since it isn't finished, but I support having it open & on the main FD organization if that is OK with Slythfox!

[chrispomeroyhale]

@lwinfree @roll My original plan was to publish draft implementations to get some initial feedback in case we wanted to change the APIs. I'm comfortable moving this TableSchema Swift implementation over now, followed shortly by what I have for DP/TPD, and we can make changes in the open as needed. I'm also open to moving the Swift CSV parsing library over if that's in the scope of FD org. How might we proceed?

[roll]

@slythfox Great! I have created a Swift team for the FD org and 2 repositories and sent you an invite.

[roll]

Please let us know if we need to arrange something else (e.g. a repo for the parser). Also, @lwinfree will share the news about new implementation when you're ready to ask for feedback from the community.

BTW One of the main channels: https://gitter.im/frictionlessdata/chat

[roll]

@slythfox Sorry, I've missed your message in Gitter but now it's answered. Also, feel free to ask any questions here in comments (Github notifies about new messages much better)

[lwinfree]

Hi @slythfox! Are you able to add your code to these repositories? https://github.com/frictionlessdata/datapackage-swift & https://github.com/frictionlessdata/tableschema-swift Let us know if we can help or support you in any way! Hope you are safe & healthy. Thanks

[chrispomeroyhale]

@lwinfree @roll Thank you both for your patience with me. Pardon my username change. I've gone ahead and published the TableSchema implementation. I'll start using this repo for my own project and get DataPackage published when I have a chance... honestly haven't needed to look at it for awhile. Always open to feedback and collaboration

[roll]

@chrispomeroyhale Congrats! It's really great!

Let us know when you're ready to share this with a broader community (to get feedback and if needed some help from outside collaborators).

[rufuspollock]

@chrispomeroyhale how's this going? We'd love to publicize this /cc @lauragift21 @roll @lwinfree

[chrispomeroyhale]

@rufuspollock I am currently in the process of reviewing the DP/TDP implementation again and writing up documentation in preparation to publish. I am expecting to wrap that up in just a few days.

[rufuspollock]

@chrispomeroyhale great 👏 👏

[chrispomeroyhale]

@rufuspollock @roll I've now published the DP/TDP implementation. There's always something more to do, with CI and code coverage are next on my list. I should be available to chat on Discord now as well

[chrispomeroyhale]

@roll I'd like to set up TravisCI though I don't have permissions to enable/configure for these repos. I have a fork working with TravisCI and Codecov.io for code coverage integration. I see that other implementations are using Coveralls, but getting that working seems more involved as I don't know of a Coveralls tool that supports Swift projects using Swift Package Manager.

[roll]

Hi @chrispomeroyhale,

I have enabled Travis and Coveralls. Codecov seems not to need to be enabled but it requires a secret token (I can email or discord it to you if you'd like OR you can use your own)

BTW, maybe this project can help with Coveralls - https://github.com/macmade/xcode-coveralls? For other languages, we just run coveralls command from Travis and it works with no tokens

[roll]

Hi @chrispomeroyhale,

I've got your message in Discord. I think we don't need to set COVERALLS_TOKEN. It seems to be working without the token by checking that the coverage report is sent from Travis. We haven't added this token for any of our repos.

[chrispomeroyhale]

@roll I ended up getting Coveralls working a few weeks ago using coveralls-lcov since this project builds with Swift Package Manager and not Xcode (for minimal Linux support). Turns out neither Coveralls nor Codecov needed tokens on Travis CI.

Where we stand now: Both libraries are building on Travis CI with Coveralls code coverage integration and documentation on feature implementation status in the READMEs. The libraries are used in a shipping app in conjunction with my csv-dialect-swift project. Included is a simple CLI tool for displaying warnings/errors which could be used for library comparison absent validation against JSON Schema.

For the interested reader: I do have an unpublished Core Data "atomic store" for importing/exporting which I could potentially make available upon request -- not to be confused with OKFN's Core Datasets.

Although there's always more that can be done, for now these tools are serving my purpose. My hope is this may help pave the way towards opening up data accessibility options on Apple products.

[roll]

Awesome @chrispomeroyhale!

@lwinfree @chrispomeroyhale, should we start introducing the Swift implementation to the community?

[lwinfree]

Yes I think we should share with the community now! I'll plan to write a post on discord tomorrow :-)

[chrispomeroyhale]

@lwinfree @roll Sure! I've just listed the packages on both Swift Package Directory and Swift Package Index as well.

[roll]

That's differently DONE :tada: Thanks a lot @chrispomeroyhale