Add CKAN API

[oxinabox]

Do you know the type of the hash? I can't work it out

https://stackoverflow.com/q/50712671/179081

[SebastinSanty]

No, I am not able to figure out which type of hash it is :/

[oxinabox]

The big question to me is how do we go from a landing-page on a site that supports CKAN, to a CKAN, API page.

Right now this only works for the CKAN demo site right?

Examples:

• Data.gov.au

  • Landing page: https://data.gov.au/dataset/2016-soe-atmosphere-hourly-co-and-24h-pm2-5-concentrations-measured-during-the-hazelwood-mine-fire
  • API page: https://data.gov.au/api/3/action/package_show?id=2016-soe-atmosphere-hourly-co-and-24h-pm2-5-concentrations-measured-during-the-hazelwood-mine-fire

• Data.gov

  • Landing Page: https://catalog.data.gov/dataset/consumer-complaint-database
  • API page: https://catalog.data.gov/api/3/action/package_show?id=consumer-complaint-database

Maybe the solution is to just replace the first use if the word dataset in the URL with the string api/3/action/package_show?id=

Which is roughly what you are doing.

It seems simple and fragile, but if the whole site is actually built onto of CKAN maybe it will always have that structure.

[codecov-io]

Codecov Report

Merging #26 into master will increase coverage by 0.25%. The diff coverage is 96.77%.

@@            Coverage Diff             @@
##           master      #26      +/-   ##
==========================================
+ Coverage    93.8%   94.06%   +0.25%     
==========================================
  Files          12       13       +1     
  Lines         210      236      +26     
==========================================
+ Hits          197      222      +25     
- Misses         13       14       +1

Impacted Files	Coverage Δ
src/DataDepsGenerators.jl	93.54% <100%> (+0.21%)	:arrow_up:
src/CKAN.jl	96% <96%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update cad5daa...ec5fd70. Read the comment docs.

[oxinabox]

Got an answer on stack overflow: https://stackoverflow.com/a/50758819/179081

So looks like the hash isn't consistent between upload tools. And I'm guessing we can't tell what the upload tool was.

So I think best would be to just not emit the hash, and fall back to DataDeps.jl's default behavor where it tells the user the hash of the file on the first download so they can edit it into their source.

We could think about making PR's to all the clients (Line numbers are linked in the S.O post) but that would be a long term solution.

[oxinabox]

Other thing is we need integration tests. After that, I think can merge.

It might be worth defining a macro or function that triggers the integration and reference tests and just using that everywhere.

Does generate take a datadep name as an argument? It should.

Then can do something like

function test_generation(repo, url, name)
    @testset "name" begin
        registration = generate(repo, url; name=name)
        @reference_test "references/$name" registration

        eval(parse(registration))
        @test length(readdir(@datadep_str(name))) > 0 
    end
end

Might be better to do that as another PR, another time.